Crowd Sourcing

About: peer grading: inventing the light bulb « mooctalk

If you are still interested in crowd-sourcing a fact checker check out the professor's experience with the Stanford MOOC on mathematical thinking of the peer grading etc.
I'm Dr. Keith Devlin, a mathematician at Stanford University. In fall 2012, I launch my first free, open, online math course. This blog chronicles my experiences as they happen.


  1. crowd sourcing
  2. fact checking
  3. mooc


Mark de LA says
It is too bad I never went through the final peer review process. It would have been interesting experience of crowd sourcing for other contexts, I expect.

Mark de LA says
The email pointing to his blog also contains reflections on cheating:
source: ... Several students have contacted me about cases of cheating on the Final Exam. Frankly, why anyone who do this in a course that focuses on learning and offers no credentials, beats me. Students who cheat are really cheating themselves. If you are sure an answer is plagiarized from somewhere else (often easy to determine with a quick web search), you could simply award 1's everywhere, which amounts to a score of 0. Whether you do that for one question or the whole exam is up to you. If there is any doubt that the student has broken the honor code, you have to give the student the benefit of that doubt. 0 on the whole exam is more significant than 0 on one question. Though again, the stakes here are essentially zero; it's mostly about self esteem, surely. For truly egregious cases, send me the details (your login id and the Student number (1, 2, or 3) and I can take it from there. Violation of the honor code is cause for expulsion from the class. Expulsion has occurred, in this class and others, I'm sad to say.

Meanwhile, if you want to know how evaluation training and peer grading looked from our side (the instruction team and the folks at Coursera who were making sure the ship stayed afloat), check out my latest post at to see what was going on behind the scenes. Enjoy! :-)

Mark de LA says
seth 2012-10-31 09:45:42 16266
Was there something specific that you got from Devlin's the peer-review grading that bears on a crowd sourced fact checker?
Yes, the difficulty of organizing ~ 7000 people just to grade one document. The need for good software to do it.  The need for good organization.  The idea of cheating.  It's just an FYI, if it didn't trigger anything for you so be it.

Mark de LA says
  .... & that was with just a mathematical logical subject not even politics!

See Also

  1. Thought No need to fact check the will to believe with 96 viewings related by tag "FactChecking".
  2. Thought about: how it works - ideabuzz with 13 viewings related by tag "CrowdSourcing".
  3. Thought about: how a lone hacker shredded the myth of crowdsourcing with 5 viewings related by tag "crowd sourcing".
  4. Thought about: flattr - social microdonations with 0 viewings related by tag "crowd sourcing".