Worker Evaluation in Crowdsourcing: Gold Data or Multiple Workers?

Evaluating the quality of workers on crowdsourcing environments is standing problem.

The two most common approaches for dealing with the problem are:

• Use "gold" data: Give to the workers questions for which we already know the answers and see how the workers are performing. (Crowdflower is using this approach.)
• Use multiple workers: Give the same answer to multiple workers and then use a latent class model ala Dawid & Skene to estimate the quality of the workers.

The nice thing about the two approaches is that they can be seamlessly combined into a unified algorithm.

What was not clear to me, though, was the relative importance of the two. How much better and faster can we estimate the quality of the workers if we use gold data? What is the rate of improving the quality estimation as we add more gold data? Generating high-quality gold data is an expensive process, so having good answers to these questions is important.

Being a professor, the moment these questions came up, I knew what I had to do: Ask a PhD student to give me the answers! (Thanks Jing!) So, I am just the messenger here, Jing did all he work and the analysis. You know where to send your thanks.

The assumptions: We have examples belonging to 2 categories. The examples are equally distributed in the two categories (i.e., 50% in each). We created a set of workers with their quality picked randomly and uniformly from the range (55% correct) to (100% correct), for an average quality of ~77%. The workers assigned (noisy) labels to the examples, with an accuracy directly proportional to their quality.

We examined the performance of the Dawid & Skene algorithm, which we modified to take into consideration the existence of gold data. We measured two things:

• Classification error: How well the algorithm estimates the correct class of the examples
• Quality estimation error: How well the algorithm estimates the quality of the workers

We experimented with having 0% gold examples in the data, 25% gold, 50% gold, and 75% gold.

Results on classification error

So, we measured the accuracy of estimating the correct class of each example. The results are listed below:

3 workers per example

5 workers per example

10 workers per example

One immediate observation is that the value of having gold data is limited when we have significant number of workers per label. With 10 workers per example, no matter if we have gold data, the difference is minimal. Even with just 5 workers, the additional value of gold data is small.

The cases where it makes sense to use gold data is when we have only small number of workers per label. (Not an uncommon case!) 

An interesting observation, though, is that we can achieve the same effect by simply forcing workers to work on more examples. Once a worker has given us 30 answers, the completely unsupervised algorithm can work almost as well as the algorithm that uses 75% gold data. This holds even when we have just 3 workers per example.

Of course, on an environment like Mechanical Turk, forcing workers to work on a large number of HITs may not be feasible. But we can always bundle multiple questions in a single HIT, achieving the same result. 

OK, so gold data do not seem to be very useful in for getting better accuracy in class estimation. 

But, it should help in estimating the quality of workers, right?

Results on worker quality estimation

For the quality estimation, we also calculated the error when having 100% gold examples. (This is the lower bound for the estimation error, of course.)

3 workers per example

5 workers per example

10 workers per example

As expected, with 10 workers per example, we gain almost nothing in terms of quality estimation when we use gold data. 

With 3-5 workers per example, having gold data improves the quality estimation in a rather consistent manner. Again, though, we observe that if we have each worker completing a large enough number of assignments, we can get most of the benefits of having gold data, without actually having gold data.

Why gold data then?

So, given the results above, someone would ask: Why do we even need gold data? The unsupervised approach seems to work very well!

In reality there are a few reasons for which we may still need to have gold data:

• Imbalanced datasets: When we have very imbalanced data sets, the estimation becomes more challenging. For imbalanced data, we need to quickly and preemptively test workers using data from all categories, rather than waiting for the occasional object from the minority category to appear. To give an example: if we monitor a security camera trying to detect the presence of people in prohibited areas, we want to ensure that the workers will be tested early on with images that have people in them. Otherwise it may take a long time to get a reliable estimate of their ability to classify correctly examples from the minority class.
• Very low quality of workers: When workers have very low quality, we need more workers per example and more labels per worker to replicate the results above. In this case, having gold data allows us quickly to get rid of the workers that do not meet quality standards. This is very useful on high-noise environments like the "unprotected" Amazon Mechanical Turk marketplace (by unprotected I mean without using qualification tests or other quality assurance mechanisms).
• Giving confidence to non-technical people: If you say that you test the workers with known examples, everyone understands the process. If you say that you rely on agreement between workers, or on latent class models ("what?") and on expectation maximization or Bayesian estimation ("come again?") , most people will start feeling uncomfortable. Everyone understands random tests, not everyone is willing to let unsupervised methods to direct the quality assurance process. So, even if the gold examples do not help much, it is a very reassuring factor for people that just want to know that there is a familiar, understandable, and easy-to-explain quality control mechanism in place.
• Calibrating results and giving feedback: One of the final reasons for having gold data is to be able to calibrate workers and give them feedback about the expected coding standards. For example, when rating pages as porn or not and into degrees of severity, different people have different levels of sensitivity. If we have enough "sensitive" workers in the workforce, we may end up with results that are consistent but shifted upwards in terms of severity. (Or vice versa if the coders are more tolerant, the results may be shifted downwards.) This can, incorrectly, give the impression that all data collected through crowdsourcing are wrong. However, if the final user of the data provides a few gold data as anchor points, the Dawid & Skene code gives back results that are more in line with expectations. At the same time, these gold data points can be used to give immediate feedback to the workers about their errors and implicitly direct them to use the expected rating guidelines and self-calibrate.

In practice, the last two reasons are often more important than the technical aspects of estimation. So, before starting any big crowdsourcing annotation project, spend some time and create some gold data. Or, alternatively, take a small dataset and label it using a large number of workers per example. Then verify the outcome, correct and calibrate some of the unexpected results, and run Dawid & Skene again. The generated data will be close enough to gold. Having such gold data will pay back the effort and cost multiple times during the overall process.

Read more »

Analytics for Class Lectures

The classes for the new academic year have started, so naturally I started thinking about teaching-related topics.

Mining video interactions

A few days back, FXPal released TalkMiner, a system for indexing and searching video of lecture broadcasts. One of the interesting ideas is that it is possible to mine the interactions of students with the video, to see what are the topics of interest for the students, what parts of the class get skipped, and so on. From the blog post of FXPal:

The Berkeley webcasting system (developed by our president Larry Rowe while he was a professor there) showed that

� students almost always watched the lectures on-demand rather than in real-time, and they rarely watched the entire lecture.  Students use the webcasts to study for exams � we could see this clearly by patterns of usage � and, they primarily wanted to review selected material covered by the instructor.  In one class we discovered that for over 50% of the lectures, students watched less than 10 minutes from a 50-minute lecture and students watched the entire lecture only 10% of the time.  Consequently, for using the system, effective search is a big issue.

At Stern, all the classes get recorded and are available to students for reviewing the class material. The students get access to a layout like the following and have the ability to rearrange the layout, emphasizing the slides, or the video. (You can see a lecture of mine; login: scribe and password: Scribe987!)

It seems to be a natural next step to show to the instructor the patterns of interaction that students have with the videos. It would be very interested to see what parts of the class go largely unexamined and which ones are played again and again. Needless to say, these are either complicated topics, or topics that the instructor did not explain clearly.

Mining search queries using transcripts

Another interesting idea is to also have transcripts of the class. (For example, for this lecture [login: scribe and password: Scribe987!] see the transcript, done by CastingWords for $0.75/min.) This would allow students to search the class not only using text in the slides but also to recall particular points of the class discussion. This is especially important for courses that have a significant component of in-class discussion. We already know, from web search, that query logs are important source of information. Doing the same for class content would easily identify what students are looking for in the class recordings.

One problem with transcription is that it is rather expensive. CastingWords and SpeakerText seem to charge one or two dollars per minute for human-verified transcriptions. (Fully-automatic solutions are not ready for prime time, as the automatic transcription of these YouTube videos shows. Make sure to click the "cc" button and then "transcribe audio".) With approximately 28 lectures a semester, 75 minutes each, at 1-2 dollars per minute, we have a cost of $2000 to $4000 per semester. At this cost level, it is certainly more beneficial to hire an extra TA rather than provide the transcription of the lecture to the students.

Mining class participation 

Another thing that I would love to have is the ability to transcribe not only what the instructor said but also who are the students that contributed to the discussion, together with what they said. This would allow not only to track and quantify participation but also uncover some patterns that may not be obvious to the instructor.

For example, take a look at this diagram below, created as part of the yearly teaching evaluation that we undergo at Stern:

The diagram was created by an evaluator who sat in my class, tracked the composition of the student body, where each student was sitting in the amphitheater, how many times they raised their hand, and how many times I asked them to answer a question. (To answer the inevitable question: No, the teaching feedback is not focused only on such analyses. In my earlier years, the feedback was focused more on substantial issues, e.g., structuring lectures and discussions, encouraging participation, etc. Now, with feedback and experience, the more substantial and important issues are addressed.  So we focus on such, seemingly more superficial, but also important, stuff...)

The results? I was paying significantly more attention to the left part of the amphitheater: I asked 80% of the time students sitting in the left, and only 20% of the time I asked students on the right. Also, the percentage of female students participating in the discussion was significantly lower: 50% for male students participated, but only 21% of the female students did.

These are patterns that are hard to understand while teaching, but would be easier to find out if we had detailed transcripts of the class discussion together, potentially, with a standardized seat chart. I was also told that some universities (the rumor is about Harvard Business School) use software to track student participation. However, I was not able to locate any such software offerings. 

Moving forward

The ability to videotape lectures has been around for a while and is being used extensively for distance learning applications. (Columbia Engineering had a well-established distance learning program when I joined the PhD program back in 1999.) However, it was mainly a broadcast mechanism, and not a medium for providing feedback to the instructor (and even to the students who can see that they are lacking in terms of participation). 

It would be interesting to start having such technologies for providing feedback on teaching. Analytics have been changing many industries. Education has been surprisingly behind in that respect.

Read more »

Reputation Integration and the Future of Reviews

Last week I was at Google for the annual Google Faculty Summit. While discussing research challenges related to social media and shopping, a common theme and questions emerged: How can we evaluate the trustworthiness of the reviews that abound on the Internet?

Past Performance predicts Future Performance

Current solutions focus on the trustworthiness and history of a reviewer in each site. For example, Amazon has the reviewer rank, which is computed using the amount of reviews contributed, the helpful votes that they amassed, and other secret-sauce factors. Other sites, like Yelp, follow similar approaches to identify the best reviewers.

Of course, knowing the good reviewers is very valuable: Unlike investments, past performance is a strong indicator of future performance. Reviewers that wrote good reviews in the past are likely to write good reviews in the future. Or, more general, high-quality users in the past are likely to be high-quality in the future.

So, problem solved! We just need to know which user has high quality!

Network Effects and Closed Reputation Platforms

Well, at this point we have a problem. Today, we treat site-specific user profiles as separate individuals. A profile of a user on Amazon is not connected to the corresponding reviewer profile in Epinions, in NewEgg, in B&H. So, reviewers that have written tens of reviews and amassed thousands of helpful votes will be insignificant newbies if they decide to write some reviews on B&H website. Similarly, a person that has contributed plenty of reviews and discussions on Chowhound over the years, will be an insignificant newbie on Yelp. The result? We cannot trust the reviews of these individuals, even if they have proven themselves trustworthy in the past!

This is a lock-in associated with network effects, which is similar to the lock-in that happens under closed  and proprietary standards. For example, Microsoft has achieved domination in Office Productivity software by keeping proprietary the file formats for Office. Since no other office productivity suite could inter-operate with Office, smaller players could simply not compete: the minority of users that did not use Office could not exchange files with the Office users. Under such scenarios, either we see a market split into isolated markets (the case of reputation-based sites today), or a single dominant player (e.g., in Office).

Closed Standards and Networks Effects

Interestingly enough, under closed standards, when the market ends up being fragmented, this is not optimum. An excellent example is the SMS market in the United States. Before 2000, the different telecoms did not allow their subscribers to send SMS messages to subscribers of other companies. The main rationale was to force users to adopt the network already chosen by their friends, if they wanted to text each other. So the market looked like that:

Interesting approach but the result was not ideal: texting in the US was essentially non-existent before 2000. (I vividly remember when I arrived in the US how nobody was using SMS to communicate with each other. In Europe sending SMS was commonplace.) However, once the networks decided to cooperate and allow SMS to flow freely across networks, the market took off, attracting more players and being much more useful. 

In other words, if we do not have a monopoly, closed networks are suboptimal. Fragmented markets for goods with network effects are almost never optimal, even for the current dominant players.

The Need for Open Reputation and Reputation Integration

I believe that we see a similar case in the reputation identities. Identities are fragmented across sites and users have little incentive to contribute high-quality content in sites in which they do not have established presence: Nobody pays attention to them in any case.

This also makes it difficult to start from scratch any website that requires the existence of trustworthy reviews in order to attract viewers. Visitors will not come as there are no trustworthy reviews, and reviewers already established in other sites will not join.

So, it is important to start thinking on how to integrate reputations across different websites. It is simply not optimal to have fragmented identities. Even though there are privacy concerns for anonymous profiles, there should be at least the capability to connect identities across sites for users that do not try to preserve their anonymity. By integrating profiles across websites, we can allow reputation to flow across websites and creating a better, comprehensive profile of the participants in today's Internet.

This can allow participants to build their reputation profiles without worrying about lock in. OpenID moves towards the right direction. Next versions should allow explicit, distributed connection of profiles, without requiring a dominant players to control the profile. (Thank you for the offer, Facebook. I will pass.)

Third-party services can play this role as well. By identifying profiles across sites that belong to the same individual, we can quickly learn about the identity of a new contributor. The history can follow the users that are not worried about anonymity. Such strong reputation signals can help improve any existing site that relies on user contributed content. In the presence of strong reputations, low-quality contributors will simply never have a chance of getting a dominant position in the marketplace.

As I mentioned in my earlier blog post, lack of credible reputation mechanisms simply degenerates any market into a market for lemons. In the attention economy, lack of strong reputation signals simply gives the incentives to spammer to come and pollute. Reputation integration mechanisms can solve this issue.

Cross-posted at the Reppify blog (disclaimer: I am an advisor for Reppify) 

Read more »

Mechanical Turk, Low Wages, and the Market for Lemons

In HCOMP this year, one of the memorable and discussed presentations (although highly unconventional) was by M. Six Silberman who discussed the "Sellers' problems in human computation markets". The basic question: can we protect the workers there from exploitation and from sweatshop salaries?

Luis von Ahn posted a similar post on his blog. In the comments of the blog post, someone suggested that the low wages on Mechanical Turk is simply the result of high supply of workers and low demand for their work. As there is more supply, the salaries drop. And having minimum wages, would interfere with the free market.

I actually disagree with this interpretation. First of all, there is no oversupply of labor on Mechanical Turk. The distribution of completion times (follows a power law), suggests that the market operates at maximum capacity. My gut instinct actually tells me that there are not enough workers available for the posted work, not vice versa.

I can hear the protests: If there is not enough supply of workers, why don't requesters simply increase the offered prices?

My explanation: The requesters already pay minimum wages for work that is worth minimum wage. How is that possible given the effective hourly rate of $2/hour?

The basic problem: Spammers. Given that many large tasks attract spammers, most requesters rely on redundancy to ensure quality. So instead of having a single worker to do a task, they get 5 workers to work on it. This increases the effective rate from $2/hr to $10/hr.

Effectively, what Amazon Mechanical Turk is today is a market for lemons, following the terminology of Akerlof's famous paper, for which he got the 2001 Nobel prize.

A market for lemons is a market where the sellers cannot evaluate beforehand the quality of the goods that they are buying. So, if you have two types of products (say good workers and low quality workers) and cannot tell who is whom, the price that the buyer is willing to pay will be proportional to the average quality of the worker. So the offered price will be between the price of a good worker and a low quality worker. What a good worker would do? Given that good workers will not get enough payment for their true quality, they leave the market. This leads the buyer to lower the price even more towards the price for low quality workers. At the end, we only have low quality workers in the market (or workers willing to work for similar wages) and the offered price reflects that.

This is exactly what is happening on Mechanical Turk today. Requesters pay everyone as if they are low quality workers, assuming that extra quality assurance techniques will be required on top of Mechanical Turk.

So, how can someone resolve such issues? The basic solution is the concept of signalling. Good workers need a method to signal to the buyer their higher quality. In this way, they can differentiate themselves from low quality workers. Unfortunately, Amazon has not implemented a good reputation mechanism. The "number of HITs worked" and the "acceptance percentage" are simply not sufficient signalling mechanisms.

Here are some ideas:

• Allowing workers to get endorsements from reputable requesters (to avoid scam rings like on eBay)
• Allowing requesters to post machine readable feedback on the performance of the workers, disconnecting evaluation from the approval rate.
• Certifications and qualification tests that indeed measure ability on different tasks (e.g., language abilities, reading comprehension tests, etc)
• Publishing the reputation history of the workers, so that requesters can evaluate the quality of the worker.

Of course, similar measures can be adopted for requesters! There is a symmetric market for lemons on that side! Scam requesters post HITs, behave badly, and cause good workers to avoid any newcomer. New requesters then get only low quality workers, get disappointed with the quality of the results and they leave the market.

In other words, Amazon can only gain by taking the time to build a more robust reputation system on top of Mechanical Turk. Trust is at the very core of marketplaces. If Mechanical Turk wants to "grow up", then a good reputation system for both sides of the market is grossly overdue.

Read more »

Peer Reviewing for Oral Presentations?

Everyone who has attended a conference knows that the quality of the talks is very uneven. There are talks that are highly engaging, entertaining, and describe nicely the research challenges and solutions. And there are talks that are a waste of time. Either the presenter cannot present clearly, or the presented content is impossible to digest within the time frame of the presentation.

So, my question is: Why don't we have peer reviewing for oral presentations? 

We already have reviewing for the written part. The program committee examines the quality of the written paper and vouch for its technical content. However, by looking at a paper it is impossible to know how nicely it can be presented. Perhaps the seemingly solid but boring paper can be a very entertaining presentation. Or an excellent paper may be written by a horrible presenter.

Why not having a second round of reviewing, where the authors of accepted papers submit their presentations (slides and a YouTube video) for presentation to the conference. The paper will be accepted and be included in the proceedings anyway but having a paper does not mean that the author gets a slot for an oral presentation.

Under an oral presentation peer review, a committee looks at the presentation, votes on accept/reject and potentially provides feedback to the presenter. The best presentations get a slot on the conference program. This also allows the conference to accept more papers that are worthy of inclusion to the proceedings, without worrying about capacity constraints. 

Some other side benefits of this scheme:

• Presentations are accessible in an archival format
• Authors have hard incentives to be better presenters
• The time of the attendees in conferences is not wasted in clearly sub-par presentations.

And if someone says that this system is biased towards good and sleek presenters, I would argue that the system is already biased towards good authors. A well-written paper will eventually have a higher impact than one that is badly written. Same thing for presentations.

Learning to communicate properly the results of our research should be a goal, not an afterthought.

Read more »