Crowdsourcing goes professional: The rise of the verticals

Over the last few months, I see a trend. Instead of letting end-users interact directly with the crowd (e.g., on Mechanical Turk), we see a rise of the number of solutions that target a very specific vertical.

• CastingWords: Audio transcription
• SpeakerText: Video transcription
• Serv.io Translate: Translation (by CloudCrowd)
• Serv.io Edit: Proofreading (by CloudCrowd)
• CrowdFlower Business Listing Verification
• CrowdFlower Search Relevance
• CrowdFlower Product Categorization
• BetterOCR: Improving Optical Character Recognition
• Tagasauris: Photo tagging
• MediaPiston: Targeted content generation

Add services like Trada for crowd-optimizing paid advertising campaigns, uTest for crowd-testing software applications, etc. and you will see that for most crowd applications there is now a professionally developed crowd-app.

Why do we see these efforts? This is the time that most people realize that crowdsourcing is not that simple. Using Mechanical Turk directly is a very costly enterprise and cannot be done effectively by amateurs: The interface needs to be professionally designed, quality control needs to be done intelligently, and the crowd needs to be managed in the same way that any employee is managed. Most companies do not have time or the resources to invest in such solutions. So, we see the rise of such verticals that address the most common tasks that were accomplished on Mechanical Turk.

(Interestingly enough, if I remember correctly, the rise of vertical solutions was also a phase during web search. In the period in which AltaVista started being spammed and full of irrelevant results, we saw the rise of topic-specific search engines that were trying to eliminate the problems of polysemy by letting you search only for web pages within a given topic.)

For me, this is the signal that crowdsourcing will stop being the fad of the day. Amateurish solutions will be shunned, and most people will find it cheaper to just use the services of the verticals above. Saying "oh, I paid just $[add offensively low dollar amount] to do [add trivial task] on Mechanical Turk" will stop being a novelty and people will just point to a company that does the same thing professionally and in a large scale.

This also means that the crowdsourcing space will become increasingly "boring." All the low-hanging fruits will be gone. Only people that are willing to invest time and effort in the long term will get into the space. 

And it will be the time that we will get to separate the wheat from the chaff.

Read more »

Definining Probability in Prediction Markets

The New Hampshire Democratic primary was one of the few(?) events in which prediction markets did not give an "accurate" forecast for the winner. In a typical "accurate" prediction, the candidate that has the contract with the highest price ends up winning the election.

This result, combined with an increasing interest/hype about the predictive accuracy of prediction markets, generated a huge backslash. Many opponents of prediction markets pointed out the "failure" and started questioning the overall concept and the ability of prediction markets to aggregate information.

Interestingly enough, such failed predictions are absolutely necessary if we want to take the concept of prediction markets seriously. If the frontrunner in a prediction market was always the winner, then the markets would have been a seriously flawed mechanism. In such a case, an obvious trading strategy would be to buy the frontrunner's contract and then simply wait for the market to expire to get a guaranteed, huge profit. If for example Obama was trading at 66 cents and Clinton at 33 cents (indicating that Obama is twice as likely to be the winner), and the markets were "always accurate" then it would make sense to buy Obama's contract the day before the election and get $1 back the next day. If this was happening every time, then this would not be an efficient market. This would be a flawed, inefficient market.

In fact, I would like to argue that the late streak of successes of the markets to always pick the winner of the elections lately has been an anomaly, indicating the favorite bias that exists in these markets. The markets were more accurate than they should, according to the trading prices. If the market never fails then the prices do not reflect reality, and the favorite is actually underpriced.

The other point that has been raised in many discussions (mainly from a mainstream audience) is how we can even define probability for an one-time event like the Democratic nomination for the 2008 presidential election. What it means that Clinton has 60% probability of being the nominee and Obama has 40% probability? The common answer is that "if we repeat the event for many times, 60% of the cases Clinton will be the nominee and 40% of the cases, it will be Obama". Even though this is an acceptable answer for someone used to work with probabilities, it makes very little sense for the "average Joe" who wants to understand how these markets work. The notion of repeating the nomination process multiple times is an absurd concept.

The discussion brings in mind the ferocious battles between Frequentists and Bayesians for the definition of probability. Bayesians could not accept that we can use a Frequentist approach for defining probabilities for events. "How can we define the probability of success for an one-time event?" The Frequentist would approach the prediction market problem by defining a space of events and would say:

After examining prediction markets for many state-level primaries, we observed that 60% of the cases the frontrunners who had a contract priced at 0.60 one day before the election, were actually the winners of the election. In 30% of the cases, the candidates who had a contract priced at 0.30 one day before the election, was actually the winners of the election, and so on.

A Bayesian would criticize such an approach, especially when the sample size of measurement is small, and would point to the need to have an initial belief function, that should be updated as information signals come from the market. Interestingly enough, the two approaches tend to be equivalent in the presence of infinite samples, which is however rarely the case.

I just could not help but notice that the fight between the proponents and enemies of the prediction markets was reminiscent of the battles between Bayesians and Frequentists :-)

Read more »

Political Prediction Markets: Some Thoughts

Apparently, my last postings on the predictability of the political prediction markets generated some interest. Niall O'Connor decided to check how accurate our predictions are, and after a few days he checked again to see how well we have done.

Our prediction that the price for Hillary Clinton will go down proved to be correct: the price declined from 70, on Dec 2nd (the time of the original post), to 63 on Dec 9th, a 10% decline. Similarly, for Mitt Romney we predicted a decline and the price declined from 24, on Dec 4th, to 18.5 on Dec 9th., a 23% decline.

For Guiliani, we said "The analysis is more difficult in this scenario, but for the next few days we see stabilizing signals with a trend to go upwards" and we were proven wrong: the price declined from 43 on Dec 2nd, to 39.5 on Dec 9th, an 8% decline. I realized what was wrong in my reasoning. What was stabilizing was the sentiment index, not the price. And a stabilized sentiment around 50% tends to be a pretty bad adviser on how the market will move.

The fact that it is possible to predict the prediction markets, bring automatically the question: "why?". What makes the markets predictable? The first answer is liquidity. The markets are not liquid, there are not enough participants, and there is a lot of momentum trading. However, I would like to list another explanation (already posted as a comment on Midas Oracle)

My understanding is that Betfair odds moved from 1.44 to 1.50 (according to the screenshot in the original posting). While indeed this corresponds to a drop from 69% to 66% (an almost 4% drop in share price) this is not as drastic as a drop from 69% to 50% within such a short period of time. Plus, the Betfair drop from 69% to 66% is comparable with the drop in Intrade (from 67% to 64%).

Also, I am not sure about the liquidity hypothesis for explaining the inefficiency. An alternative explanation is the following:

Political markets are not stock markets. They reflect the aggregate opinion of the traders about public's intention for the candidate. Notice that we have two levels of beliefs: one for what traders believe about the public's intentions, and a second for what the public actually intends to vote for.

Not every member of the voting public reads every piece of information. When the same news are being repeated over and over in the mainstream news outlets, then more voters are influenced. Hence, the longer the news about a candidate stay around, the longer the public gets influenced by the same, stale news and changes intentions. This is correspondingly reflected in the prediction markets, potentially in an efficient manner.

This may indicate that it is not that the markets are not efficient, but that the voting public is not "efficient" (i.e., voters do not incorporate all the available information in their voting decisions.)

We can test this hypothesis by testing the efficiency/predictability of political prediction markets vs. the efficiency/predictability of non-political markets.

We will work further with George Tziralis on the topic, and we will keep you posted.

Public commitment is always a good motivation to work harder :-)

Then, Bo Cowgill commented that indeed using text mining together with prediction markets is indeed a good idea. Bo's comment made me think about parallels in "prediction market trading" and "stock market trading". As Bo pointed out, in existing stock markets, there is a significant amount of algorithmic trading. This algorithmic trading makes the stock market significantly more efficient than, say, in the early 1980's where the programmatic trading was at its infancy. In fact, I have heard many stories from old-timers, saying that in the early days it was extremely easy to find inefficiencies in the markets and get healthy profits. As algorithmic trading proliferated, it became increasingly harder to spot inefficiencies in the market.

Something similar can happen today with prediction markets. If we have a prediction market platform that allows automatic/algorithmic trading, then we can improve tremendously the efficiency of today's prediction markets. Furthermore, such a tool (if done with play money) can be used as a great educational tool, similar to the now inactive Penn-Lehman Automated Trading (PLAT) Project. Allowing also for some data integration from the existing prediction markets (BetFair, Intrade, etc.) we could have a pretty realistic tool that can be used for many educational purposes that, at the same time, can generate useful and efficient prediction markets.

Now, I need to find someone willing to fund the idea. Ah, there are a couple of NSF call for proposals still open :-)

Read more »

Are Prediction Markets Efficient?

I was reading Fred Wilson's post about the information efficiency of the venture capital market.

While reading the posting, I started wondering whether prediction markets are efficient, according to the definition of Eugene Fama. In other words, how long does it take for a prediction market to incorporate all the available information about an event? Liquidity seems to be an issue for the existing prediction markets, preventing them from reaching equilibrium quickly. But if we had enough liquidity, how long would it take for humans to "agree" on prices that incorporate all the available information about an event?

Read more »