The Gong Show

Kaggle and Crowdsourced Competitions

I’m fascinated by Kaggle, a web service that provides competitions with prize bounties for crowdsourced to data prediction problems.  They power the Heritage Health Prize, which is attempting to find the best model to predict which patients will be diagnosed with new medical conditions in the coming year. Beyond that one famous competition, there’s many more competitions listed on Kaggle from a wide variety of organizations.

A Learning Resource

Kaggle is great for budding machine learning analysts because they do the most annoying part of real-world machine learning problems for you: data cleansing. It’s a huge pain to take the messy and incomplete data that’s scattered across multiple databases, each in their own data format, and try to collect it all together in a normalized structure.  This practice of data cleansing is important to become a successful analyst in data mining, but it’s a hairy and annoying step that gets in the way of learning the tactics and algorithms needed to become proficient in data prediction analysis. So, all the data sets that Kaggle provides with their competitions are terrific for education purposes.

The Measuring Stick

I’m impressed when startups are able to become *THE* leaderboard which all the people in a vertical use to measure themselves against. For example, TopCoder for years was the de facto measuring stick by which programmers compared their talents to each other internationally. I remember being at Stanford in the early part of the decade and people in the computer science department being really proud that a Stanford team had won TopCoder that year. Being able to capture this type of thought leadership is very valuable.

Kaggle has the opportunity to becomes *THE* leaderboard for machine learning geeks and statisticians.  And once they are in that valuable position, they can use the data exhaust from their service to help solve problems for organizations (for-profit or non-profit) that can help make a large dent in the world. It’s a great opportunity, and I’ll be curious to see how they do executing against it.