Big Data For Fun And Profit

I have seen the future, and it's pretty weird.

Several entities out there have set about ... well, just sitting and listening. And collating what they hear, trying to find patterns. Some we know about, others we don't. Whether we like it or not, the era of Big Data is upon us. No one, least of all those who are trying to tap into it, know exactly what that means yet.

What can you discover, if you have a big enough data set? What kinds of answers can you tease out of it?

That's part of the philosophy behind Wolfram Alpha, which I've written about before. Alpha is kind of like Google, but more focused. Let's say you wanted to know how many people lived on Earth in 1863. You can search Google for resources that will tell you about historical planetary population. Or you can go to Alpha, type "world population in 1863", and it'll straight up tell you that in 1863, the world's population was 1.26 billion people. If you're curious, you can revise your query to "India population 2013", and you're treated to the notion that the equivalent of the entire human race circa 1863 lives in today's India. I'm ... not entirely sure what to make of that. But it's definitely food for thought.

The point is, between them Google and Wolfram have harnessed an immense amount of publicly-available data, made it massively interconnected, and set it loose on the public at large. On the whole, this is a good thing. Back when I was in school, one of the first things they taught us was how to use the library's card catalog. You could find a lot of stuff in that card catalog. Well, nowadays, just about everyone carries a card catalog that indexes almost the entirety of human knowledge in their pocket. And with just a little more effort, they can unleash an agent who will go search that catalog, giving them just the information they're looking for. You can search for any kind of data: population, financial, historical, whatever.

And then, there's Akinator.

Akinator's conceit is that a genie is playing guessing games with you: you think of a character, and Akinator will ask questions until he guesses who you're thinking about. I suspect -- but I don't know for sure, since they're not really telling -- that it's an enormous database, one that grows and learns from each of its defeats. It's Big Data applied to amusement, as opposed to research. Yes, you can stump Akinator, but you really have to work at it. At least it's honest. Well-known characters and historical personages, he'll guess in fairly short order; more obscure references may take time to narrow down. Within another year, if they're still running by then, it may be nigh-impossible to put one over on the old boy.

Finally, you have the prediction markets, which are another expression of Big Data ... sort of. By allowing people to collaborate anonymously, they allow a real-time expression of the Wisdom of Crowds principle. This is an invaluable resource for ... well, just about all of us. In 2008 and 2012, no one who was regularly reading Intrade was surprised by the election outcome. Which is unfortunate, because government busybodies shut Intrade down earlier this year. Maybe they'll be back. I sure hope so. In the meantime, there are other prediction markets out there. I'm going to be giving the Iowa Electronic Markets a workout in next year's Congressional races, and I'll let you know how it turns out.

Lastly ... what must it be like, to grow up in this world? Our kids have never really known a world where everything wasn't indexed. I've touched on this topic before, and don't really have anything new to add. Whether we want to or not, whether we like it or not, we are now raising the first generation of cyborgs. They do not understand what it's like to be involuntarily lost. They do not understand what we mean by "privacy." And they do not understand what it's like not to have information at their fingertips. And increasingly, it's going to be their world.

And we're going to have to adapt to living in it.

But you know what? We will. That's what we do. We shape our tools, then our tools shape us, in an endless recursion. The future always looks weird to those who first see it.

But, eventually, we all get used to it.