Big Data For Fun And Profit

I have seen the future, and it's pretty weird.

Several entities out there have set about ... well, just sitting and listening. And collating what they hear, trying to find patterns. Some we know about, others we don't. Whether we like it or not, the era of Big Data is upon us. No one, least of all those who are trying to tap into it, know exactly what that means yet.

What can you discover, if you have a big enough data set? What kinds of answers can you tease out of it?

That's part of the philosophy behind Wolfram Alpha, which I've written about before. Alpha is kind of like Google, but more focused. Let's say you wanted to know how many people lived on Earth in 1863. You can search Google for resources that will tell you about historical planetary population. Or you can go to Alpha, type "world population in 1863", and it'll straight up tell you that in 1863, the world's population was 1.26 billion people. If you're curious, you can revise your query to "India population 2013", and you're treated to the notion that the equivalent of the entire human race circa 1863 lives in today's India. I'm ... not entirely sure what to make of that. But it's definitely food for thought.

The point is, between them Google and Wolfram have harnessed an immense amount of publicly-available data, made it massively interconnected, and set it loose on the public at large. On the whole, this is a good thing. Back when I was in school, one of the first things they taught us was how to use the library's card catalog. You could find a lot of stuff in that card catalog. Well, nowadays, just about everyone carries a card catalog that indexes almost the entirety of human knowledge in their pocket. And with just a little more effort, they can unleash an agent who will go search that catalog, giving them just the information they're looking for. You can search for any kind of data: population, financial, historical, whatever.

And then, there's Akinator.

Akinator's conceit is that a genie is playing guessing games with you: you think of a character, and Akinator will ask questions until he guesses who you're thinking about. I suspect -- but I don't know for sure, since they're not really telling -- that it's an enormous database, one that grows and learns from each of its defeats. It's Big Data applied to amusement, as opposed to research. Yes, you can stump Akinator, but you really have to work at it. At least it's honest. Well-known characters and historical personages, he'll guess in fairly short order; more obscure references may take time to narrow down. Within another year, if they're still running by then, it may be nigh-impossible to put one over on the old boy.

Finally, you have the prediction markets, which are another expression of Big Data ... sort of. By allowing people to collaborate anonymously, they allow a real-time expression of the Wisdom of Crowds principle. This is an invaluable resource for ... well, just about all of us. In 2008 and 2012, no one who was regularly reading Intrade was surprised by the election outcome. Which is unfortunate, because government busybodies shut Intrade down earlier this year. Maybe they'll be back. I sure hope so. In the meantime, there are other prediction markets out there. I'm going to be giving the Iowa Electronic Markets a workout in next year's Congressional races, and I'll let you know how it turns out.

Lastly ... what must it be like, to grow up in this world? Our kids have never really known a world where everything wasn't indexed. I've touched on this topic before, and don't really have anything new to add. Whether we want to or not, whether we like it or not, we are now raising the first generation of cyborgs. They do not understand what it's like to be involuntarily lost. They do not understand what we mean by "privacy." And they do not understand what it's like not to have information at their fingertips. And increasingly, it's going to be their world.

And we're going to have to adapt to living in it.

But you know what? We will. That's what we do. We shape our tools, then our tools shape us, in an endless recursion. The future always looks weird to those who first see it.

But, eventually, we all get used to it.

Sesquicentennial, Part XXXIII: The Cause

--FIRST -PREV NEXT-

There's a scene from Band of Brothers that I'm fond of. Maybe I shouldn't be. But I enjoy a good rant, and here, he rips off a good one. David Webster is a private with the 502nd Parachute Infantry Regiment. He's well-educated, a Harvard man, and is able to find the exact words to give life to his frustrations.



Of course, shortly thereafter, they'd find out why they were there. Oh, not the reason why they'd set out. But they would discover to their horror why this war, and their sacrifice, was absolutely necessary. Once they'd discovered the camps, exactly no one ever asked that question ever again.

In late 1863, the Union was facing a similar quandary. Many people were beginning to ask if the restoration of the Union was worth the horrifying cost in blood and treasure that the task was demanding, with no clear end in sight. It was with this question as a backdrop that a new soldiers' cemetery was to be dedicated, at Gettysburg.

The Battle of Gettysburg, fought that summer, was the bloodiest battle yet fought in the Western Hemisphere. Indeed, a century and a half later, it remains so. Some of the soldiers, when identified, were sent home to be buried. Many couldn't. A section of the battlefield was set aside, then, as their final resting place, and in November a dedication ceremony was to be held.

It was a grand affair, with bands, choirs, and an oration by one of the most celebrated speakers of the day, Edward Everett. Everett's speech was a real stem-winder, lasting about two hours. The man delivered it from memory, which I find astounding. Men were made of sterner stuff in those days, speakers and audiences alike. This was to be the focal point, the main event, of the day. After Everett wound down, President Lincoln was to make a short speech billed as "Dedicatory Remarks". No one quite expected what was about to happen.

Four score and seven years ago our fathers brought forth on this continent a new nation, conceived in liberty, and dedicated to the proposition that all men are created equal. 

Now we are engaged in a great civil war, testing whether that nation, or any nation so conceived and so dedicated, can long endure. We are met on a great battlefield of that war. We have come to dedicate a portion of that field, as a resting place for those who here gave their lives that that nation might live. It is altogether fitting and proper that we should do this. 

But, in a larger sense, we can not dedicate, we can not consecrate, we can not hallow this ground. The brave men, living and dead, who struggled here, have consecrated it, far above our poor power to add or detract. The world will little note, nor long remember what we say here, but it can never forget what they did here. It is for us the living, rather, to be dedicated here to the unfinished work which they who fought here have thus far so nobly advanced. It is rather for us to be here dedicated to the great task remaining before us -- that from these honored dead we take increased devotion to that cause for which they gave the last full measure of devotion -- that we here highly resolve that these dead shall not have died in vain -- that this nation, under God, shall have a new birth of freedom -- and that government of the people, by the people, for the people, shall not perish from the earth.

Edward Everett himself said afterwards, "I should be glad if I could flatter myself that I came as near to the central idea of the occasion, in two hours, as you did in two minutes." Others weren't nearly as kind. The Chicago Times, a paper with a heavily Democratic editorial board, wrote, "The cheek of every American must tingle with shame as he reads the silly, flat and dishwatery utterances of the man who has to be pointed out to intelligent foreigners as the President of the United States." But love it or loathe it, Lincoln got one thing wrong. The world would long remember what was said here. Because while between them Vicksburg and Gettysburg would prove to be military turning points in the war, this speech would prove to be a political turning point. Lincoln was beginning to reframe the conflict.

His opening phrase, describing America as a nation "conceived in liberty, and dedicated to the proposition that all men are created equal," is a "take that" aimed directly at the President of the Confederacy, Jefferson Davis. Davis, and his Vice-President, Alexander Stephens, characterized their nation as a union between States, and that only. The people have no business in the business of their nation, except second-hand, as citizens of their State. Also, "all men are created equal," a phrase drawn directly from the Declaration of Independence, stands in direct counterpoint to an address by Alexander Stephens, where he said, "Our new Government is founded upon exactly the opposite ideas; its foundations are laid, its cornerstone rests, upon the great truth that the negro is not equal to the white man, that slavery, subordination to the superior race, is his natural and normal condition."

Here, he makes the distinction plain, without coming right out and saying it: this war must become a war for freedom. It must become the Final Solution to the Slavery Question. Too much has been paid, too many have been lost, for this war to merely be about recovering territory. It must become bigger than that, it must become better than that. The sentiment caught, and spread like wildfire.



What had been a war to restore the Union was becoming a crusade for human liberty. In some people's eyes it had always been such, for others, they'd be slow in coming around. But come around they would.

They'd paid too much and bled too deeply for it to be anything else.