Our lives are being "datafied" in ways that range from the plain silly to the sheer scary. Thankfully, the "big data" revolution that is transforming the worlds of commerce, science and medicine is producing enough new insights to justify intrusions on our privacy. If we can tolerate the silly and tame the scary, we could be on the cusp of major progress.
The data revolution really got started about a decade ago, when Wal-Mart harnessed the trillions of bytes of information generated by past purchases to figure out that sales of beer and Pop-Tarts rocketed in advance of hurricanes. Ever since, big retailers have been relying on algorithms to pitch their products. Amazon, iTunes and news websites "suggest" books, songs and articles based on our previous browsing habits.
And each time we click on their suggestions, we surrender a bit of our own free will. Serendipity is suppressed and our world becomes a bit smaller. We allow the data to define us.
Data analysts now vet Hollywood scripts to ensure they contain the elements of box office success. The likely result is even more formulaic films as personal creativity is replaced by soulless algorithms. Jaron Lanier, author of Who Owns the Future?, argues that this has already happened in the music business.
He sees an even more sinister development in the ability of Facebook and Google to suck up our personal data for free and then use it to sell us stuff. Yet, most people don't seem to care about this exploitation as long as they can "friend" and "be friended," blissfully blind to the fact that social media entrap as much as empower.
The good news is that not only profiteers are benefiting from the big data revolution. It's changing how doctors define and treat diseases, breaking down the walls between medical disciplines to improve patient care. Where most doctors still diagnose and treat brain disorders such as depression or autism based on often highly subjective symptoms, scientists are increasingly building huge data sets that render such narrow diagnoses obsolete.
"The realization is that many of these diseases are not totally independent," explains Donald Stuss, president of the Ontario Brain Institute. "Why do some people with autism have ADHD or OCD and others do not? Where do the boundaries cross?"
Dr. Stuss is the brain behind a year-old OBI initiative to collect a dizzying array of data on thousands of Ontarians suffering from everything from Alzheimer's disease to ADHD in order to home in on commonalities, genetic or otherwise, among patients diagnosed with supposedly unrelated conditions. The data, stored in a supercomputer at Queen's University, could help explain why some brain disorders progress differently in different people. It could lead to the development of highly idiosyncratic treatments and a shift away from mass market drugs to those targeted at tiny subsets of patients.
The goal is to lure researchers from around the world to conduct clinical trials in Ontario based on this treasure trove of data, which Dr. Stuss argues can only be collected in a single-payer health system. It might not have been possible had Premier Kathleen Wynne not recently stepped up to commit $100-million to the OBI, despite a tight provincial budget.
The wisdom of the OBI's approach, and the province's investment, is revealed in the current spat over the Diagnostic and Statistical Manual of Mental Disorders, the bible of U.S. psychiatry. Thomas Insel, head of the prestigious National Institute of Mental Health, is at odds with the DSM task force, which will release the manual's fifth edition this month. Dr. Insel argues that the DSM's approach of treating mental illnesses based on symptoms does patients a disservice.
"People think that everything has to match DSM criteria," he told The New York Times this week. "But you know what? Biology never read that book."
In a blog post, Dr. Insel announced that his organization, which awards $1.5-billion (U.S.) in federal research grants annually, will shift its research away from DSM categories. "We need to begin collecting the genetic, imaging, physiologic and cognitive data to see how all the data – not just the symptoms – cluster, and how these clusters relate to treatment response."
Dr. Stuss, a distinguished neuroscientist, figured that out some time ago. At 71, he is now at the forefront of a profession from which he was set to retire before big data reeled him back in.
Konrad Yakabuski