Folks ask me about the dangers of trusting computer-generated algorithms and artificial intelligence. The conversation usually brings up a future scenario in which the machines outsmart humans.

But there’s a more current problem: we trust machines to build algorithms based on incomplete or biased data that we feed them, and they perpetuate poor and unfounded decisions under the guise of ‘scientificness’ because a computer made the decision.

Cathy O’Neil’s book Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy does an excellent job of explaining the case for this, with plenty of real-world examples from algorithms that whir away on our credit scores, college prospects, and political leanings. 

I like to keep short collections of notes on the books I read to help me synthesize ideas and jog my memory later. Here are my one-page notes on WMD:

The book avoids the fearmongering approach of labeling all algorithms dangerous, which I appreciate. Instead, O’Neil explains what makes an algorithm dangerous—namely its opacity, scale, and impact. When we fail to understand how our models are making decisions and we fail to measure or improve their outcomes, we propagate their simplified version of reality—to the detriment of many, usually marginalized, people in our more complex actual reality.

While the book does an excellent job of defining, explaining, and exemplifying the problem, it fails to address how we might solve the problem. A few solution ideas make it into a brief set of paragraphs in the ‘Conclusion’ chapter. Those solutions, as you can see from the visual notes, amount to asking data scientists to take an oath and imposing stricter regulations on the way companies collect and use data.

I want to see more. I want to see guidance on how to make the first step toward implementing these solutions. I want to see a specific call to action, to a specific party, with a specific timeline. I want to know what we should be doing, and what data scientists are doing, to put these oaths and regulations in place. Are they lobbying their representatives or coalescing around some industry standards?  I understand that this discussion may be outside the purview if this particular book. I’d like to see it somewhere—and an experienced data scientist like O’Neil seems well-placed to take a crack at it. Here is her website, where she might presumably provide such guidance in the future.

Even with regulations in place, a purely regulatory solution assumes that industry is keen on following the regulations. We have seen that this is not the case. O’Neil explains early in the book how the 2008 crash prompted government to require audits for financial institutions from outside parties. O’Neil also explains how financial institutions hold those audits at arms’ length and exercise their power as consumers to demand good ratings from the auditors. Regulations aren’t a bad idea, but they aren’t always airtight, and they aren’t always followed, and they change at the whim of frequently-changing political administrations.

So this begs the question: is there a way to change the incentive structure such that companies want to use data fairly?

WMDs survive because they offer a black-box scapegoat tool for boosting profits. The profits are a visible, measurable incentive. So how do we advertise and measure the advantages of frequently-tested, feedback-driven, fair algorithms?

I don’t have the answer to this question, and I’m not asking one book to have it, either. But I’m eager to know what sorts of answers have been proposed and which organizations are working on those. There’s an opportunity for disruption here—and not just ‘Uber disruption,’ in which a business model redistributes wealth among the already wealthy. Instead, this constitutes an opportunity to upend the opportunity gap across demographic strata and create wealth. How we shift attention to that opportunity could shape the direction of our algorithms of the future.