Weapons of Math Destruction: How Big Data Increases Inequality and Threatens Democracy

Although this book focuses on math, models and algorithms, Cathy O'Neill manages to bypass all of the technical jargon and explain how these models work in plain English. O'Neill's key point are that many of these models, such as evaluating teacher performance by using test scores, software used to determine which resumes to look at for a job, and models designed to predict where crime will happen are unfair. She uses that term a lot, but she makes a compelling argument to back it up. Her main points are that certain models (but not all) have the ability to devastate and discriminate against a whole section of people, ad are currently doing so. The models that do this she calls Weapons of Math Destruction or WMDs. WMDs have three key components that make them dangerous, and without any one of them, they wouldn't be. These three characteristics are: Opacity on how the models makes determinations/what factors are included, Scale (only software that can affect a large number of people are WMDs), and lastly, these models have no feedback loop for errors. To clarify that last point, these models don't know when they've got it wrong. There is no way for it to know if a teacher it recommended get fired was actually a good teacher that eventually became principal at his or her new school and improved student success. There is no way for it to know if a person whose resume they weeded out would have taken the company into new directions. As she says on page 133, "The system has no inkling that it got one person, or even a thousand people entirely wrong." There simply isn't a way for the model to know that it's gotten it wrong, and rethink things.




O'Neill spends most of the book going into detail about many different models and their unfairness. Part of the problems is that in our society's effort to create models to predict behavior, we use proxies so that we can manipulate the numbers, but proxies can never represent the real thing because life is complicated and cannot be boiled down to a set of numbers. Often these proxies help misrepresent the truth and in the case of most of these models, the ones who suffer are often those who are poor because the data that can be gathered on them doesn't read well: credit scores, zip codes, education, etc. None of these data points however, can tell you if they're trying to put their life back together but circumstances have been cruel, and so a large population of underserved people are now underserved by software. As she says on page 204, "Big Data processes codify the past. They do not invent the future." When a algorithm reinforces the past, it reinforces all of the things that come along with it, including the racism and, the classism and spits that back out.



O'Neill only stops to tell us how to fix this problem in the conclusion. After promising for the whole book to talk about how to fix this problem, it's only in the last twenty pages that she gets down to her proposed solutions. While her solutions seem good, they didn't seem as well researched as the rest of the book, and I felt rushed between one solution to the next. The only solution that seemed like it might actually work was the one that the EU implemented, only because it had actually been done before. If O'Neill had spent as much time researching and writing about her solutions as she did on even a fourth of the book, it would be a much better book, but instead it feels a little one-sided, without much of an idea on how to change this and move forward, much like some of the models she discusses.

Full disclosure: I received an advance copy of this book from the publisher in exchange for an honest review.

Interesting study into the important of good data and ethical AI

The subject matter that the book deals with is becoming increasingly important to be aware of nowadays, and it includes many instances where the effects of WMDs are not readily apparent to the naked eye. BUT, the author's writing felt too tedious and hard to get through.
3.25/5

A pretty thought-provoking read, even if you're already familiar with the idea. She unearths so many ways that algorithms affect our lives, without us even realizing it, and her central thesis is strong & clear: These things can be made to help us, or to harm us, but they are not neutral. She advocates powerfully for transparency and oversight (typically in the form of auditing) in the process, which really would benefit all stakeholders. And she reminds us again & again that we need to constantly consider our true priorities. When we don't think carefully about it, we will often tend to value monetary gains over justice & fairness. But if we take the time to assess what we're doing, it's usually possible to alter that calculus, and use algorithms in ways that suit our true societal values.

Definitely recommend this.

A great introduction to malicious use of algorithms from an aouthor that worked in algorithmic predictions herself. I really enjoyed how the author doen't condem all algorithms and machine leaning but gives a nuanced picture of different uses and possible threats. Easy to read! It is very US-specific, therefore the examples given are not as relevant to non-US readers.