Cathy O’Neil’s Weapons of Math Destruction is a multi-award winning best seller that takes a close up look at how algorithms and AI are transforming decision making in all areas of society. Predictive algorithms and model-driven decision processes are becoming pervasive in education leading to unintended problems. What O’Neil calls WMDs: “weapons of math destruction” are algorithms that try to quantify traits like teachers ability, creditworthiness, and quality of schools, but often have harmful outcomes and reinforce inequality.
These algorithms are supposed to be impartial; blind to gender, race and privilege. They are often deployed with a thought at reducing bias, by taking the human element out, it should reduce the bias in decision making like college acceptance. But the truth is that technology reflects the prejudices and beliefs of the people creating it. Even with the best intentions, the data selected for a model’s algorithm can lead to a reinforced bias because historical information is being used to predict the future. If you choose our next president based on a training set of all our past presidents, there’s a good chance he would be white and practically a guarantee he would be male. This is why relying on “historical data of successful candidates” is so problematic. And because algorithm’s make decisions within a black box, it is hard to know why any specific decision was made. So once an algorithm becomes part of the system, it is very difficult to dislodge.
The book deals with specific case studies. Several of the studies highlight problems related to education, including the questionable algorithms used to evaluate teacher performance, complications introduced by the U.S. News and World Report college ranking system, and the issues surrounding for-profit colleges that derive most of their revenue from taxpayer-funded student financial aid.
Something I didn’t understand before reading this book is that the US News and World Report college ranking system is directly tied to the astronomically rising cost of college. It was infuriating to read that the original editors (not scientists or mathematicians) who put together the ranking model purposely left out cost as a variable because they knew it would drive the Ivy League universities lower in the ranking. They wanted the schools that everyone thought were best to come out on top so the ranking would be seen as accurate. In other words, they purposely used data that would twist the model to reinforce past prejudices about the best universities. They didn’t even have to leave it to the algorithm to do it for them. So this ranking, which drives student applications to higher rated colleges, completely ignores how much students are being charged, and puts high value on things like larger sports stadiums and state-of-the-art facilities. As a result, colleges and universities have focused disproportionately on high cost building projects to push their ranking higher and have passed the rising bill on to the students and their families.
There have been efforts to combat these rising costs, including President Obama’s College Scorecard but, if anything, US News and World Report college ranking system has risen in popularity.
The section on teacher evaluation looks at how even when you have all the right data in place, how you weigh each variable can throw the answers off and her chapter on for-profit colleges shows how once an algorithm is in use, unscrupulous people who understand can game the system and reap benefits at the expense of those less knowledgeable.
O’Neil has some ideas of how to solve these issues, including suggesting that predictive algorithms should be transparent, especially to those people affected by the algorithms. She also thinks the algorithms should account for new data as it becomes available, and be continually adjusted to account for anomalies and changes in data trends. She also believes algorithms should be viewed as only one tool among many to evaluate specific individuals. Whether any of these suggestions are ever implemented is questionable but we have to start somewhere.
Weapons of Math Destruction is an excellent and thought provoking book that I recommend highly. Cathy O’Neil also has a worthwhile blog that focuses on many of these issues.