'Weapons of Math Destruction': NYT best-seller Cathy O'Neil talks data privacy breaches

Penn Libraries hosted New York Times best-selling author and mathematician Cathy O'Neil to celebrate Data Privacy Day, a day to promote and raise awareness of privacy and data protection.

Many people, particularly those who are mentally ill or racial minorities, are unaware that their data may be used by employers and the criminal justice system to discriminate against them, O'Neil said at the talk held at Penn Museum's Harrison Auditorium on Tuesday evening.

O'Neil added that because hirers consider medical and criminal records and a person's address when conducting evaluations, hiring algorithms and criminal profiling systems are susceptible to biases. She said that data collection tactics, which include using a person's health records, are an unconstitutional violation of privacy, because they use subjective criteria to assess individuals.

O'Neil described workplace personality tests administered by employers as a “weapon of math destruction," used to reject job applicants who fail the test according to their potential employer. O’Neil referred to Kyle Behm, a Vanderbilt student with bipolar disorder, who was unable to get an interview for a part-time job after being rejected based on the results of his personality tests. O’Neil said the tests are used by 60% of employers in the United States.

“More and more of our life choices are being defined algorithmically, being predicted algorithmically, and being decided algorithmically, by people hiding behind the authority of mathematics who are in charge,” O’Neil said. “They’re in power, and choose the definition of success that works for their agenda, not for your agenda.”

O’Neil added that increased racial profiling is a consequence of “weapon of math destruction” algorithms. She presented the methodology of the crime rate score used within the criminal justice system. The crime rate score decides the length of an individual’s prison sentence based on their mental health records, gang affiliations, addiction history, and neighborhood of origin, O'Neil said.

“When determining someone’s crime rate score, ‘success’ within the algorithm means that they will be re-arrested within two years of leaving prison,” O'Neil said. “Whoever is in power gets to choose what ‘success’ looks like."

After earning her Ph.D. at Harvard University in mathematics, O’Neil founded O'Neil Risk Consulting & Algorithmic Auditing, an algorithmic auditing company that strives to hold companies accountable for using data to cut job applicants. She also wrote The New York Times bestselling book, "Weapons of Math Destruction," which was assigned to all incoming members of the Class of 2023 last summer as part of the Penn Reading Project.

O'Neil spoke for approximately 30 minutes and subsequently answered questions from the audience.

Engineering sophomore Katherine Hann, who attended the event, said she was concerned by the lack of transparency in data privacy. 

“This is our warning that there could be algorithms that are working against us, and we need to be aware that there could be some bias against us based on something that doesn’t necessarily relate to the actual topic that the algorithm is judging us on,” Hann said. 

Hann added that unethical data privacy practices are a frustrating problem for others in the engineering field. 

College junior and attendee Victor Diaz expressed concern about how data privacy violations could be resolved.

“I thought it was nuts, but also very poignant because it’s like, okay, we have to move on and work with what we actually have," Diaz said. "It’s kind of like a call-to-action in that sense.” 

O'Neil said that privacy, including medical information, should be legally protected.

“We cannot leave this to the experts, because the experts do not have our best interests in mind,” O'Neil said.