FairSense tool catches AI bias early

Machine learning software helps agencies make important decisions, such as who gets a bank loan or what areas police should patrol. But if these systems have biases, even small ones, they can cause real harm. A specific group of people could be underrepresented in a training dataset, for example, and as the machine learning (ML) model learns that bias can multiply and lead to unfair outcomes, such as loan denials or higher risk scores in prescription management systems.

Researchers at Carnegie Mellon University's School of Computer Science (SCS) created FairSense to help developers address unfairness in ML systems before the harm occurs. Currently, most fairness checks look at a system at a specific point in time, but ML models learn, adapt and change. FairSense simulates these systems in their environments over long periods to time to measure unfairness.

"The key is to think about feedback loops," said Christian Kästner, an associate professor in the Software and Societal Systems Department (S3D). "You might have a tiny bias in the model, like a small discrimination against a gender or race. When it's deployed, the model produces an effect in the real world. It discriminates against people — they get fewer opportunities, less money or end up in jail more often. And then you train the system on the data influenced by that model, which might amplify the bias over time. So it might be small in the beginning, but because it has an effect in the real world and then the model learns from that again, it could become a vicious cycle where the bias grows."

In "FairSense: Long-Term Fairness Analysis of ML-Enabled Systems," SCS researchers explored how fairness changes as these ML systems are used over time. They focused on testing these systems in a dynamic environment rather than a static state.

To use FairSense, developers provide information about the machine learning system, a model of the environment it will be used in and the metric that indicates fairness. For example, in a bank, the system could be software that predicts applicants' creditworthiness and makes loan decisions. The environment model includes relevant information from the applicant's credit history and how credit scores might be affected, and the fairness metric could be the parity between different groups of people approved for loans.

Along with Kästner, the team included S3D's Yining She, a doctoral student, and Eunsuk Kang, an associate professor. Sumon Biswas from Case Western Reserve University also participated in the research, which the team presented earlier this year at the International Conference on Software Engineering.

"We simulate how the fairness might change over a long period of time after the system is deployed," She said. "If we observe an increase in unfairness over time, the next step is identifying the core factors affecting this fairness so the developer can address these issues proactively."

Since ML-enabled systems are deployed in varied and complex situations that aren't always predictable, FairSense can capture and simulate that uncertainty in the environment model. In loan lending, for example, credit score updates and new loan applicants are uncontrollable and could affect how the system behaves over time. FairSense's simulation generates a wide range of possible scenarios based on these variables and allows developers to identify factors, such as credit score thresholds or other parameters, that could have the most significant impact on long-term fairness issues.

"A lot of the software we build can negatively affect people," Kang said. "The systems we build have societal impact. The people who build these systems should be thinking about the issues that may arise over time, not just right now. When you build and deploy the system, what potential bad things could happen down the road? I hope that reading papers like this one will encourage software developers to think more broadly about potential harms caused by the systems they create and proactively address these types of issues before they are deployed into the real world."

Researchers plan to expand FairSense's work to continuously monitor the fairness of ML systems and to develop a tool to explain how these systems can become unfair.