FairBench: A Unified Benchmarking Framework for Evaluating Fairness, Reliability, and Ethical Compliance in AI-Powered Software Tools

The rapid proliferation of AI-powered software tools across high-stakes domains has created an urgent need for standardized methods to evaluate their fairness, reliability, and ethical compliance. Existing evaluation approaches are fragmented: fairness toolkits use inconsistent metrics, reliability assessments lack cross-tool comparability, and ethical compliance checks remain largely qualitative. This paper introduces FairBench, a unified benchmarking framework providing a structured suite of quantitative metrics, curated evaluation datasets, and a reproducible scoring protocol enabling systematic cross-tool comparison across three dimensions: statistical fairness, predictive reliability under distributional shift, and ethics principle alignment. We apply FairBench to evaluate six widely deployed AI fairness and ethics tools across four benchmark datasets spanning employment, lending, healthcare, and criminal justice domains. Results reveal significant performance divergence across tools on identical datasets, a reliability-fairness tension that existing tools handle inconsistently, and no single tool achieving satisfactory performance across all three benchmark dimensions. FairBench is released as an open-source framework to support reproducible evaluation and drive progress in trustworthy AIware.