Actionable Auditing: Investigating the Impact of Publicly Naming Biased Performance Results of Commercial AI Products

Inioluwa Deborah Raji and Joy Buolamwini follow Gender Shades with an August 2018 re-evaluation on the Pilot Parliaments Benchmark. They report reduced subgroup disparities for the three targeted companies and examine disclosure and corporate responses.^[1]

What does improvement establish?

The design discussion warns that performance on this public benchmark may not transfer to other datasets. Training changes and new data composition remain unclear. These observations do not independently isolate the causal effect of naming companies or establish current product performance.^[1]

An original analogy: practicing a familiar obstacle course can improve your time there without demonstrating the same improvement on an unfamiliar course. A follow-up measurement asks whether performance changed; a transfer test asks where that improvement holds.

Historical context

MIT’s repository dates the paper to 2019; chronology uses year precision. Selected audit design, follow-up methods and design limitations checked; rendered PDF p.6 inspected. Underlying API runs, data and full corporate communications were not independently audited.^[1]