babylonpolice.com

Biases with respect to socially-salient attributes of individuals have been
well documented in evaluation processes used in settings such as admissions and
hiring. We view such an evaluation process as a transformation of a
distribution of the true utility of an individual for a task to an observed
distribution and model it as a solution to a loss minimization problem subject
to an information constraint. Our model has two parameters that have been
identified as factors leading to biases: the resource-information trade-off
parameter in the information constraint and the risk-averseness parameter in
the loss function. We characterize the distributions that arise from our model
and study the effect of the parameters on the observed distribution. The
outputs of our model enrich the class of distributions that can be used to
capture variation across groups in the observed evaluations. We empirically
validate our model by fitting real-world datasets and use it to study the
effect of interventions in a downstream selection task. These results
contribute to an understanding of the emergence of bias in evaluation processes
and provide tools to guide the deployment of interventions to mitigate biases.