Click here to flash read.
In modern recommendation systems, the standard pipeline involves training
machine learning models on historical data to predict user behaviors and
improve recommendations continuously. However, these data training loops can
introduce interference in A/B tests, where data generated by control and
treatment algorithms, potentially with different distributions, are combined.
To address these challenges, we introduce a novel approach called weighted
training. This approach entails training a model to predict the probability of
each data point appearing in either the treatment or control data and
subsequently applying weighted losses during model training. We demonstrate
that this approach achieves the least variance among all estimators without
causing shifts in the training distributions. Through simulation studies, we
demonstrate the lower bias and variance of our approach compared to other
methods.
No creative common's license