Click here to flash read.
Cheap-to-Build Very Large-Language Models (CtB-LLMs) with affordable training
are emerging as the next big revolution in natural language processing and
understanding. These CtB-LLMs are democratizing access to trainable Very
Large-Language Models (VLLMs) and, thus, may represent the building blocks of
many NLP systems solving downstream tasks. Hence, a little or a large bias in
CtB-LLMs may cause huge harm. In this paper, we performed a large investigation
of the bias of three families of CtB-LLMs, and we showed that debiasing
techniques are effective and usable. Indeed, according to current tests, the
LLaMA and the OPT families have an important bias in gender, race, religion,
and profession. In contrast to the analysis for other LLMs, we discovered that
bias depends not on the number of parameters but on the perplexity. Finally,
the debiasing of OPT using LoRA reduces bias up to 4.12 points in the
normalized stereotype score.
No creative common's license