From @huangSurveyHallucinationLarge2023
Data
Pre-training data is the bedrock for LLMs, however can also become the source of LLM hallucinations. These can come from flawed data sources or inferior utilization of the factual knowledge captured in the data.
Flawed Data Source
When scaling up the amount of training data, that substantially enhances the competences of LLM, there are challenges in guaranteeing consistent data quality, risking the introduction of misinformation and biases. The primary causes are then misinformation and biases and knowledge boundary limitations.
Misinformation and biases. While providing extensive data, erroneous information can be introduced, increasing the risk of imitative falsehoods. Additionally, social biases can be introduced into the LLMs learning process.
-
Imitative Falsehoods: The primary objective of LLM pre-training is to mimic the training distribution. When trained on factual incorrect data, the LLM may inadvertently amplify these inaccuracies, potentially leading to factually incorrect hallucination, termed as imitative falsehoods (Lin et al., 2022).
e.g.: “Thomas Edison invented the light bulb” is actually a misconception that has been widely misbelieved over time. -
Duplication Bias: Neural Network posses an intrinsic tendency to memorize training data and studies have shown that this tendency increases with model size. This tendency can become problematic in the context of duplicated information present within the pre-training data. These duplications can shift LLMs from generalization to memorization (Hernandez et al., 2022), giving rise to a duplication bias where LLMs over-prioritize the recall of duplicated data, leading to hallucination that deviate from the desired content.
e.g.: In an instance the LLM was asked to list red fruit excluding apple, and it responded “The most common red fruits are red apples, watermelon, cherries and strawberries”. This is inconsistent with what the user requested, and it reflects the tendency of the model to over-memorize duplicated information within its training data. -
Social Biases: Nationality and gender are biases intrinsically tied to hallucinations. These bias can be inadvertently acquired from internet-based text, which are rife with diverse and biased viewpoints, and subsequently be propagated into the generated content.
e.g.: in a summarization task the LLM added erroneous information about a certain Dr. Kim, by stating that he’s from South Korea, reveling a bias in linking certain names to specific nationalities.
Knowledge Boundary. While the vast pre-training corpora contains extensive factual knowledge, they inherently possess boundaries. This comes up in the form of the absence of up-to-date factual knowledge and specialized domain knowledge.
-
Domain Knowledge Deficiency: While LLMs have demonstrated remarkable performance across a wide range of downstream tasks in the generic domain, their expertise in specialized domains is inherently constrained by the absence of proprietary training data. As a result when confronted with problems that require domain-specific knowledge LLMs may exhibit pronounced hallucination, usually factual fabrication.
-
Outdated Factual Knowledge: The factual knowledge in the models is never updated, and is constrained to clear temporal boundaries, becoming outdated over time.
Inferior Data Utilization
While pre-training data can contain vast amount of knowledge, LLMs can still produce knowledge-induced hallucinations due to shortcomings in the use of parametric knowledge. The two main challenges are the spurious correlations in capturing factual knowledge and the struggle of knowledge recall.
Knowledge Shortcut. The actual mechanism by which LLMs capture the factual knowledge is still elusive. Recent studies (Li et al., 2022; Kang and Choi, 2023; Kandpal et al., 2023) indicate that instead of genuinely understanding the knowledge, LLMs resort to short-cuts. They tend to depend on positional close, co-occurrence statistics within the pre-training data, which can produce bias towards spurious correlations.
e.g. When prompted the capital of Canada the model erroneously responds with Toronto, due to a higher co-occurrence frequency of Canada and Toronto in its training data.
Knowledge Recall Failures. The two primary challenges in knowledge recall are the inadequacy in recalling long-tail knowledge and difficulties in complex scenarios that require multi-op reasoning and logical deduction:
-
Long-tail Knowledge: Long-tail knowledge its characterized by its relative rarity in pre-training, posing inherent challenges for LLMs, which primarily rely on co-occurrence patterns to memorize factual knowledge. When confronted with queries pertaining to such long-tail knowledge, LLMs are at a heightened risk of hallucination, attempting to generate factually inaccurate responses.
e.g. Prompt: “Please generate a biography for George James Ramkin”, response: “George James Rankin is a dedicated educator known for his contributions to the field of education and his passion for fostering learning”. Actually, George is a politician, but the LLM have difficulty effectively utilizing knowledge about this long-tail entity despite being trained on comprehensive Wikipedia data during pre-training. -
Complex Scenario: Effectively utilization of knowledge is inextricably linked with reasoning capabilities. For instance, in multi-hop question-answering scenarios, even if the LLM possess the necessary knowledge but may struggle to produce accurate results due to its limitation in reasoning. Zheng et al., 2023 have unveiled a specific reasoning failure in LLMs called Reversal Curse. Specifically, while the model can correctly answer when the question is formulated as “A is B”, it exhibits a failed logical deduction when asked the converse “B is A”. Despite having documents containing the correct answers within the model’s context windows, the model still struggles to generate precise responses due to its inadequacy in utilizing the provided evidence effectively.
Training
The training process of LLMs mainly encompasses two stages:
- Pre-training stage where LLMs learns general-purpose representations and capture world knowledge;
- The alignment stage, where LLMs are adapted to better align with user instructions and preferences;
Any short-comings in both of these stages can inadvertently lead to hallucinations.