Mak, Moon-Kuen, and Tiejian Luo. 2025. “A Framework for Evaluating Cultural Bias and Historical Misconceptions in LLMs Outputs.” BenchCouncil Transactions on Benchmarks, Standards and Evaluations 5 (3): 100235. https://doi.org/10.1016/j.tbench.2025.100235.

Notes

• Check if the LLMs are reproducing prevailing historical inaccuracies
  • Like the inaccuracies that were debunked later
• Amplifying existing dominant voices (?)
• World Important Events Vs Local Important Events
  • Hypothesis: AI Models might be good at world information but they are lacking in local informationC

In-text annotations

"Cultural Bias Score (CBS) and the Historical Misconception Score (HMS)" (Page 1)

"present a structured evaluation framework for systematically identifying and measuring cultural bias and historical distortion in LLM outputs." (Page 2)

"Given that LLMs are trained on historical texts shaped by these epistemic imbalances, their outputs risk reproducing entrenched biases, necessitating systematic evaluation frameworks" (Page 2)

"Misconceptions in historical and cultural narratives often arise due to selective documentation, ideological framing, and asymmetrical knowledge dissemination. Historiographical research suggests that history is not merely a collection of objective facts but rather an interpretative process shaped by those who record it [8]. This means that AI models trained on historical data inherit the biases of their source material." (Page 2)

"Models disproportionately trained on English-language sources tend to reinforce Western perspectives while neglecting non-Western historiographical traditions" (Page 2)

"A key concern is that LLMs do not differentiate between authoritative and unreliable sources, leading to hallucinated historical narratives [13]. Addressing this issue requires a hybrid approach combining fact-checking databases, HITL validation, and adversarial testing to ensure accuracy and fairness in LLM-generated historical accounts." (Page 2)

"Specifically, we use a carefully selected subset of 100 historical events from the World Important Events (WIE) dataset to explicitly demonstrate the robustness, feasibility, and effectiveness of our Cultural Bias Score (CBS) and Historical Misconception Score (HMS) metrics" (Page 5)