Image Bias and Misrepresentation Evaluation

Scratch Pad:

• GenAI Images are a representation of a represented work
• Generate Multimodal learning content based on the curricular frameworks personalized to the location of the learners and semiotically analyze the content?

AJ meeting:

• AI images more prevalent than AI generated text in social media
• VLMs - Generating captions
• Multimodal auditing
  • Rutgers

Questions:

• Does the faces look the same?
• According Panofsky's Theory, “The knowledge required for iconographic analysis is built on the practical knowledge that enables one to recognize factual and expressional meaning. It might be called "educated knowledge" as it requires its possessor to have a "familiarity with specific themes and concepts as transmitted through literary sources, whether ac quired by purposeful reading or by oral transmission."15 Panofsky is obviously using "literary sources" in a very broad sense, appearing to equate literary sources with any form of linguistic communication.” (Carter and Baughman, p. 44)
  • On a similar tangent, for the AI technologies to be able to conduct this analysis, they need to have this pre-requisite knowledge or cultural understanding. Do they have them?

Scope of evaluation

• Evaluation theme
  • World Important Events

Methods

• Computation of Cultural Bias Score and Historical Misconception Score
  • A framework for evaluating cultural bias and historical misconceptions in LLMs outputs
• • Framework for analyzing visuals

Taxonomy

• Misalignment Types
  • Bu, Fan, Zheng Wang, Siyi Wang, and Ziyao Liu. 2025. “An Investigation into Value Misalignment in LLM-Generated Texts for Cultural Heritage.” IEEE Transactions on Emerging Topics in Computational Intelligence, 1–15. https://doi.org/10.1109/TETCI.2025.3597289.
  • Types of Misalignments
    • Detail Inaccuracy
    • Cultural misunderstanding
    • Knowledge gap
    • Premature certainty
    • Cultural reductionism
    • Historical bias
• Misrepresentations taxonomy
  • Cultural Misrepresentations in Longform Texts
    • Lehengas in Schools? Evaluating the Cultural Representation of AI-Generated Stories in the Indian Context. n.d.
    • Types of misrepresentations
      • Cultural Inaccuracies
      • Unlikely Scenarios
      • Cliches
      • Oversimplifications
      • Factual Errors
      • Linguistic Errors
      • Logical Errors
• Flaws of Image-Based AI-Generated Content
  • Vasir, Gursimran, and Jina Huh-Yoo. 2025. “Characterizing the Flaws of Image-Based AI-Generated Content.” Proceedings of the Extended Abstracts of the CHI Conference on Human Factors in Computing Systems (New York, NY, USA), CHI EA ’25, April 25, 1–7. https://doi.org/10.1145/3706599.3720004.
  • Flaws
    • Logical Fallacy
      • Not Summing to a Whole
      • Redundancy
      • Contradiction
    • AI Surrealism
      • Unfinished
      • Over Detail
      • Smoothness
    • Misinformation
      • Factually Untrue
      • Improbable
    • Cultural Bias
      • Stereotypes
      • Lack of Cultural Understanding
• Culture Taxonomy
  • Liu, Chen Cecilia, Iryna Gurevych, and Anna Korhonen. 2025. “Culturally Aware and Adapted NLP: A Taxonomy and a Survey of the State of the Art.” Transactions of the Association for Computational Linguistics 13 (July): 652–89. https://doi.org/10.1162/tacl_a_00760.
  • Taxonomy for Culture in NLP
    • Ideational
      • Concepts (basic units of meaning that structure and facilitate thought, bridging sensory experience) (ex: cuisines, holidays)
      • Knowledge (Information acquired through education or practical experience)
      • Values (Bias, Perception of Hate Speech, Beauty Standards)
      • Norms and morals (Normative, Descriptive)
      • Artifacts (Books, Memes, Art)
    • Linguistic elements
      • Dialects (regionolects, sociolects, etc)
      • Styles, Registers, Genres (Formality, Politeness, Slang)
    • Social Elements: focus on social interactions and communication among humans
      • Relationship (Family, Fictive)
      • Context (Situational, Historical)
      • Communicative goals (Greeting, Requesting, Rejection)
        • Different communication style based on communicative goals
      • Demographics (Age, Income, Education)
• Framework for Cultural Awareness
  • Hershcovich, Daniel, Stella Frank, Heather Lent, et al. 2022. “Challenges and Strategies in Cross-Cultural NLP.” Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 6997–7013. https://doi.org/10.18653/v1/2022.acl-long.482.
  • Framework for cultural awareness
    • Linguistic form and Style (how things are expressed in language)
    • Common Ground (the shared knowledge based on which people reason and communicate)
      • Conceptualization
      • Commonsense Knowledge
    • Aboutness (what people care to convey)
    • Objectives or Values (the goals people strive for)
• Proxies of Culture
  • Adilazuarda, Muhammad Farid, Sagnik Mukherjee, Pradhyumna Lavania, et al. 2024. “Towards Measuring and Modeling ‘Culture’ in LLMs: A Survey.” arXiv:2403.15412. Preprint, arXiv, September 4. https://doi.org/10.48550/arXiv.2403.15412.
  • Proxies of Culture
    • Demographic proxies
      • Ethnicity
      • Education
      • Religion
      • Race
      • Gender
      • Language
      • Region
    • Semantic Proxies
      • Emotions and Values
      • Food and Drink
      • Social and Political Relations
      • Basic Actions and Technology
      • Names
• Factual Errors in LLM-generated Content
  • Wang, Cunxiang, Xiaoze Liu, Yuanhao Yue, et al. 2026. “Survey on Factuality in Large Language Models.” ACM Computing Surveys 58 (1): 1–37. https://doi.org/10.1145/3742420.
  • Causes of Factual Errors
    • Model-level causes
      • Domain Knowledge Deficit
      • Outdated Information
      • Immemorization
      • Forgetting
      • Reasoning Failure
    • Retrieval-level Causes
      • Insufficient Information
      • Misinformation Not Recognized by LLMs
      • Distracting Information
      • Misinterpretation of Related Information
    • Inference-level Causes
      • Snowballing
      • Erroneous Decoding
      • Exposure Bias

keywords:
image primitives - basic contents of the image
logical attributes