Survey Analysis Tips

Tips shared by my labmates

Survey Implementation and Design Best Practices

  1. Pilot Survey:

    • Conduct a pilot survey before the main implementation to test the survey's effectiveness.
    • Ensure that participants' understanding aligns with the survey questions to avoid misinterpretation.

  2. Online Survey Considerations:

    • Be cautious of bots that may skew results, particularly those generating responses using language models (LLMs).
    • Consider the trade-offs between using attention checks and maintaining response quality.

  3. Response Quality:

    • Address concerns about response quality, which often arise in reviewer comments.
    • Evaluate how the sample of survey respondents relates to the research questions.
    • Ensure that the sample is representative and generalizable to the target population.

  4. Making Survey-based Research Studies more interesting:

    • Incorporate experiments into the study often makes the studies more interesting as we could try connecting aspects of the intervention with the constructs in the survey.

  5. Survey Design Intent:

    • Surveys can be designed to elicit specific responses that researchers may prefer.
      • In some disciplines, it's acceptable and not so in some.
    • Clearly define the variable that the survey aims to measure and ensure the theoretical model is validated.

  6. Scale Validation:

    • Differentiate between using validated scales and adapting or creating new ones, as reactions may vary across research communities.
    • Understand the concepts of validity, including:
      • Construct Validity: Ensures the survey measures the intended construct.
      • Predictive Validity: Assesses how well the survey predicts outcomes.
      • Cronbach's Alpha: A measure of internal consistency to determine if a set of questions measures the same construct.

  7. Language and Bias:

    • Manage leading language in survey questions to avoid biasing responses.
    • Consider adding a negative question in Likert-based instruments as a check to validate responses.
      • It can also be annoying for some respondents.

  8. Survey Research in HCI:

    • Refer to "Survey Research in HCI" by @mullerSurveyResearchHCI2014 for insights on mediation models, which explore how different elements are interconnected.
    • Investigate how various factors influence teachers and justify the importance of each survey question.

  9. Pre-Post Surveys:

    • It is acceptable to modify post-survey questions based on insights gained from the pilot.
    • Ways to calculate difference scores:
    • (Post - Pre) ~ T
    • Post ~ (T + Pre)

  10. Coding Open-Ended Responses:

    • Check out Ryan Baker's work for coding qualitative responses effectively.

  11. Advanced Modeling:

    • Structural Equation Models (SEMs) are advanced regression models that can analyze complex relationships between variables. It has it's own limitations and can be too complex at times.

  12. Additional Resources: