factors that affect validity and reliability in qualitative research

Therefore, this study is missing internal validity. They will also know what you expect them to do and how they’re supposed to perform the task. This study setup favors design B because, when they get to it, participants will be already used to the testing situation and with the task domain - if they’re testing car-rental sites, they will already know what a LDW (loss-damage waver) is when they get to site B and they may have certain expectations regarding the steps of the rental process. Is this study design likely to produce accurate results, that reflect the reality? In other words, will this study identify the better design? You are trying to decide which of the two is better and you always show the participants in your study design A first, ask them to complete some tasks on it, then move to design B and show them the same tasks.

Think of a study that compares two sites - site A and site B. But reliability is no good without validity: a study with high reliability and low validity is one where you get a really good measurement of the wrong thing. There are plenty of statistical methods to calculate the degree of study reliability, and the main way to increase reliability is to test more participants. Reliability of a study simply means that you will get the same result if you repeat the study. But before we do, let’s note that validity is separate from reliability. We’ll talk about each of these separately.

External-validity errors that capture behaviors or situations which are not characteristic for our target audience.

Internal-validity errors that bias participants towards a certain response or behavior.

There are two big types of study-design errors:

What percentage of our user population will be able to subscribe to our newsletter? What major usability issues will people encounter on our site? Is design A more usable than design B for our target audience? But any time we set up a UX-research study, whether quantitative or qualitative, there is danger that it will not reflect the reality we want to capture because the study is poorly designed. Any UX-research study aims to answer general questions about our design or about our users.

Factors that affect validity and reliability in qualitative research