Department of Education, Skills and Employment

Cognitive Interviewing

Ensuring meaningful survey results with cognitive interviewing

Wallis Social Research was commissioned by the Department of Education, Skills and Employment (the Department) to undertake testing of questions for a new job seeker survey using a cognitive interviewing methodology.

The overall objective was to provide the Department with advice and recommendations that ensured questions in the survey were clear and easy to understand for as many job seekers as possible.

Client feedback

“This project has been really informative, it’s helped improve the survey a lot and has built on our survey design skills more generally too. [Wallis] have been really great to work with – everything ran very smoothly from our perspective which we really appreciated. So a big thank you for that!”

Cognitive interviewing

Cognitive interviewing is an invaluable tool to explore respondents’ comprehension and interpretation of survey questions and provide assurance that a survey will deliver meaningful results.

During cognitive interviewing, Wallis’ experienced consultants utilise cognitive interviewing frameworks to observe and understand behaviours and thought processes of a participant when interpreting and responding to a question.

Where cognitive interviewing identifies a problem, the survey content can be revised to ensure survey questions are interpreted as intended.

Process

Initial pre-test refinement of the survey instrument

An initial refinement of the questionnaire was conducted based on Wallis’ experience conducting research with job seekers, together with data the Department had from their own quantitative testing of the initial questionnaire.

Recruitment

Recruitment was managed by Wallis’ qualitative fieldwork coordinator and utilised a rigorous recruitment and reminder process. A combination of phone calls, email and SMS ensured that once participants were booked into the research, they had all the information required to participate.

Phase one of cognitive testing

The first phase of cognitive testing consisted of ten cognitive interviews with a diverse range of job seekers. Testing was carried out across both CATI and online modes to test for the different ways questions were interpreted whether read or spoken.

Following the completion of round one, a report was provided with recommended changes and refinements to the questionnaire.

Phase two of cognitive testing

Following consultation with the Department regarding changes to the questionnaire, an updated questionnaire was prepared, and a second phase of five more cognitive tests was carried out. These interviews allowed for further testing of existing concepts, but with particular attention paid to the testing of changes made to the survey instrument after phase one.

Final report

Following the second phase of cognitive testing, a final report covering the entirety of the exercise was prepared, with a focus on action-centred recommendations for the final survey instrument.

Have a similar project? Get in touch.