Download the full Evaluation Guide as a pdf documentAnnex 3
Your Health, Your Care, Your Say - a case example of evaluation1 / 2 / 3 / 4 / 5 / 6 / 7 / 8 / 9
Doing the evaluation - how to do it?
- Scoping the evaluation:
- The overall purpose of the evaluation was “to determine the success of the
listening exercise at meeting the objectives specified at its outset”. In addition,
the evaluation was to consider:
- how success had been achieved (whether specific components were particularly important)
- whether any failures were intrinsic to the approach adopted or contingent on the manner of its implementation, and
- whether any opportunities were missed.
- a learning approach was the key focus, although it proved equally important to test the extent to which various targets had been met (the audit approach), such as ensuring a representative cross-section of the public and inclusion of ‘seldom heard’ groups
- a cross-government advisory group was established, chaired by the Department of Constitutional Affairs to reinforce the independence of the evaluation, which advised on the key issues to be addressed, the evaluation research process and provided feedback on findings as they emerged. It was agreed that participants should only be involved in providing information and views
- it was agreed that a full report would be published, and quotes used from participants, but that interview transcripts and the identity of interviewees would remain confidential within the evaluation process.
- Collecting data. The main processes, which provided quantitative / statistical and
qualitative data, and quotes from participants, were:
- detailed analysis of the data that had been collected on questionnaires completed by participants at the events
- structured interviews with participants from all the main deliberative events (national and regional), organisers of devolved events, facilitators and process designers, policy makers in the Department of Health, stakeholders (from the stakeholder task forces working alongside the public engagement) and internal Department of Health organisers of the process; interviews were undertaken at the end of the engagement process, after the publication of the White Paper and the reconvened event, to get feedback on perceptions of impacts on policy
- observation at the main national deliberative events.
- Analysing data. Data was analysed against the aims and objectives of the engagement process and against the agreed criteria for success on those objectives.
- Testing findings. Findings were tested with the cross-government advisory group, and they also gave feedback on draft reports. In addition, a seminar on the YHYCYS initiative was set up by the Government Communications Unit at the Cabinet Office (who were represented on the advisory group). This gave valuable feedback on the findings at that stage and raised questions that could be addressed more fully in the design and research of the final stage of the evaluation.
- Report writing. The main report provides a detailed description of the process, the evaluation and its findings. A summary report was also produced and sent to all interviewees for the evaluation research. Both these reports were published on the Department of Health website in August 2006.
- After the evaluation. Articles and other written outputs are planned, including the possibility of wider guidance on deliberative public engagement by central government, drawing on the lessons of this evaluation.