Applying the critical appraisal worksheet to a published evaluation
Applying the critical appraisal worksheet to a published evaluation
A mixed methods evaluation (Sridharan et al, 2018) was performed of an online global public health capacity building programme (Heller et al, 2022). Applying the checklist to the published evaluation finds the checklist easy to use and that most of the criteria have been met, although there are a few gaps. The table below shows the items on the checklist and whether the evaluation met the identified criteria.
|
Item on Evaluation Based Practice framework |
Can you identify this in the evaluation? |
Did the evaluation meet the criteria? |
|
Define the purpose of the evaluation |
Asking the right question |
Yes, the purpose was defined appropriately for the context and framed the evaluation question |
|
Identifying mechanisms |
Yes, the mechanisms for change were clearly identified, however they probably underestimated the complexity of the programme |
|
|
Defining expected outcomes |
Yes, in relation to the perspective of the students, but not other stakeholders such as employers. Most but not all of the OECD criteria were covered, equity outcomes were not. |
|
|
Measure and understand process and outcome |
Collecting data |
Yes, data collection was well described and was appropriate |
|
Analysing data |
Yes, data analysis we appropriate |
|
|
Appraising the data |
Mixed. The methods were appropriate for the question and the theory of intervention, but resources, cost effectiveness, values and not all the OECD criteria were appraised. |
|
|
Interpreting the results |
Yes, answers have been produced to initial question, they are relevant and likely to influence practice. |
|
|
Synthesising the information |
Yes, synthesis into previous body of evidence is appropriate |
|
|
Learning from the evaluation |
Implementing the results |
Yes, there has been appropriate reflection, and the results are likely to be relevant to other similar programmes. The paper does not describe how the results have been communicated to the stakeholders. |
|
Exploring inequity |
No, socio-cultural diversity nor equity were explored. |
|
|
Reconsidering the purpose of the evaluation |
No, this was not discussed. |
|
|
Cycle of learning through feedback |
Adapting to changes during the evaluation |
No, there has not been a discussion about how the evaluation might have adapted to change. |
Please examine the report and see if you agree with the way the checklist has been completed.