An ecosystem approach to the evaluation and impact analysis of heterogeneous preventive and/or early interventions projects for veterans and first responders in seven countries

Abstract

Background Cumulative exposure to critical incidents and life-threatening events leads to significant risk for Veterans and First Responders (VFRs) developing mental ill health and disorders. Philanthropic organisations, Movember and Distinguished Gentleman's Ride, funded 15 organisations to conduct early intervention Projects across seven countries. The Projects aim to improve the mental health and wellbeing of VFRs, their families/significant others. This paper describes the novel external evaluation and impact analysis methods to identify effective Projects having positive impact on VFRs and their families, provide return on investment and the overall Grant Program. Methods We take an ecosystem real-world approach, which recognises the context and aims to manage the complexities involved, uses a complexity and systems perspective, multi-step mixed methods and approaches. The evaluation is from three perspectives of: Projects; Project comparisons; Grant program. Embedded in the evaluation design are methods, knowledge sharing and organisational learning activities for all stakeholders. Data is collected by the Projects and evaluation team on input, throughputs, and output indicators. Analysis tools include Global Impact Analytics Framework, multi-layered statistical analysis, performance evaluation using an efficient decision support approach, Project and Grant program social return on investment, visual linking and data connection platform and assessment of gendered lens approaches. Implications The complexity and heterogeneity of Projects implemented in the real world continues to present significant evaluation challenges and limitations for project leads, stakeholders, researchers and evaluators. Our ecosystem approach and novel evaluation methodology will reduce the uncertainty around real world implementation, provide key learnings for project stakeholders and more broadly implementation researchers.