The Decision
A financial services firm has invested £2.8M annually in a management development program. Participation is high. Post-program surveys are positive. Promotion rates among participants are above average. The HR team presents these as evidence that the program works.
The board asks the right question: would the participants have been promoted anyway? The survey measures satisfaction, not learning. The promotion rate comparison has no control group — high-potential employees are disproportionately nominated for the program, confounding the comparison. None of the current metrics answers whether the program causes career progression — or whether it simply enrolls people who were already on that trajectory.
The Causal Structure
The causal graph encodes four variables that affect promotion probability: baseline performance rating, management visibility (exposure to senior stakeholders), structured skill development, and tenure. The program affects two of these: management visibility (through cohort events) and structured skill development. Baseline performance rating is the confounding variable — it causes both program nomination and subsequent promotion. Without adjusting for it, the program’s apparent effect is inflated by the selection mechanism.
The graph also distinguishes two pathways for program effect: skill development → performance improvement → promotion, and visibility → sponsor relationship → promotion. These are causally distinct interventions with different implications for program design.
What the Model Answers
Attribution: After adjusting for baseline performance (the confounder), the program’s causal effect on promotion probability is 8 percentage points — compared to the unadjusted apparent effect of 23pp. The majority of the apparent effect was selection bias. The genuine effect is real but substantially smaller than reported.
Mechanism decomposition: Of the genuine 8pp effect, 3pp flows through skill development and 5pp through management visibility. The visibility pathway is stronger — which implies that the program’s causal mechanism is primarily network access, not capability development. This has direct implications for program design.
Heterogeneity: The 8pp average effect masks significant variation. Participants from underrepresented groups show a 4pp effect; those with existing senior sponsor relationships show 2pp; first-time managers without existing visibility show 14pp. The program’s impact is concentrated in those who needed visibility most — and diluted by participants who already had it.
The Result
Program redesigned to prioritize visibility-building for employees without existing sponsor relationships. Selection criteria revised to reduce high-performer bias and increase reach to the subgroup with highest expected uplift. The same budget now produces an estimated 11pp causal effect on the highest-value target group — at the same cost.
If your training program effectiveness is measured by completion rates and satisfaction scores, it is not measured. Thirty minutes to map the causal structure.