Three experts, three DAGs

A curriculum committee wants to know what changes to the high-school maths sequence would do to college readiness. The first move is a causal DAG — nodes for the courses, edges for direct influence, the standard Pearl machinery. To get the DAG, three subject-matter experts are asked to draw one each, in isolation. They produce three different graphs.

Curriculum director Algebra I Algebra II English 11 Physics Readiness Maths department head Algebra I Algebra II Alg. fluency Pre-Calc Physics Readiness College counsellor Algebra I Algebra II English 11 US History Physics Readiness US History
Three DAGs of the same domain, three different shapes. Differences in yellow (maths head splits out algebraic fluency) and pink (counsellor draws an edge from US History, dashed).

The disagreements are not noise. The maths head has split algebraic fluency out of Algebra II as a separate node, claiming it influences Pre-Calculus and Physics independently of the course grade. The counsellor draws an edge from US History to College Readiness, citing what admissions officers say they look for. The director's DAG is the smallest.

The reconciliation

Each disagreement gets the same treatment: ask the expert who drew the edge to describe a scenario where it would visibly matter. Three outcomes are possible.

  • The edge is redundant. The lone expert describes a consequence the others would predict anyway from the rest of the DAG. Drop the edge. (The counsellor's US History edge resolves this way: English 11 and Pre-Calculus already absorb the signal.)
  • The disagreement reveals a hidden variable. The dissenting expert is pointing at something the others collapsed. Add the variable. (The maths head's algebraic fluency is split out as a node; both other DAGs are extended.)
  • The disagreement is testable and unresolved. The experts describe consequences the data could distinguish, but the data is not yet in hand. The edge stays out of the model. The disagreement goes into the register.

Reconciliation in the Pearl tradition is structural, not majoritarian. There is no voting on edges. Each disagreement is processed by asking what observation would change someone's mind — and the answer either resolves the disagreement or records why it cannot yet be resolved.

The reconciled DAG Algebra I Algebra II Alg. fluency Pre-Calc Physics English 11 Readiness
The reconciled DAG. Algebraic fluency (yellow) survives as a latent node from the maths head’s claim. The US-History-to-Readiness edge is dropped — English 11 absorbs the admissions signal. The Chemistry/Physics direction question and the question of whether self-regulation is a latent common cause both go to the disagreement register.

The disagreement register

The published model is the reconciled DAG. The disagreement register is what the model carries with it: a written list of each unresolved disagreement, what each expert believes, and what evidence would settle it. When new evidence arrives — a cohort of students taking a new sequence, a curriculum reform, a natural experiment — you check it against the register before updating the model.

Most elicitation exercises skip this step. The disagreements get averaged into a parameter, or one expert's view becomes the model and the others are forgotten. Both moves destroy information that the model could otherwise carry forward.

For the curriculum example, the reconciled DAG is published. The disagreement register contains the items not yet resolved — a Chemistry/Physics direction question the cohort data can settle within a year, the open question of whether self-regulation as a latent variable improves the fit enough to justify adding it. Both stay live. The model gets updated when the evidence does.

Pearl, J., 2009, Causality: Models, Reasoning, and Inference (2nd ed.), Cambridge University Press · Morgan, M.G. & Henrion, M., 1990, Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis, Cambridge University Press · Howard, R.A. & Abbas, A.E., 2015, Foundations of Decision Analysis, Pearson.