Reporting Guide

This guide covers how to report results from causal inference analyses. It follows the nine-step causal workflow and demonstrates best practices for communicating average treatment effects, heterogeneous effects, and sensitivity analyses. This resource directly supports the research report assessment (40%).

The nine-step causal inference workflow

Before reporting results, ensure each step has been addressed.

Steps 1–3: problem definition

1. Well-defined treatment. Specify the exposure precisely, including the contrast (e.g., "weekly religious service attendance vs. less than weekly").
2. Well-defined outcome. State the outcome measure, its scale, and when it was assessed.
3. Target population. Define who the results apply to, including any weighting for population representativeness.

Steps 4–6: identification strategy

4. Exchangeability. Describe how conditional independence was achieved (e.g., "rich baseline covariate control including 32 covariates").
5. Consistency. Explain why the treatment is well-defined and uniform across individuals.
6. Positivity. Report verification through transition tables showing exposure variation across covariate strata.

Steps 7–9: implementation

7. Measurement validity. Note the psychometric properties of outcome scales.
8. Attrition handling. Describe how panel dropout was addressed (e.g., inverse probability of censoring weights).
9. Transparent reporting. Document all analytical decisions and assumptions.

Reporting average treatment effects

Standard ATE table format

Include these elements for each outcome:

Key reporting elements

1. Effect sizes: report in standard deviation units for continuous outcomes.
2. Confidence intervals: show uncertainty around each estimate.
3. E-values: indicate robustness to unmeasured confounding.
4. Sample size: total analysed after exclusions and weighting.

Example results text

"Weekly religious service attendance showed positive causal effects across all cooperation measures. The largest effects were observed for social outcomes: sense of belonging ($\beta =0.18$, 95% CI: 0.14–0.22) and social support ($\beta =0.15$, 95% CI: 0.11–0.19). All effects were robust to moderate unmeasured confounding (E-values > 1.6)."

Reporting heterogeneous treatment effects

RATE results summary

When heterogeneity is detected, report it systematically:

Example heterogeneity text

"We found substantial heterogeneity in treatment effects for social outcomes (RATE-Qini > 0.08, $p<0.05$), but limited heterogeneity for behavioural outcomes. This suggests that while religious service benefits most people socially, individual responses vary considerably in magnitude."

Reporting policy tree results

Present policy trees with both standardised and original-scale interpretations.

Subgroup identification with data-scale effects

High-response subgroups for charitable donations:

1. Older adults with high agreeableness (age > 45, agreeableness > +1 SD)

   • Standardised effect: $\beta =0.28$ (95% CI: 0.21–0.35)
   • Data-scale effect: NZ$847 additional annual donations (95% CI: NZ$635–1,058)
   • Sample proportion: 23%

2. Parents with medium conscientiousness (parent = yes, conscientiousness > 0)

   • Standardised effect: $\beta =0.22$ (95% CI: 0.16–0.28)
   • Data-scale effect: NZ$665 additional annual donations (95% CI: NZ$484–846)
   • Sample proportion: 31%

3. All others

   • Standardised effect: $\beta =0.08$ (95% CI: 0.04–0.12)
   • Data-scale effect: NZ$242 additional annual donations (95% CI: NZ$121–363)
   • Sample proportion: 46%

Example policy tree text

"Policy tree analysis identified two subgroups with enhanced treatment response for charitable donations. Older adults (45+) with high agreeableness showed the largest increase ($\beta =0.28$ SD, equivalent to NZ$847 additional annual donations), representing 23% of the sample. In practical terms, targeted interventions toward these subgroups could generate 2.8–3.5 times more charitable giving than population-wide approaches."

Sensitivity analysis: E-values

Interpretation

E-values quantify robustness to unmeasured confounding. The E-value is the minimum strength of association, on the risk ratio scale, that an unmeasured confounder would need with both the treatment and the outcome to explain away the observed effect.

Example sensitivity text

"To assess robustness to unmeasured confounding, we calculated E-values for all estimates. The observed effect on sense of belonging (E-value = 2.4) would require an unmeasured confounder associated with both religious service and belonging by a risk ratio of 2.4 each to explain away the result."

Methods section template

A complete methods section following the nine-step workflow should include:

1. Treatment definition: what the exposure is, how it is coded, and the contrast of interest.
2. Outcome definition: measures used, timing of assessment, any transformations applied.
3. Target population: sampling frame, weighting strategy, eligibility criteria.
4. Causal identification: covariates conditioned on, justification for conditional exchangeability.
5. Statistical analysis: estimation method, key tuning parameters (e.g., number of trees, minimum node size).
6. Attrition handling: censoring weights, stages of dropout addressed.
7. Heterogeneity assessment: RATE metrics, false discovery rate correction, policy tree depth.
8. Sensitivity analysis: E-values for all primary estimates.

Reporting checklist

Do report

• Both standardised and data-scale effects
• Effect sizes with confidence intervals
• Sample sizes after exclusions and weighting
• E-values for sensitivity analysis
• Clear practical interpretation of effect magnitudes
• Subgroup sizes and effect magnifications
• Target trial framework and causal question
• Explicit treatment and outcome definitions

Do not report

• Model coefficients without interpretation
• p-values alone without effect sizes
• Only standardised effects for policy-relevant outcomes
• Technical details that obscure main findings
• Causal claims beyond your identification strategy

Figure presentation

ATE plots

• Use forest plots with confidence intervals.
• Order by effect magnitude or E-value.
• Include sample sizes.

Policy tree plots

• Show decision rules clearly.
• Include sample proportions in each node.
• Provide plain-language interpretation alongside the tree.