Markets matter: a simulation study of the bias-variance trade-off in comparison group selection for difference-in-differences analysis.
Recent research reveals many pitfalls when selecting a comparison group for difference-in-differences analyses of observational data. Recommendations from this research, although important, can be difficult to apply to a complex, real-world evaluation. In this study we explore the potential value of...
| Publicado en: | Health Services & Outcomes Research Methodology Vol. 25; no. 2; pp. 166 - 182 |
|---|---|
| Autores principales: | , , , , |
| Formato: | equations & formulas research tables/charts Journal Article |
| Publicado: |
Springer Nature
Jun2025
|
| Acceso en línea: | Ver este registro en EBSCOhost |
| fields | @attributes: recordID: 1 pdfLink: plink: https://search.ebscohost.com/login.aspx?direct=true&db=ccm&AN=185099897&site=ehost-live header: @attributes: shortDbName: ccm uiTerm: 185099897 longDbName: CINAHL Complete uiTag: AN controlInfo: bkinfo: dissinfo: jinfo: jid: 13873741 OG0 jtl: Health Services & Outcomes Research Methodology issn: 13873741 maglogo: N pubinfo: dt: Jun2025 vid: 25 iid: 2 pid: 237 pub: Springer Nature place: New York, New York artinfo: ui: 185099897 177893856 185099897 185099897 10.1007/s10742-024-00332-7 185099897 ppf: 166 ppct: 16 formats: tig: atl: Markets matter: a simulation study of the bias-variance trade-off in comparison group selection for difference-in-differences analysis. aug: au: Forrow, Lauren Vollmer Rotter, Jason Blue, Laura Vogler, Jake Hatfield, Laura A. affil: https://ror.org/02403vr89 Mathematica, Inc., Cambridge, MA, USA sug: subj: Control Group Bias (Research) Models, Statistical Causality Study Design Primary Health Care United States Research Subject Recruitment Human Comparative Studies Funding Source United States Computer Simulation Medicare Fee for Service Plans Health Services Research ab: Recent research reveals many pitfalls when selecting a comparison group for difference-in-differences analyses of observational data. Recommendations from this research, although important, can be difficult to apply to a complex, real-world evaluation. In this study we explore the potential value of tailored simulation studies to inform comparison group designs for difference-in-differences analysis. We developed a simulation study mirroring the evaluation of Primary Care First (PCF), a primary care transformation model offered in select regions of the United States. We simulated primary care practices that volunteer for the intervention (intervention group) and two potential comparison groups: non-participating practices in PCF regions (within-market) and practices outside PCF regions (out-of-market). We assumed within-market comparison (i.e., non-participating) practices differ systematically from participating practices, whereas out-of-market practices are more similar to participating practices but experience different market-level outcome trends and shocks. We used Medicare fee-for-service spending data to quantify the likely magnitude of these forces. Using difference-in-differences analysis, we then estimated a hypothetical intervention's impact relative to each comparison group. We compared the mean squared error (MSE) of these impact estimates between the two comparison groups. The simulation revealed a bias-variance trade-off: the greater variation in impact estimates for the out-of-market comparison group generally led to higher MSE than the greater systematic bias of impact estimates for the within-market comparison group. More generally, a bespoke simulation study grounded in real data can quantify forces relevant to comparison group design and offer guidance to researchers seeking the best approach for their own evaluations. pubtype: Academic Journal doctype: equations & formulas research tables/charts Journal Article ougenre: Article language: English refInfo: holdings: @attributes: islocal: N |
|---|