# Data dictionary - ypN0 clinical predictor datasets

This document accompanies three CSV files in `predictor_ypN0/data/`:

1. `ypN0_rates_by_subtype.csv` - axillary pCR rates by breast cancer subtype.
2. `predictor_effect_sizes.csv` - odds ratios / hazard ratios for ypN0.
3. `regimen_effect.csv` - regimen-level (chemo / chemo-IO / NET / dual-HER2) pCR rates.

All rates are stored as fractions (0-1). PMID is stored as a string (no leading zeros expected).
Every row has a `source_file` pointing to the corresponding `.md` in this repository.

---

## 1. `ypN0_rates_by_subtype.csv`

| column        | type   | description |
|---------------|--------|-------------|
| study_id      | str    | short unique identifier `Author{Year}_TrialName` |
| pmid          | str    | PubMed ID; empty if not indexed |
| first_author  | str    | first author family name + initials |
| year          | int    | publication year |
| journal       | str    | journal name |
| n_total       | int    | total cohort size in the source study (may exceed n_subgroup) |
| subtype       | str    | standardised subtype value (see below) |
| n_subgroup    | int    | N in the subtype/subset for which the rate applies (NULL if not reported) |
| ypN0_n        | int    | numerator (axillary pCR events). NULL if not directly reported |
| ypN0_rate     | float  | proportion (0-1) of ypN0 within the subgroup |
| ci_low        | float  | lower bound 95% CI (NULL if not reported and not computable) |
| ci_high       | float  | upper bound 95% CI |
| definition    | str    | how the source defined the outcome |
| source_file   | str    | repo-relative path to the `.md` |
| notes         | str    | caveats (approximate, pooled, Wilson-approximated CI, etc.) |

### Allowed values for `subtype` (controlled vocabulary)
- `Overall`
- `HR-/HER2+` (= HER2-enriched, ERBB2-positive HR-negative)
- `HR+/HER2+` (= triple-positive)
- `HER2+` (combined HR-/HER2+ and HR+/HER2+, used by some meta-analyses)
- `TNBC` (triple-negative, ER-/PR-/HER2-)
- `HR+/HER2-` (luminal-like, ER+ and/or PR+, HER2-)
- `Luminal A`
- `Luminal B`
- regimen / context modifiers in parentheses, e.g. `TNBC (pembro+chemo)`,
  `HR+/HER2- (high genomic risk)`, `HR+/HER2- (age<50, Ki67>=20%)`,
  `HR+/HER2- (NAC)` / `HR+/HER2- (NET)`, `Lobular HER2+`, etc.

### CI approximation policy
When the source reports `ypN0_n` and `n_subgroup` but no 95% CI, we approximate
with the Wilson score interval (function `wilson_ci` in `build_dataset.py`).
Rows using this method are flagged in `notes`.

### Caveats by source
- **Samiei 2021 (PMID 33881478)**: pooled random-effects estimates with no subgroup
  N or CI in abstract. Pooled HER2+ (combined) row preserved because some literature
  references this aggregate; modellers should pick either `HER2+` or split rows but
  not both to avoid double counting.
- **van Amstel 2024 (PMID 39302345)**: only ORs reported in abstract; subtype-specific
  ypN0 rates approximated from the bibliographic summary
  `10_pronostico_supervivencia/RESUMEN_05_predictores_ypN0.md`. Flagged as approximate.
- **Barron 2018 (PMID 30193375)**: rates are **conditional on breast pCR**, not
  marginal axillary pCR rates. Useful for the calculator's joint-pCR scenario.
- **Montagna 2024 (PMID 38662396)**: cohort selected by ypN0 (rate = 1.0 by design);
  use as outcome anchor (axillary recurrence 1.0% at 5y) rather than predictor.
- **Chen 2021, Mukhtar 2025, Boughey 2022 Ki67**: subgroup rates given but sub-N
  not explicit; CI not computable.

---

## 2. `predictor_effect_sizes.csv`

| column        | type   | description |
|---------------|--------|-------------|
| study_id      | str    | unique identifier |
| pmid          | str    | PubMed ID |
| year          | int    | publication year |
| predictor     | str    | controlled vocabulary (see below) |
| comparison    | str    | the contrast for the OR/HR (e.g. "HR-/HER2+ vs HR+/HER2-") |
| OR            | float  | odds ratio OR hazard ratio depending on `model_type` |
| ci_low        | float  | lower 95% CI |
| ci_high       | float  | upper 95% CI |
| p_value       | float  | p-value if reported |
| n             | int    | analytic N |
| model_type    | str    | e.g. "multivariable logistic regression", "Cox PH", "univariable" |
| source_file   | str    | repo-relative path |
| notes         | str    | caveats |

### Allowed values for `predictor`
- `subtype` (most common; comparison field carries the contrast)
- `Ki67` (typically >=20% vs <20%; sometimes >=30%)
- `age` (continuous OR `<50 vs >=50`)
- `cN_stage` (cN2/3 vs cN1)
- `histology` (ILC vs IDC)
- `regimen` (with vs without pembro, pertuzumab, carboplatin, immunotherapy, NAC vs NET)
- `tumor_size` (cT category, continuous)
- `genomic_risk` (Oncotype RS, MammaPrint, BluePrint)
- `breast_pCR` (yes vs no; modelled as predictor of axillary pCR)
- `axillary_burden_PETCT` (limited vs advanced cN extent on baseline FDG-PET/CT)
- `race` (Black vs White, etc.)
- `regional_RT` (omission vs receipt, used in OPBC-07)

### Sign convention
- OR > 1 = higher probability of ypN0 (axillary pCR) -- preferred direction for
  predictors of interest.
- One row (Boughey 2022 genomic, low vs high risk) uses **OR for residual nodal
  disease**, NOT ypN0; the calculator must invert (1/OR) before composing.
- OPBC-07 rows are **hazard ratios for axillary recurrence**, not ORs for ypN0;
  documented in `notes` -- they belong in the downstream outcome model, not in the
  Nivel-1 logistic calculator. Modellers should filter `model_type` to retain only
  logistic / OR rows for the Nivel-1 calculator.

---

## 3. `regimen_effect.csv`

| column              | type   | description |
|---------------------|--------|-------------|
| study_id            | str    | unique identifier |
| pmid                | str    | PubMed ID |
| year                | int    | publication year |
| regimen_name        | str    | short label (e.g. "KEYNOTE-522 (pembro+chemo)") |
| regimen_components  | str    | drugs / sequence |
| subtype_target      | str    | target subtype population |
| n                   | int    | N in the regimen-specific arm |
| ypN0_rate           | float  | axillary pCR rate as fraction (NULL if not reported) |
| breast_pCR_rate     | float  | breast-only pCR rate as fraction |
| source_file         | str    | repo-relative path |
| notes               | str    | caveats |

### Notes
- KEYNOTE-522 and TRAIN-2-like dual-HER2 are the canonical examples; we cover them
  with the Kuemmel 2026 surgical-outcomes paper (KEYNOTE-522) and the Chowdhary 2026
  pilot (dual HER2 blockade).
- NeoSTOP, WSG-ADAPT-TN are listed for TNBC carboplatin context. KATHERINE and
  CheckMate-816 are excluded as instructed (residual setting / non-breast).
- NET vs NAC contrast for HR+/HER2- is captured both at the pooled meta-analysis
  level (Vasigh 2024) and at the institutional level (An 2024).

---

## Reproducibility

Run `python build_dataset.py` from this directory. The script is **idempotent**:
all data are hard-coded inside it, so re-running regenerates the same CSVs. Add new
rows by extending the corresponding `_xxx_rows()` factory functions, then rerun.

The Wilson 95% CI is computed via `wilson_ci(k, n)` (see top of script). Float
formatting uses Python `round(..., 3)`. All percentage values from the literature
were divided by 100 at the point of insertion (no late conversion).
