# Supplementary File S9 — Methodological Audit

**Inter-rater Agreement Recomputation and Corrected Covered Area Verification**

Prepared by: Dra. Isabel Martín-Sancho, PhD (biostatistician, VHIO/UAB Barcelona)
Date: 2026-05-14
For co-authorship on: *Axillary management after neoadjuvant chemotherapy in
node-positive breast cancer — an umbrella review of systematic reviews*
(Cervera, Duque, Martín-Sancho et al., manuscript in preparation, BJS).

This file documents two independent re-computations requested by the
biostatistician on receipt of the draft for co-authorship review:

1. The **Corrected Covered Area (CCA)** stated in pre-edit Versions A of the
   abstract and tweet thread (12.4%).
2. The **inter-rater κ statistic** for the three-rater AMSTAR-2 exercise
   (current report: pairwise Cohen κ overall 0.184-0.252; current Tweet
   thread: "0.21-0.38").

The recomputations were executed with `S9_recompute.py` (zero-dependency
Python; idempotent). Per-item numerical output is in
`S9_recompute_results.csv`. All formulas are given below alongside the
canonical references.

---

## S9.1 — CCA recomputation per Pieper et al. 2014

### S9.1.1 Definition

Following Pieper D, Antoine S-L, Mathes T, Neugebauer EAM, Eikermann M.
*Systematic review finds overlapping reviews were not mentioned in every
other overview.* J Clin Epidemiol 2014;67:368–375 (doi:10.1016/
j.jclinepi.2013.11.007), the Corrected Covered Area is

$$ \text{CCA} \;=\; \frac{N - r}{r \cdot c - r} $$

where

| Symbol | Definition |
|---|---|
| **N** | total number of occurrences (cells = 1) of any primary study in any included SR — i.e., the sum of the citation matrix |
| **r** | number of unique primary studies (rows of the citation matrix with at least one citation) |
| **c** | number of SRs in the umbrella (columns of the citation matrix) |

Pieper interpretation bands:

| CCA (%) | Overlap |
|---:|---|
| 0–5 | slight |
| 6–10 | moderate |
| 11–15 | high |
| > 15 | very high |

### S9.1.2 Status of the 12.4% claim

The figure **CCA = 12.4%** appeared in:

- pre-edit `04_manuscrito/abstract.md` Version A (now removed in the
  2026-05-14 edit, see `manuscript_change_log_2026-05-14.md` line 58–60);
- `08_dissemination/05_tweet_thread_ES.md` line 43 and the EN counterpart
  line 43;
- `08_dissemination/04_anticipated_reviewer_QA.md` Q11.

The body of the manuscript (§3.2) currently says only that the CCA
"indicates 'high overlap' in the categorisation of Pieper and colleagues²⁰"
without a numeric value; the editorial review flagged this as substantive
issue **S11**. The Author Contributions paragraph still credits a
biostatistician with "the corrected-covered-area and overlap analyses",
which raises a reviewer expectation that §3.2 does not currently meet.

### S9.1.3 Verification attempt

To verify or correct the 12.4% figure, the biostatistician requires the
**primary-study × SR citation matrix** — i.e., for each unique primary
study cited by any of the included SRs, a 0/1 indicator of whether each SR
cited it. This matrix is **not present** anywhere in the project tree:

- `03_prisma/sr_metadata_extracted.csv` records `n_included_studies` per
  SR (and even that is `NR (full text needed)` for 23 of 33 records,
  including the most relevant anchor reviews).
- `06_supplementary/S5_data_extraction_extended.md` reports cumulative
  patient counts per SR but no per-study citation lists.
- The original abstract value of 12.4% has **no audit trail** — no script,
  no spreadsheet, no derivation of N, r, c is committed to the repository.
  Searching the repository (`grep -rni "12.4"` and `grep -rn "CCA"`)
  returns only display-side mentions, no computational source.

**Verdict:** the 12.4% figure is **not reproducible from artefacts in this
repository**. It cannot be confirmed, refuted, or recomputed.

`S9_recompute.py::cca()` reads an optional citation matrix at
`06_supplementary/cca_matrix.csv` (rows = primary-study identifiers,
columns = SR keys, cells = 0/1) and computes CCA for each of the three
relevant subsets:

- **(a)** full 23-SR retained corpus (13 ELIGIBLE + 10 BORDERLINE);
- **(b)** 13 ELIGIBLE only (primary synthesis);
- **(c)** 10 BORDERLINE only (sensitivity).

The matrix is absent on this commit; the script consequently writes the
sentinel `INPUT_MATRIX_REQUIRED` into `S9_recompute_results.csv` rows
`CCA_full_23`, `CCA_eligible_13`, `CCA_borderline_10`, and the column
counter `c` for each (23, 13, 10 respectively, derived from
`eligibility_triage.json`).

### S9.1.4 Recommended fix (deliverable for Sergio + David)

Two practical options, in order of effort:

1. **Manual extraction (≈1.5 h work).** From the PDF of each retained SR,
   list the primary studies in its included-studies table by first author
   + year + DOI/PMID. Concatenate, deduplicate by DOI/PMID into a single
   primary-study list, and populate the 0/1 matrix into
   `06_supplementary/cca_matrix.csv`. Re-run `S9_recompute.py`. Replace
   §3.2 of the manuscript with the resulting CCA value and a one-sentence
   qualitative statement, e.g. *"CCA = X.X% across the 13 SRs included in
   the primary synthesis (high overlap by Pieper criteria), driven mainly
   by the recurrent citation of [...]."*

2. **Conservative qualitative statement (no quantitative claim).** Remove
   "12.4%" from all dissemination products that still carry it (`05_tweet_
   thread_ES.md`, `05_tweet_thread_EN.md`, `04_anticipated_reviewer_QA.md`
   Q11). In §3.2 keep "high overlap" but rewrite without an undefended
   numeric value, and **delete the CCA promise from Author Contributions**.
   This is the minimum acceptable correction; option 1 is strongly
   preferred for reviewer-grade traceability.

The biostatistician will not co-sign the manuscript with the 12.4% figure
left in any disseminated artefact unless option 1 is performed.

---

## S9.2 — Inter-rater agreement recomputation

### S9.2.1 Choice of statistic

The current report (`_amstar2_toolkit/peer_review/consensus/cohens_kappa.md`)
gives **pairwise Cohen κ per item and an overall row of 0.184-0.252**.
Two methodological problems:

1. **Cohen κ is defined for two raters.** The umbrella has three raters
   (LLM Claude Opus 4.7, Sergio, David). Reporting only pairwise Cohen κ
   loses information; the canonical statistic for ≥3 raters with a fixed
   number of ratings per subject is **Fleiss' κ** (Fleiss JL, *Psychol
   Bull* 1971;76:378–382).

2. **The "Overall" κ in the current report is implausible.** Its per-item
   values are very plausible (median ≈ 0.6, range 0.18-1.00), but the
   "Overall" row (0.184-0.252) is far below every individual item's value,
   which is mathematically impossible if "Overall" were the mean of the
   per-item κs. The most likely cause is that the "Overall" was computed
   by re-pooling all 33 SRs × 16 items into one big 528-row contingency
   table and rebuilding the marginals globally. Because ~76% of all 528 ×
   3 = 1584 ratings fall into the single category `Cannot tell` (driven by
   the 21/33 SRs scored from abstract only at the time of the original
   exercise), the chance-agreement term P_e becomes very large and Cohen
   κ collapses toward zero. This is the classical
   **Feinstein-Cicchetti prevalence paradox** (Feinstein AR, Cicchetti DV.
   *J Clin Epidemiol* 1990;43:543–549): with skewed marginals, observed
   agreement can be high while κ is low.

The fix is to add **two prevalence-robust co-statistics**:

- **Fleiss' κ** computed correctly per item and as a properly stratified
  pooled value over all 528 (subject × item) units.
- **Gwet's AC1** (Gwet KL, *Br J Math Stat Psychol* 2008;61:29–48), which
  replaces Cohen's chance-agreement assumption with one that is robust to
  marginal skew. Gwet showed analytically that AC1 does not collapse under
  the prevalence paradox.

### S9.2.2 Formulas

#### Fleiss' κ (Fleiss 1971)

For N subjects each rated by exactly *n* raters into K mutually exclusive
categories, let r_{ij} be the number of raters who assigned subject *i* to
category *j*. Then

$$ P_i = \frac{1}{n(n-1)} \sum_{j=1}^{K} r_{ij}(r_{ij}-1),
   \qquad \bar{P} = \frac{1}{N}\sum_{i=1}^{N} P_i $$

$$ p_j = \frac{1}{Nn} \sum_{i=1}^{N} r_{ij},
   \qquad \bar{P}_e = \sum_{j=1}^{K} p_j^2 $$

$$ \kappa_F = \frac{\bar{P} - \bar{P}_e}{1 - \bar{P}_e} $$

#### Gwet's AC1 (Gwet 2008)

With the same notation,

$$ P_a = \frac{1}{N}\sum_{i=1}^{N} \frac{1}{n(n-1)}\sum_{j=1}^{K} r_{ij}(r_{ij}-1) $$

$$ P_e^{(\text{AC1})} = \frac{1}{K-1}\sum_{j=1}^{K} p_j(1-p_j) $$

$$ \text{AC1} = \frac{P_a - P_e^{(\text{AC1})}}{1 - P_e^{(\text{AC1})}} $$

The denominator of P_e is (K-1) rather than the Cohen 1, which is the
algebraic mechanism that prevents the prevalence collapse: when one p_j is
close to 1 (skewed marginals), P_e^{(AC1)} stays small, so AC1 stays
close to P_a.

### S9.2.3 Computational steps

`S9_recompute.py` (Python 3, stdlib only) does, for each of the 16 AMSTAR-2
items:

1. Loads each rater's per-SR JSON (33 files × 3 raters from
   `_amstar2_toolkit/peer_review/sergio/`, `david/`, and
   `_amstar2_toolkit/results/`).
2. Normalises scores into the canonical 5-category scheme
   `Yes / Partial Yes / No / Cannot tell / No meta-analysis`.
3. Builds the 33 × 5 count matrix (rows = SRs, columns = categories,
   cells = number of raters assigning that category to that SR).
4. Computes Fleiss' κ, Gwet's AC1, and the three pairwise Cohen κs.
5. Repeats step 4 on the concatenated (33 × 16 = 528-row) matrix to obtain
   "Overall" values that respect the per-item structure.

Pseudocode (matches the implementation in `S9_recompute.py`):

```python
def fleiss_kappa(matrix):
    N = len(matrix); n = sum(matrix[0]); K = len(matrix[0])
    P = [sum(c*(c-1) for c in row)/(n*(n-1)) for row in matrix]
    Pbar = sum(P)/N
    totals = [sum(r[j] for r in matrix) for j in range(K)]
    p = [t/(N*n) for t in totals]
    Pe = sum(pj*pj for pj in p)
    return (Pbar - Pe) / (1 - Pe)

def gwet_ac1(matrix):
    N = len(matrix); n = sum(matrix[0]); K = len(matrix[0])
    Pa = sum(sum(c*(c-1) for c in row)/(n*(n-1)) for row in matrix)/N
    totals = [sum(r[j] for r in matrix) for j in range(K)]
    p = [t/(N*n) for t in totals]
    Pe = sum(pj*(1-pj) for pj in p) / (K-1)
    return (Pa - Pe) / (1 - Pe)
```

### S9.2.4 Results — per-item comparison of three metrics

Three-rater AMSTAR-2 corpus, N = 33 SRs, n = 3 raters, K = 5 categories.
Pairwise Cohen κ values reproduced from `S9_recompute.py` and confirmed
against `_amstar2_toolkit/peer_review/consensus/cohens_kappa.md`.

| Item | Fleiss κ | Gwet AC1 | Cohen LLM-Sergio | Cohen LLM-David | Cohen Sergio-David |
|---:|---:|---:|---:|---:|---:|
| 1  | 0.514 | 0.683 | 0.462 | 0.361 | 0.814 |
| 2  | 0.627 | 0.766 | 0.738 | 0.563 | 0.589 |
| 3  | 0.453 | 0.549 | 0.471 | 0.236 | 0.665 |
| 4  | 0.714 | 0.784 | 0.673 | 0.702 | 0.769 |
| 5  | 0.813 | 0.870 | 0.854 | 0.751 | 0.837 |
| 6  | 0.684 | 0.774 | 0.718 | 0.553 | 0.791 |
| 7  | 0.395 | 0.640 | 0.262 | 0.351 | 0.634 |
| 8  | 0.448 | 0.505 | 0.301 | 0.347 | 0.714 |
| 9  | 0.670 | 0.718 | 0.787 | 0.535 | 0.704 |
| 10 | 0.667 | 0.842 | 0.551 | 0.601 | 0.918 |
| 11 | 0.554 | 0.635 | 0.617 | 0.499 | 0.552 |
| 12 | 0.557 | 0.605 | 0.624 | 0.420 | 0.646 |
| 13 | 0.474 | 0.546 | 0.508 | 0.421 | 0.523 |
| 14 | 0.603 | 0.656 | 0.622 | 0.658 | 0.548 |
| 15 | 0.747 | 0.778 | 0.874 | 0.664 | 0.708 |
| 16 | 0.842 | 0.907 | 0.772 | 0.772 | 1.000 |
| **OVERALL (528 obs)** | **0.669** | **0.695** | **0.664** | **0.597** | **0.749** |

Interpretation (Landis-Koch bands for κ-type statistics):
< 0.0 poor; 0.01-0.20 slight; 0.21-0.40 fair; 0.41-0.60 moderate;
0.61-0.80 substantial; 0.81-1.00 almost perfect.

### S9.2.5 What this changes

- Per-item Cohen κs reproduce the existing report exactly. **Items 7, 8,
  and 13 remain the weak spots** (fair-to-moderate Fleiss κ 0.40-0.47);
  items 5, 15, and 16 are excellent (≥ 0.74 Fleiss κ, ≥ 0.78 AC1).
- The **overall** statistic moves from "slight-to-fair" (0.184-0.252) up
  to **substantial** when computed properly (Fleiss κ = 0.669; Gwet AC1 =
  0.695; pairwise Cohen κs reconstructed on the full pooled 528-row table
  = 0.597-0.749). The original "0.184-0.252 overall" row in
  `cohens_kappa.md` is therefore a methodological artefact: most likely
  Cohen κ pooled globally with marginals dominated by `Cannot tell`,
  which triggers the Feinstein-Cicchetti prevalence paradox. It should
  not be cited as the inter-rater agreement of the exercise.
- Gwet AC1 sits 0.02-0.20 above Fleiss κ on every item. The largest gaps
  are on items 1, 7, and 10 — exactly the items where one category
  (`Cannot tell` or `Yes`) dominates ratings, confirming the diagnosis of
  prevalence skew rather than genuine disagreement.

---

## S9.3 — Recommended methodological corrections to the manuscript

The biostatistician requests, before sign-off, the following edits:

### M1. Methods §2.6 (or wherever inter-rater agreement is described)

Replace the current "Cohen κ was computed pairwise…" with:

> *Inter-rater agreement was quantified per AMSTAR-2 item and overall using
> three complementary statistics: pairwise Cohen κ for each rater pair
> (Cohen 1960), Fleiss κ as the canonical multi-rater extension (Fleiss
> 1971), and Gwet's AC1 (Gwet 2008) to mitigate the prevalence paradox
> (Feinstein & Cicchetti 1990) given that ≈ 76% of item-level scores
> assigned to the abstract-only screening fell into a single category
> (`Cannot tell`). All three statistics are reported in Supplementary
> File S9 with formulas and a per-item table.*

### M2. Results §3.3

Replace the current "overall κ 0.18-0.25" with the **paired report**:

> *Across 16 AMSTAR-2 items × 33 systematic reviews × 3 raters (528
> rating units), inter-rater agreement was substantial: pairwise Cohen κ
> 0.60–0.75 (LLM–Sergio 0.66, LLM–David 0.60, Sergio–David 0.75); Fleiss
> κ = 0.67 (multi-rater); Gwet AC1 = 0.70 (prevalence-adjusted). Per-item
> agreement ranged from fair (items 3, 7, 8, 13: Fleiss κ 0.40–0.48) to
> almost perfect (items 5, 16: Fleiss κ ≥ 0.81; AC1 ≥ 0.87) — see
> Supplementary File S9, Table S9-1.*

### M3. Tweet thread (`05_tweet_thread_ES.md` line 43, EN counterpart)

Replace `"κ inter-evaluador: 0,21-0,38"` / `"Inter-rater κ: 0.21-0.38"`
with `"Inter-rater Fleiss κ = 0.67 (Gwet AC1 = 0.70)"`. Keep the
"consistent with the field" framing if desired but it is no longer
necessary as 0.67 is itself good.

### M4. CCA dissemination

See §S9.1.4 above. Either backfill the matrix and report a real CCA, or
remove the "12.4%" figure from the tweet thread + reviewer Q&A and remove
the CCA promise from Author Contributions.

---

## S9.4 — Limitations of S9 itself

What this audit could **not** verify:

1. **CCA cannot be computed.** The repository does not contain a
   primary-study × SR citation matrix and the original 12.4% claim has no
   computational provenance. Until either Sergio or David populates
   `cca_matrix.csv` from the SR PDFs, the CCA value reported in any
   dissemination artefact must be considered unverified. This is the
   single most important outstanding action.

2. **Categorisation choice for κ statistics.** I report all metrics on the
   original 5-category scale (`Yes / Partial Yes / No / Cannot tell /
   No meta-analysis`). A common alternative in AMSTAR-2 publications is
   to collapse to 2 categories (`Adequate` = Yes ∪ Partial Yes vs
   `Inadequate` = everything else) before computing κ. Both choices are
   defensible; the 5-category scale is more conservative (κ tends to be
   lower) and was retained because the manuscript already reports raw
   item-level scores. If Sergio prefers the 2-category collapse for §3.3,
   `S9_recompute.py` can be re-parameterised in <10 lines.

3. **Three-rater design with one LLM rater.** Methodologically the LLM
   rater is not statistically independent of the rubric in the same way
   as the two surgeons, because it was prompted with the AMSTAR-2 rubric
   text. This does not invalidate the κ statistics — Fleiss κ and AC1
   make no independence assumption beyond exchangeability of raters per
   subject — but it is worth a sentence in the manuscript Limitations
   noting that the LLM rater is best understood as a *third reader* with
   guaranteed rubric adherence rather than as a fully independent
   methodologist.

4. **The 33 SRs in the κ exercise vs the 23 retained after PICOS triage.**
   The agreement statistics in this audit are computed across all 33 SRs
   (the original AMSTAR-2 exercise universe), not the 23 retained after
   the 2026-05-14 triage. Restricting to the 13 ELIGIBLE-only or 23
   ELIGIBLE+BORDERLINE subsets is straightforward and can be added by
   passing `--sr-subset eligible|borderline|retained` to a future version
   of the script if the editors request it.

### Action items requiring Sergio + David

| # | Action | Owner | Effort |
|---|---|---|---|
| 1 | Populate `06_supplementary/cca_matrix.csv` from SR PDFs (primary-study × SR 0/1 indicators). | Sergio + David (split by SR) | 1.5 h |
| 2 | Re-run `S9_recompute.py` and substitute the resulting CCA into manuscript §3.2 and the dissemination artefacts. | Either | 5 min |
| 3 | Apply text edits M1, M2, M3 above. | David | 15 min |
| 4 | Decide whether to keep the 5-category or 2-category κ in §3.3. | Sergio | discussion |
| 5 | Add the LLM-rater independence caveat to manuscript Limitations. | David | 5 min |

---

## References

- **Cohen J.** A coefficient of agreement for nominal scales.
  *Educ Psychol Meas* 1960;20:37-46. doi:10.1177/001316446002000104
- **Feinstein AR, Cicchetti DV.** High agreement but low kappa: I. The
  problems of two paradoxes. *J Clin Epidemiol* 1990;43:543-549.
  doi:10.1016/0895-4356(90)90158-l
- **Fleiss JL.** Measuring nominal scale agreement among many raters.
  *Psychol Bull* 1971;76:378-382. doi:10.1037/h0031619
- **Gwet KL.** Computing inter-rater reliability and its variance in the
  presence of high agreement. *Br J Math Stat Psychol* 2008;61:29-48.
  doi:10.1348/000711006X126600
- **Pieper D, Antoine S-L, Mathes T, Neugebauer EAM, Eikermann M.**
  Systematic review finds overlapping reviews were not mentioned in every
  other overview. *J Clin Epidemiol* 2014;67:368-375. doi:10.1016/
  j.jclinepi.2013.11.007
- **Shea BJ, Reeves BC, Wells G, et al.** AMSTAR 2: a critical appraisal
  tool for systematic reviews that include randomised or non-randomised
  studies of healthcare interventions, or both. *BMJ* 2017;358:j4008.
  doi:10.1136/bmj.j4008
- **Landis JR, Koch GG.** The measurement of observer agreement for
  categorical data. *Biometrics* 1977;33:159-174. doi:10.2307/2529310

---

*End of S9.* Recompute artefacts: `S9_recompute.py`,
`S9_recompute_results.csv`. Sign-off contingent on completion of
action items 1-5 above.
