# AMSTAR-2 detailed report — full-text LLM-assisted scoring (Opus 4.7)

**Compiled:** 2026-05-14
**Pipeline:** Multi-agent local Claude Opus 4.7 (Sonnet 4.6 fallback for one record). One agent per SR, 16 items + algorithmic overall confidence per Shea et al. *BMJ* 2017 (doi:10.1136/bmj.j4008).
**Inputs:** 14 of 33 SRs scored with full-text (PDFs extracted via `pdfplumber`); 19 abstract-only.

## Distribution of overall confidence

| Rating | Count | % |
|---|---:|---:|
| High | 0 | 0.0% |
| Moderate | 2 | 6.1% |
| Low | 6 | 18.2% |
| Critically Low | 25 | 75.8% |
| **TOTAL** | **33** | 100% |

### Comparison with abstract-only pilot scoring

The previous abstract-only conservative pilot rated 33/33 (100%) "Critically Low" because items 5/6/7/12/16 were systematically coded "Cannot tell" and conservatively counted as flaws. The current full-text/LLM-augmented scoring yields a more nuanced distribution: 2 Moderate, 6 Low, 25 Critically Low, 0 High. Several anchor SRs (Tee 2018 BJS, Wang 2025) move up to Moderate; others (Cao 2021, Lin 2022, Boland 2025, Ding 2025, Tan 2011, vanNijnatten 2015) rate Low. The largest pooled analysis (Samiei 2021 JAMA Surg) remains Critically Low driven by absent protocol/RoB/COI reporting in the published article — a real methodological observation, not an artefact.

## Anchor SRs — most cited / largest, with detailed scoring

| First author | Year | Journal | Design | MA? | Full text? | n_yes | n_crit | Overall confidence |
|---|---|---|---|---|---|---:|---:|---|
| Mamounas | 2012 | J Clin Oncol | Pooled individual patient data analysis of two NSABP RCTs (B-18 and B-27); not a systematic review | — | ✓ | 1 | 5 | **Critically Low** |
| Mocellin | 2015 | Int J Cancer | Systematic review and meta-analysis | ✓ | — | 1 | 0 | **Critically Low** |
| Tee | 2018 | Br J Surg | Systematic review with meta-analysis | ✓ | — | 0 | 0 | **Moderate** |
| Cao | 2021 | The Breast | Systematic review and meta-analysis of diagnostic accuracy studies | ✓ | ✓ | 5 | 1 | **Low** |
| Samiei | 2021 | JAMA Surg | Systematic review and meta-analysis | ✓ | ✓ | 6 | 4 | **Critically Low** |
| Swarnkar | 2021 | Cancers | Systematic review with pooled analysis (proportions meta-analysis) | ✓ | ✓ | 1 | 6 | **Critically Low** |
| Ma | 2023 | Front Oncol | Systematic review with meta-analysis | ✓ | ✓ | 5 | 4 | **Critically Low** |
| Pantiora | 2023 | Br J Surg | Systematic review and meta-analysis | ✓ | ✓ | 11 | 2 | **Critically Low** |
| Yu | 2024 | Breast Cancer (Auckl) | Systematic review with meta-analysis of diagnostic accuracy studies | ✓ | ✓ | 7 | 3 | **Critically Low** |
| de Wild | 2024 | Br J Surg | Systematic review with random-effects meta-analysis of proportions (metaprop) | ✓ | ✓ | 5 | 4 | **Critically Low** |
| Man | 2025 | Breast Cancer | Systematic review and meta-analysis | ✓ | ✓ | 7 | 4 | **Critically Low** |
| Abdulla | 2026 | Ann Surg Oncol | Systematic review and meta-analysis | ✓ | — | 1 | 1 | **Critically Low** |

## Item-level reporting gaps across the 33 SRs

| Item | Description | Critical | "No" count | "Cannot tell" count |
|---|---|:-:|---:|---:|
| 1  | PICO components | N | 3 | 2 |
| 2 ⚠️ | Pre-registered protocol | Y | 17 | 12 |
| 3  | Design justification | N | 16 | 14 |
| 4 ⚠️ | Comprehensive search | Y | 7 | 6 |
| 5  | Duplicate selection | N | 5 | 15 |
| 6  | Duplicate extraction | N | 8 | 16 |
| 7 ⚠️ | Excluded studies list | Y | 18 | 13 |
| 8  | Studies described | N | 2 | 13 |
| 9 ⚠️ | RoB tool | Y | 11 | 13 |
| 10  | Funding of primary studies | N | 20 | 12 |
| 11 ⚠️ | MA methods | Y | 1 | 6 |
| 12  | RoB impact on MA | N | 13 | 14 |
| 13 ⚠️ | RoB in interpretation | Y | 14 | 12 |
| 14  | Heterogeneity | N | 3 | 7 |
| 15 ⚠️ | Publication bias | Y | 12 | 10 |
| 16  | COI/funding of review | N | 0 | 19 |

## Off-topic / eligibility flags


## Per-SR rationale

> Detailed item-level rationales are in the per-SR JSON files at `_amstar2_toolkit/results/`. The notes below reproduce the principal strengths/weaknesses identified by the LLM scorer.

### Xing 2006 — Br J Surg (Critically Low)

PMID 16329089 · DOI 10.1002/bjs.5209 · meta-analysis · MA=True · FullText=False

> Abstract-only assessment limits visibility, but explicit absence of protocol (item 2), formal RoB assessment (item 9), publication bias evaluation (item 15), and uncritical interpretation (item 13) yield three critical flaws. Bayesian hierarchical modelling is a methodological strength.

### Kelly 2009 — Acad Radiol (Critically Low)

PMID 19345896 · DOI 10.1016/j.acra.2009.01.026 · Systematic review with meta-analysis (bivariate random-effects mixed model) of SLN mapping after neoadjuvant chemotherapy in breast cancer · MA=True · FullText=False

> Only abstract-level metadata available; scoring defaults to 'Cannot tell' where the abstract is silent. Critical flaws on items 2, 7, 13, 15 (No protocol pre-registration, no excluded-studies list, no RoB-informed interpretation, no publication-bias assessment) yield Critically Low confidence.

### van Deurzen 2009 — Eur J Cancer (Critically Low)

PMID 19716287 · DOI 10.1016/j.ejca.2009.08.001 · Systematic review with meta-analysis (pooled estimates) · MA=True · FullText=False

> Scoring based on abstract only (full text not available); critical flaws on items 2, 7, 9, 13, 15. Item 4 = Partial Yes (not a flaw); item 11 = Cannot tell (MA conducted but methods not described in abstract).

### Tan 2011 — J Surg Oncol (Low)

PMID 21456092 · DOI 10.1002/jso.21911 · Systematic review with meta-analysis · MA=True · FullText=False

> Assessment limited to abstract; full text not accessible. Meta-analysis confirmed (SLNB feasibility/accuracy in cN0 post-NAC breast cancer). With 0 confirmed critical flaws but 15 'Cannot tell' across critical and non-critical items, the algorithm formally yields Moderate; rating downgraded to Low to reflect substantial unverifiable methodological uncertainty across critical domains (items 2, 4, 7, 9, 11, 13, 15).

### Mamounas 2012 — J Clin Oncol (Critically Low)

PMID 23032615 · DOI 10.1200/JCO.2011.40.8369 · Pooled individual patient data analysis of two NSABP RCTs (B-18 and B-27); not a systematic review · MA=False · FullText=True

> This is a pooled IPD analysis of two NSABP RCTs (B-18, B-27), not a systematic review. AMSTAR-2 is methodologically inappropriate for this design; near-uniform 'No' scores reflect the mismatch rather than poor execution of the underlying combined trial analysis.

### Moody 2012 — Surgery (Critically Low)

PMID 22938899 · DOI 10.1016/j.surg.2012.06.017 · Meta-analysis of observational/validation studies · MA=True · FullText=False

> Eligibility questionable: study addresses SLN biopsy in multifocal/multicentric breast cancer, with NAC appearing only as a contraindication subgroup, not as the primary intervention of the NAC umbrella review. Scoring is constrained because only the PubMed abstract was available; full text via DOI/subscription was not retrieved.

### Nagaraja 2014 — J Gastrointest Oncol (Critically Low)

PMID 24772341 · DOI 10.3978/j.issn.2078-6891.2014.008 · Systematic review and meta-analysis · MA=True · FullText=True

> This SR addresses sentinel lymph node biopsy in oesophageal cancer and is OUT OF SCOPE for the axillary management post-NAC breast cancer umbrella review. Four critical items failed (item2 no pre-registration, item7 no excluded-studies list, item9 no RoB tool, item13 RoB not considered in conclusions), yielding Critically Low confidence. Primary studies are predominantly from Japan, limiting generalisability.

### Schipper 2014 — Eur J Radiol (Critically Low)

PMID 25458227 · DOI 10.1016/j.ejrad.2014.09.020 · Systematic review (narrative, no meta-analysis) · MA=False · FullText=False

> Only abstract available; full text not accessible, so most items default to Cannot tell. Despite zero confirmed critical 'No' flaws, pervasive uncertainty across critical items (2, 7, 9, 13) warrants Critically Low confidence.

### Mocellin 2015 — Int J Cancer (Critically Low)

PMID 26084763 · DOI 10.1002/ijc.29644 · Systematic review and meta-analysis · MA=True · FullText=False

> Assessment based on abstract only; full text not accessible. Most items unverifiable, so multiple Cannot tell preclude a higher confidence rating despite no explicit critical flaws.

### van Nijnatten 2015 — Eur J Surg Oncol (Low)

PMID 26329781 · DOI 10.1016/j.ejso.2015.07.020 · Systematic review and meta-analysis of diagnostic test accuracy · MA=True · FullText=False

> Assessment based on abstract only (full text not retrieved). Many items rated 'Cannot tell' due to unavailable methodology details. Conservative rating: with 0 explicit critical 'No' flaws but 10 'Cannot tell' on methodology including 5 critical items (2,7,9,13,15), reliability of conclusions is limited; overall set to Low to reflect substantial uncertainty about critical domains.

### El Hage Chehade 2016 — Am J Surg (Critically Low)

PMID 27671032 · DOI 10.1016/j.amjsurg.2016.07.018 · Systematic review with meta-analysis · MA=True · FullText=False

> Only abstract available; full text not accessible. All items scored 'Cannot tell' due to insufficient information. Conservative overall rating of Critically Low assigned because methodological quality cannot be verified from abstract alone. Meta-analysis of 19 studies, 3,398 patients on SLNB after NAC in cN+ breast cancer.

### Geng 2016 — PLoS One (Critically Low)

PMID 27606623 · DOI 10.1371/journal.pone.0162605 · Systematic review and meta-analysis of diagnostic accuracy studies · MA=True · FullText=True

> Meta-analysis of 16 diagnostic accuracy studies (n=1,456) on SLNB after NAC in initially cN0 breast cancer. Three critical flaws (items 2, 7, 13) drive Critically Low rating despite sound statistical methods and quality assessment. Pooled IR 96%, FNR 6%.

### Krug 2018 — Strahlenther Onkol (Critically Low)

PMID 29383405 · DOI 10.1007/s00066-018-1270-x · Systematic review (narrative, no meta-analysis) · MA=False · FullText=False

> Only abstract available; many items scored 'Cannot tell' for non-critical methodology details. Critical flaws confirmed on items 2 (no protocol), 4 (single-database search), and 7 (no excluded-studies list). Narrative SR of 13 retrospective studies; items 11, 12, 15 marked 'No meta-analysis'. Score is conservative pending full-text access.

### Tee 2018 — Br J Surg (Moderate)

PMID 30311642 · DOI 10.1002/bjs.10986 · Systematic review with meta-analysis · MA=True · FullText=False

> Only PubMed abstract available; full text not accessible. Most items 'Cannot tell' due to lack of methodological detail. No confirmed 'No' on critical items, so per Shea 2017 algorithm: 0 critical flaws + >1 weakness = Moderate. Rating reflects insufficient information to fully appraise; full-text retrieval via DOI 10.1002/bjs.10986 needed for definitive scoring.

### Simons 2019 — Ann Surg (Critically Low)

PMID 30312200 · DOI 10.1097/SLA.0000000000003075 · Systematic review and meta-analysis of diagnostic test accuracy · MA=True · FullText=True

> Full-text scoring. Critical flaws: item 7 (no list of excluded studies with justification) and item 15 (publication bias explicitly not assessed). Strong methodology overall (QUADAS-2, dual review, appropriate meta-analytic methods) but two critical AMSTAR-2 domains explicitly unmet drives Critically Low rating per Shea 2017 algorithm.

### Cao 2021 — The Breast (Low)

PMID 34325383 · DOI 10.1016/j.breast.2021.07.015 · Systematic review and meta-analysis of diagnostic accuracy studies · MA=True · FullText=True

> Diagnostic accuracy meta-analysis of 27 studies (3,578 patients) on SLNB after NACT in cN+ breast cancer. INPLASY-registered, PRISMA-followed, QUADAS-2 RoB. Single critical flaw: no list of excluded studies with reasons (item 7). Multiple non-critical weaknesses include single-reviewer extraction, no funding of primary studies, no RoB sensitivity analysis, English-only search.

### Samiei 2021 — JAMA Surg (Critically Low)

PMID 33881478 · DOI 10.1001/jamasurg.2021.0891 · Systematic review and meta-analysis · MA=True · FullText=True

> Critical flaws on items 2 (no protocol), 9 (no RoB), 13 (no RoB discussion), 15 (no publication bias assessment). PRISMA-DTA followed for reporting. Methodologically sound pooling but lacks core AMSTAR-2 critical safeguards.

### Swarnkar 2021 — Cancers (Critically Low)

PMID 33810544 · DOI 10.3390/cancers13071539 · Systematic review with pooled analysis (proportions meta-analysis) · MA=True · FullText=True

> Pooled analysis without formal meta-analytic model, no protocol, no RoB assessment, no list of excluded studies, no duplicate selection/extraction, no publication bias test. Six of seven critical items rated No (items 2, 4 partial, 7, 9, 11, 13, 15).

### Gnant 2022 — Curr Opin Oncol (Critically Low)

PMID 36083127 · DOI 10.1097/CCO.0000000000000906 · Narrative review · MA=False · FullText=False

> Narrative opinion review (Curr Opin Oncol); not a systematic review. No search strategy, protocol, RoB assessment, or formal methods. Critical flaws on items 2, 4, 7, 9, 13. Scoring based on abstract only (full text not accessible). AMSTAR-2 designed for SRs; results indicate low methodological rigor as expected for the format.

### Lin 2022 — Ann Surg Oncol (Low)

PMID 35018590 · DOI 10.1245/s10434-021-11297-z · Systematic review and meta-analysis of diagnostic accuracy studies · MA=True · FullText=False

> Scoring based on abstract only (full text not accessible in corpus). Multiple items defaulted to 'Cannot tell' due to limited information. Critical items 2, 7, 9, 13, 15 all 'Cannot tell' creating substantial uncertainty; conservative overall rating of Low reflects inability to verify methodological rigor on critical domains. Reassessment with full text strongly recommended.

### Song 2022 — Cancer Med (Critically Low)

PMID 35502768 · DOI 10.1002/cam4.4769 · Systematic review with meta-analysis (diagnostic test accuracy) · MA=True · FullText=True

> Critical flaws on item 7 (no list of excluded studies with justification) and item 15 (publication bias explicitly not assessed). Two critical flaws -> Critically Low. Strengths include PROSPERO registration, QUADAS-2 RoB, appropriate meta-analytic methods (metaprop), and thorough exploration of heterogeneity via subgroup/meta-regression.

### Vazquez 2022 — Clin Transl Oncol (Critically Low)

PMID 36153763 · DOI 10.1007/s12094-022-02953-1 · Systematic review with meta-analysis of diagnostic test accuracy studies · MA=True · FullText=False

> Source file is PubMed abstract only; full text not accessible in corpus. Almost all items scored 'Cannot tell' due to lack of methodological detail. Per AMSTAR-2, inability to verify critical items (2, 4, 7, 9, 11, 13, 15) precludes confidence in the review; pragmatic overall rating set to Critically Low. Re-score recommended once full text is obtained via DOI 10.1007/s12094-022-02953-1.

### Banys-Paluchowski 2023 — Scand J Surg (Critically Low)

PMID 36642957 · DOI 10.1177/14574969221145892 · Narrative review · MA=False · FullText=False

> Self-declared narrative review in Scand J Surg with no formal SR methodology. Abstract-only access (no PMC); full text unavailable. Critical items 2, 4, 7, 9, 13 all scored No. Items 11/12/15 inapplicable (no meta-analysis). Useful as topical narrative on cN+→ycN0 axillary staging but does not meet AMSTAR-2 SR criteria.

### Ma 2023 — Front Oncol (Critically Low)

PMID 37064127 · DOI 10.3389/fonc.2023.1167912 · Systematic review with meta-analysis · MA=True · FullText=True

> Meta-analysis of 9 observational studies on ypN+ rate in cN0 bpCR patients by subtype. PROSPERO registration discrepancy: abstract cites CRD42022351739, methods cites CRD42014012901. Limited search (2 DBs, English-only), no excluded list with reasons, no PubBias, no RoB impact discussion produce 4 critical flaws.

### Pantiora 2023 — Br J Surg (Critically Low)

PMID 36560842 · DOI 10.1093/bjs/znac426 · Systematic review and meta-analysis · MA=True · FullText=True

> Two critical flaws: item 2 (no protocol/registration explicitly stated) and item 7 (no list of excluded studies with justifications). Otherwise methodologically robust meta-analysis with appropriate statistical methods, multiple RoB tools, and GRADE certainty ratings.

### Vasigh 2024 — Breast J (Critically Low)

PMID 39742366 · DOI 10.1155/2024/8866756 · Systematic review and meta-analysis of prospective and retrospective studies (including RCTs and cohorts) · MA=True · FullText=True

> Critical flaws on items 2 (no protocol), 9 (CASP instead of validated RoB tool), 13 (RoB not discussed), and 10 (funding of primary studies). Items 4, 11, 15 met (partial for 4). Pooled proportions meta-analysis with appropriate random-effects, heterogeneity exploration and publication bias testing are strengths. Major weakness is absence of formal RoB assessment, which propagates to items 12 and 13.

### Yu 2024 — Breast Cancer (Auckl) (Critically Low)

PMID 38826850 · DOI 10.1177/11782234241255856 · Systematic review with meta-analysis of diagnostic accuracy studies · MA=True · FullText=True

> Confirmed: Yu 2024 performs funnel plot + Egger test for both IR and FNR (item 15 = Yes). Critical flaws on items 2 (no protocol), 7 (no excluded-studies list), 13 (RoB not discussed). Item 4 Partial Yes (no grey literature) and item 9 Partial Yes (QUADAS-2/MINORS used). Search restricted to English may bias findings.

### de Wild 2024 — Br J Surg (Critically Low)

PMID 38531689 · DOI 10.1093/bjs/znae071 · Systematic review with random-effects meta-analysis of proportions (metaprop) · MA=True · FullText=True

> Narrative SR with random-effects MA of proportions (metaprop) for pooled IR, so items 11/12/15 apply. Four critical flaws: item 2 (no protocol/registration, explicitly stated), item 7 (excluded studies counts only, no reference list), item 9 (NIH tool not satisfactory RoB instrument, single rater), item 15 (no publication bias assessment). High-impact BJS review but methodological reporting gaps drive Critically Low rating.

### Boland 2025 — Eur J Surg Oncol (Low)

PMID 40009931 · DOI 10.1016/j.ejso.2025.109684 · Systematic review with meta-analysis (observational studies) · MA=True · FullText=False

> Scoring based on PubMed abstract only; full text not accessible. Most items scored 'Cannot tell' due to lack of methodological detail. Overall confidence rated 'Low' as a conservative estimate given inability to verify critical items (protocol, search, excluded studies list, RoB-MA discussion, publication bias). True rating requires full-text review.

### Ding 2025 — J Cancer Res Ther (Low)

PMID 40317154 · DOI 10.4103/jcrt.jcrt_1279_24 · Systematic review with meta-analysis · MA=True · FullText=False

> Scoring based on abstract only; full text not available. Critical item 4 (search) scored No due to limited databases. Most items scored Cannot tell due to limited information. True confidence level cannot be determined without full text.

### Man 2025 — Breast Cancer (Critically Low)

PMID 40186790 · DOI 10.1007/s12282-025-01692-9 · Systematic review and meta-analysis · MA=True · FullText=True

> Meta-analysis of pooled proportions for axillary recurrence and survival after de-escalated axillary surgery post-NST. Critical flaws on items 2 (no protocol), 7 (no excluded studies list), 10 (no funding), 15 (no publication bias).

### Wang 2025 — Oncology (Moderate)

PMID 39899993 · DOI 10.1159/000543662 · Systematic review with meta-analysis · MA=True · FullText=False

> Abstract-only assessment; full text paywalled. No confirmed 'No' on any critical item (2/4/7/9/11/13/15); critical items 4 and 11 rated Partial Yes (not flaws), items 13 rated Yes, others Cannot tell. Per Shea 2017: 0 critical flaws + >1 non-critical weakness = Moderate. Rating may shift downward upon full-text review.

### Abdulla 2026 — Ann Surg Oncol (Critically Low)

PMID 41545613 · DOI 10.1245/s10434-025-19070-2 · Systematic review and meta-analysis · MA=True · FullText=False

> Assessment based on PubMed abstract only; full text not accessible. Most items scored Cannot tell. Item 15 scored No (critical flaw) as publication bias not addressed. With 12 Cannot tell ratings the effective evidence base is severely limited; conservatively rated Critically Low given multiple methodological reporting gaps.
