# Quality report

# Seattle Times tests H and R: two coders compared (2026-11-16)

- **Coder 1:** `../codebook-v2-coding.csv`.
- **Coder 2:** blind. It saw a copy of the codebook with the "why it matters" line removed and could open only the story files (`coding-coder2.csv`).
- **Stories:** 35 matched (coder 2 also coded the Aug 27 AAP story).

| Field | Agreement | Kappa | How to read it |
|---|---|---|---|
| r_trigger (story discusses advantage or fairness) | 97% | 0.93 | Robust |
| r_pass (says or implies WA required no hormones) | 100% | 1.00 | Robust: neither coder found it stated outright in the paper's own voice. Coder 1 called 3 stories "partial"; coder 2 counted 2 of them as an implied pass. |
| mirror trigger | 94% | 0.87 | Robust |
| mirror pass (gives the small number of trans athletes) | 91% | 0.82 | Robust: both coders found it in 10 stories |
| harm_lead | 49% | 0.30 | **Not robust** |
| harm_focus | 63% | 0.48 | **Not robust** |

**Why harm didn't hold up.** Coder 1 counted a plain description of the ban ("would bar trans girls...") as harm to trans students. Coder 2 did not. Under coder 2's stricter rule, the leads come out neither 24, trans 5, both 4, girls 3, and the overall calls lean slightly toward girls (12 vs 10). So the "trans 15 vs girls 2" lead result came mostly from that convention. **Don't publish it.**

**What holds up:**
- The no-hormones fact is never stated outright, while the "few athletes" fact appears in about 10 of 24 stories.
- Plus the earlier items: the scope errors, the two own-voice science lines (Nov 8 and Nov 13), and hedged checks of the exam claim against flat checks of Yes's claims.

**Next step for harm:** split the test in two. (a) Is describing the ban counted as harm? Write that down as an explicit rule. (b) Use a narrower count that only includes a **named person's** described harm in the lead. Then recode with both coders.


# Merge decisions and reconciliations

## Decisions made in the merge

1. **Reruns** are matched to their original by headline similarity, date (±4 days) and source. Each unique story counts once, credited to the newsroom that wrote it. 64 copies were folded in; 21 Times stories ran in 2 to 5 papers each. One copy was joined by hand: Yakima's June 2025 "schools agency worker" story is the WA State Standard story the Times also ran.
2. **Rerun-copy items:** omission rows coded on a copy are dropped, because the tests live on the canonical story row. A statement that duplicates one on the canonical copy is dropped. Anything else is kept and flagged `rerun copy`.
3. **Excluded:** one blocked Kitsap story whose relevance can't be checked.
4. **Reconciled from earlier verified work:** three Times scope errors the v3 coder missed:
   - the Jan 23 headline, "bar trans girls from school sports";
   - the Nov 9 headline, "ban on trans athletes";
   - the Oct 30 line, "all student athletes" (the v3 coder had coded it as OTHER, so it is recoded to SCOPE rather than added twice).

   All three are still on the pages. A story with its own scope error is now `partial` on p_scope, not a pass (the Times was 33/33 before, 28/33 after).
5. **Coded by me at merge time:** the 4 McClatchy reruns and 1 uncoded Spokesman story (a single coder; marked `merge coder`). One McClatchy story names a high school trans runner, so that paragraph is never quoted.
6. **"Typical sports physical" line (reverted 2026-11-17):** the Times' line that confirmation "can come from a typical sports physical's examination of… genetic makeup" (Sept 29, Oct 30, Oct 31) was briefly coded misleading, favoring Yes, then reverted to **accurate**. It tracks the measure's own wording ("as part of the required routine sports physical examination"), and FactHub rates the matching Yes claims Mostly true (wa-i638-yes-campaign-claims, wa-i638-pediatric-care-verifies-sex). No own-voice error favoring Yes was found in the Times' news stories.
7. **Fixes after the viz mockups (2026-11-16):**
   - **Rerun headlines:** a reprint that kept a headline coded as an error carries it, charged to the paper that printed it (`source` = `rerun headline`). This adds 4 items: the Jan 23 "bar trans girls from school sports" headline in the Columbian, Spokesman-Review and Yakima, and the Nov 9 "ban on trans athletes" headline in the Columbian.
   - **"What's true" notes:** the ST38 and ST04 notes no longer cite the Air Force "12% faster after two years" run figure. A larger 2023 follow-up found the run gap closed by two years, and FactHub's trans-advantage article v5 corrected this.
   - **Blank source cells:** the merge-coded stories now have zeros.



# Quote check

Every statement quote was matched, after normalizing quote marks and spaces, against the saved text of the story it came from. All matched.
