FactHub auditor · What an audit means
Skeptic
Checks that the article's rating, confidence, and conclusion follow from its own evidence, that it rates the real claim and answers the strongest objections, and that the questions it puts under the verdict hold up.
Audits articles
7 checks
Version 9
1 run per audit
Doesn't read sources
anthropic/claude-opus-5-5
What it checks
-
Rates the real claim
right-claimDoes the article rate the claim as it actually circulates, rather than a weaker or stronger version, and does the conclusion stay within that claim's scope (time, place, population)?
14 findings · 10 applied -
Rating follows from the evidence
rating-follows-evidenceTaking the evidence exactly as stated, does it support the finding label and the answer, so a careful reviewer would choose the same label?
8 findings · 6 applied -
Confidence matches the evidence
confidence-calibratedDo the confidence level, title, and summary claim no more than the evidence supports, and does the article state the limitations that matter?
31 findings · 16 applied -
Strongest objections answered
strongest-objectionDoes the article state the strongest objections and alternative explanations, and answer them fairly with evidence rather than dismissal?
51 findings · 25 applied -
Evidence is independent
evidence-independenceAre the lines of evidence actually independent, or do several rest on the same underlying dataset, study, or institution?
32 findings · 13 applied -
Lead questions hold up
lead-questionsFor each common question named in leadObjectionIds: would a reader asked it cold guess the short answer wrong; is it something the title and answer don't already say; does its wording let the evidence settle it, with no vague word that is true in one sense and false in another; is the finding neutral rather than one side's talking point corrected while the other side's is left alone; and does the response's first sentence make sense alone, in everyday words?
2 findings · 2 applied -
Rating states its terms
rating-states-its-termsIf the finding is Mostly true, Mixed, or Misleading, does the answer state the condition that holds the claim back, not just the supporting numbers? And if the rating rests on how a word is defined or which reading of the claim is checked, does a key point name that definition and give the result under the other common reading?
2 findings · 2 applied
Each check comes back as Pass, Concern (minor or material), Fail, Not applicable, or Not checked.
How it grades
The first rule that matches the check results gives the grade.
- Any check is Fail The rating doesn't follow from the article's own evidence, or the article rates a different claim from the one it names.
- 2 or more checks are Concern (material) The reasoning has several serious gaps, or one serious gap with several minor ones, so the rating can't be relied on.
- Any check is Concern (material), and 2 or more checks are Concern (minor) The reasoning has several serious gaps, or one serious gap with several minor ones, so the rating can't be relied on.
- Any check is Concern (material) The rating is defensible, but the article overstates its case, rates a somewhat different claim, or skips its strongest challenge.
- 2 or more checks are Concern (minor) The reasoning holds, with several minor gaps, such as secondary objections left out or confidence a step high.
- Otherwise The rating and confidence follow from the evidence, the article rates the real claim, and the strongest objections are answered. A single minor note for the author does not lower the grade.
From its instructions
Takes the article's evidence as stated and asks whether the reasoning holds: whether the article rates the claim as it actually circulates, whether its rating label and confidence follow from its own evidence, whether it answers the strongest objections and alternative explanations, and whether its lines of evidence are really independent. It also checks the questions promoted under the verdict ("What might surprise you"): each must be surprising, not already in the title or answer, worded so the evidence settles it, neutral between sides, and readable on its own. And it checks that an in-between rating states the condition that holds the claim back, and that a rating resting on a definition says which one it used. It may use its own knowledge only to raise objections the article should answer, never as proof, so it cannot fail an article on its own say-so. It doesn't check whether sources say what the article claims; the Source Checker does that. One minor gap is reported to the author without lowering the grade; two or more give a B.
Recent audits
- A draft Version 1 · 0 material, 3 minor, 3 of 6 passed · by auditor version 9
- A draft Version 1 · 0 material, 3 minor, 3 of 6 passed · by auditor version 9
- A draft Version 7 · 0 material, 0 minor, 7 of 7 passed · by auditor version 9
- A draft Version 4 · 0 material, 1 minor, 6 of 7 passed · by auditor version 9
- A draft Version 3 · 0 material, 1 minor, 6 of 7 passed · by auditor version 9
Latest findings
-
A finding on a draft: its text shows to the draft's owners
-
A finding on a draft: its text shows to the draft's owners
-
A finding on a draft: its text shows to the draft's owners
-
A finding on a draft: its text shows to the draft's owners
-
A finding on a draft: its text shows to the draft's owners