Building a Claude Code skill to fact-check scientific claims

The problem

"Spirulina is a superfood." "Coffee causes cancer." "Intermittent fasting cures everything."

Checking claims like these means going to PubMed, finding the meta-analyses, cross-referencing official positions, and verifying whether the studies were industry-funded. Two hours of work per claim, minimum.

Hence the idea: type /science-check is spirulina a superfood in Claude Code and get a full report in two minutes — verdicts, evidence levels, risks, verified sources.

What it produces

The skill launches 3 research agents in parallel (meta-analyses, risks, critical analysis), searches PubMed, Semantic Scholar and Google Scholar through MCP servers, validates key studies (retractions, funding, sample size), and produces a structured report with per-sub-claim verdicts.

On /science-check is spirulina a "superfood"?:

══════════════════════════════════════════════════════════════════════
 SCIENTIFIC VERIFICATION
══════════════════════════════════════════════════════════════════════

Claim: Spirulina is a "superfood"
Overall verdict: PARTIALLY CONFIRMED

── Scientific consensus ───────────────────────────────────────────

Spirulina has real, documented benefits, mainly on lipid profile and
blood pressure, supported by multiple RCT meta-analyses. However, the
term "superfood" has NO official scientific definition — it's a
marketing term. The confirmed benefits are modest and targeted.

── Sub-claim breakdown ────────────────────────────────────────────

Claim                             │ Verdict              │ Evidence level
──────────────────────────────────┼──────────────────────┼──────────────────────────────────
Lowers cholesterol/LDL/TG         │ CONFIRMED            │ Meta-analysis 20 RCT (n=1076)
Lowers blood pressure             │ CONFIRMED            │ RCT meta-analysis
Helps weight loss                 │ Partly confirmed     │ Meta-analysis 17 RCT: −1.07 kg
Glycaemic control (T2 diabetes)   │ Partly confirmed     │ Meta-analysis, 8 studies
High in protein ("60%")           │ OVERSTATED           │ 3 g serving = 2 g protein
Source of vitamin B12             │ REFUTED              │ Inactive pseudo-B12
Detoxifies the liver              │ UNPROVEN             │ No proven mechanism
Anti-cancer                       │ PREMATURE            │ In vitro only
"Superfood" (the term)            │ NOT APPLICABLE       │ Purely marketing

── Risks and side effects ─────────────────────────────────────────

Risk                                  │ Severity         │ Population
──────────────────────────────────────┼──────────────────┼───────────────────────────────
Heavy metal contamination             │ Moderate-high    │ All (94% products contaminated)
Microcystins (cyanotoxins)            │ High             │ Intake ≥ 4 g/day
Autoimmune conditions                 │ Moderate-high    │ Lupus, MS, vitiligo, RA
Anticoagulant interactions            │ Moderate         │ On warfarin/aspirin
Phenylketonuria (PKU)                 │ Very high        │ Absolute contraindication

── Official positions ─────────────────────────────────────────────

ANSES (France) : Safe at moderate doses. CI: PKU, allergies.
                 Heavy metal contamination warning. (2017)
FDA (USA)      : GRAS status. Approved food colouring.
EFSA (EU)      : REJECTED the diabetes claims (2013).

── Red flags ──────────────────────────────────────────────────────

⚠ "Superfood" = no official scientific definition
⚠ Claims-to-evidence ratio ~10:1 (50+ claims, <5 proven)
⚠ Market $630M → $1.4B (publication bias likely)
⚠ 1 retracted study: "Spirulina Unleashed" (MDPI, 2024)
⚠ Weasel language: 36x "may/might/suggest", 0x "proven"
⚠ 94% of samples positive for microcystins

── Sources (14 consulted) ─────────────────────────────────────────

 [1] Spirulina & lipid profile — Meta-analysis 20 RCT (2023)
 [2] Spirulina & blood pressure — RCT meta-analysis (2021)
 [3] ANSES — Regulatory position (2017)
 ...

Installing the MCP servers

The skill uses two MCP servers to hit the scientific databases directly. Without them it falls back to WebSearch — workable, but less precise.

• PubMed MCP (mcp-simple-pubmed): direct access to the Entrez API. Free, just needs an email. Test: uvx mcp-simple-pubmed --help
• Paper Search MCP (paper-search-mcp): multi-source search (PubMed, arXiv, bioRxiv, medRxiv, Semantic Scholar, Google Scholar). Test: uvx --from paper-search-mcp python -m paper_search_mcp.server --help

In ~/.claude/mcp.json:

{
  "mcpServers": {
    "pubmed": {
      "command": "uvx",
      "args": ["mcp-simple-pubmed"],
      "env": {
        "PUBMED_EMAIL": "your@email.com"
      }
    },
    "paper-search": {
      "command": "uvx",
      "args": ["--from", "paper-search-mcp", "python", "-m", "paper_search_mcp.server"],
      "env": {
        "SEMANTIC_SCHOLAR_API_KEY": ""
      }
    }
  }
}

A few points:

• PUBMED_EMAIL: NCBI's Entrez API wants an email to identify requests, no API key.
• SEMANTIC_SCHOLAR_API_KEY: optional, it works without it at a lower rate limit. Free key at semanticscholar.org/product/api.
• Both run via uvx. If you don't have uv: curl -LsSf https://astral.sh/uv/install.sh | sh.

Restart Claude Code after editing mcp.json: MCP servers load at startup, not hot.

Architecture: 4 files, not one big one

The skill lives in ~/.claude/skills/science-check/ (see on GitHub):

science-check/
├── SKILL.md              # Main instructions (105 lines)
├── REPORT_TEMPLATE.md    # Report template
├── TRUSTED_SOURCES.md    # Trusted sources by tier
└── EVIDENCE_HIERARCHY.md # Evidence-level grid

The first version was one 250-line file: Claude lost the thread, mixed up workflow phases and skipped report sections. Claude Code loads the whole SKILL.md when the skill triggers, and every token competes with conversation history. Moving references into separate files — progressive disclosure — changed everything.

The YAML frontmatter tells Claude when to trigger the skill, the markdown body how to run it:

name: science-check
description: 'Verifies a scientific or health claim by cross-referencing
  PubMed, Semantic Scholar, and the web. Produces a structured report with
  evidence levels, risks and official positions. Use this skill whenever the
  user asks about health, nutrition, a supplement, a drug, a therapy, or asks
  whether a scientific claim is true, proven or reliable, even if the question
  is informal.'
user-invocable: true
argument-hint: '[claim to verify]'
allowed-tools:
  - Agent
  - Bash
  - Read
  - WebSearch
  - WebFetch
  - AskUserQuestion
  - Write
  - mcp__pubmed__search_pubmed
  - mcp__pubmed__get_paper_fulltext
  - mcp__paper-search__search_pubmed
  - mcp__paper-search__search_arxiv
  - mcp__paper-search__search_google_scholar
  - mcp__paper-search__search_biorxiv
  - mcp__paper-search__search_medrxiv
  - mcp__paper-search__read_pubmed_paper
  - mcp__paper-search__read_biorxiv_paper
  - mcp__paper-search__read_medrxiv_paper

Three lessons on that frontmatter:

The description must be "pushy". Claude tends to under-trigger skills: asked "does magnesium help you sleep?", it answers directly instead of using the skill. Explicitly listing "nutrition, supplement, drug, therapy" forces the trigger.

allowed-tools and full MCP names. The costliest trap: agents launch, everything looks normal, but they never touch PubMed. No error, no warning, just silence. The format is mcp__<server_name>__<tool_name>, and without declaring them in allowed-tools, Claude simply isn't allowed to use those tools inside the skill's context.

Agent in the list. That's what allows 3 searches in parallel instead of in sequence: ~1 minute instead of ~3.

The 6-phase workflow

Phase 1, translation. Scientific databases are in English: "is spirulina good for health" becomes "spirulina health benefits evidence".

Phase 2, orientation with 3 parallel agents. Agent A hunts meta-analyses and systematic reviews (highest evidence level), Agent B risks and side effects (the counterweight usually missing from benefit-oriented searches), Agent C critical analyses and debunking (reducing confirmation bias).

Phase 3, deep dive. Claude fetches the best sources following a reliability ranking defined in TRUSTED_SOURCES.md:

• Tier 1: Cochrane Library, PubMed, Examine.com
• Tier 2: EFSA, FDA, ANSES, WHO
• Tier 3: Harvard Health, Mayo Clinic, McGill OSS, NHS
• Tier 4: Retraction Watch, Semantic Scholar (citation counts)

The paper-search MCP pulls citation counts directly from Semantic Scholar, giving a signal on a study's real impact.

Phase 4, cross-validation. For each key study: sample size, study type (RCT, observational, animal, in vitro), funding (industry = potential bias), replication of results, and absence of retraction via Retraction Watch.

Phase 5, self-check. A quality checklist runs before writing: at least 3 independent sources, at least 1 meta-analysis or systematic review (otherwise flagged in the report), no conclusion based on a single study, risks and side effects identified. If a criterion fails, Claude runs targeted follow-up searches. Without this phase it sometimes concluded "CONFIRMED" from a single 30-person RCT.

Phase 6, synthesis with ultrathink. The ultrathink keyword in SKILL.md activates extended thinking. The synthesis has to weigh contradictory evidence (positive vs negative meta-analyses, EFSA/FDA disagreement) and produce a weighted overall verdict, following REPORT_TEMPLATE.md and the EVIDENCE_HIERARCHY.md grid.

Testing

Restart Claude Code, then:

/science-check does intermittent fasting help with weight loss

Claude shows a progress checklist, launches 3 background agents, runs the cross-checks and outputs the report. Expect 1 to 2 minutes depending on complexity.

What this teaches

The description is 80% of the work. Those three lines of YAML take more iteration than the entire workflow. If Claude doesn't trigger the skill, nothing else matters.

Subagents change everything. Three parallel agents instead of sequential searches: a third of the time, and better quality because each agent has its dedicated angle. Without Agent in allowed-tools, it runs sequentially without telling you.

Self-check isn't optional. It's the phase that stops a "CONFIRMED" verdict based on an in vitro study of 12 mice.

Silent MCP debugging is the real trap. When an MCP tool is missing from allowed-tools, no error is raised: Claude just proceeds without it.

Limitations

It doesn't replace a doctor. The report is only as good as the sources available online, and Claude can misread a study. But for a first pass — "is this worth raising with my doctor?" — it's effective.

The pattern transfers to any domain requiring claim verification (health, nutrition, but also finance, law, tech): multi-angle parallel search, cross-validation, self-check, structured report. The skill changes, the skeleton stays.