← All work
Live

Fake-News Detector that can say "I don't know"

Answers REAL, FAKE or UNCERTAIN with a confidence score, and checks claims against live search when quota allows.

FastAPI · Gemini · Google Search grounding · Pydantic

  • 27 s to about 4 sSource: Live demo response time before and after model fallback
  • 3 answers, not 2Source: REAL, FAKE or UNCERTAIN; detector.py

The problem

The BERT classifier we started with had only seen old news, so it flagged genuine recent stories as fake. A detector that is confidently wrong about today is worse than none.

What was built

  • Replaced the classifier with an LLM that returns REAL, FAKE or UNCERTAIN plus a confidence score, as structured output.Source: detector.py and the README in the NLP-AAT repository open
  • Quota-aware search grounding: when the free key has no search quota, the page says the live check is paused, and does not fail silently.Source: Commit 7c79bbf; visible on the live demo open
  • Multi-model fallback on overload, which cut live response time from 27 seconds to about 4.Source: Timed against the live demo before and after commit 7c79bbf
  • An evaluation harness on the ISOT dataset that scores abstentions separately from wrong answers.Source: evaluate.py in the NLP-AAT repository

What this does not show

  • On a free API key the live web check is often paused, and the demo tells you when it is.
  • I do not quote an accuracy figure. The harness exists, and a number would only mean something on news newer than the model.

Receipts

ClaimWhere it comes from
27 s to about 4 sLive demo response time before and after model fallback
3 answers, not 2REAL, FAKE or UNCERTAIN; detector.py
Replaced the classifier with an LLM that returns REAL, FAKE or UNCERTAIN plus a confidence score, as structured output.detector.py and the README in the NLP-AAT repository open
Quota-aware search grounding: when the free key has no search quota, the page says the live check is paused, and does not fail silently.Commit 7c79bbf; visible on the live demo open
Multi-model fallback on overload, which cut live response time from 27 seconds to about 4.Timed against the live demo before and after commit 7c79bbf
An evaluation harness on the ISOT dataset that scores abstentions separately from wrong answers.evaluate.py in the NLP-AAT repository
NextGitHub compliance evidence collector