Data quality
Every dataset has flaws. Most vendors do not publish theirs. Here are ours, with numbers, because an agent building an analysis on this deserves to know where the floor is soft.
Field reliability
| Field | Coverage | How much to trust it |
|---|---|---|
url, host, tld | 100% | Facts. Read from the URL. |
title | 100% | Reliable. Occasionally includes the publication name where the markup does. |
text | 100% | Good. Boilerplate removal is imperfect: a small share retain a trailing related-articles list or a subscription prompt. |
lang | 99.9% present | Declared, not detected. Read from publisher markup, so multilingual sites mislabel their non-primary editions. |
published_at | 100% present | Exact to the minute for 84%. The rest are day-accurate or crawl-time fallbacks.
Check date_source. |
country | 85% | Inferred. Check country_source before
counting. |
sitename | 91% | Whatever the publisher declares. Not normalised: the same outlet may appear under two spellings. |
image | 88% | A link to the publisher's server. May 404 later; we do not copy images. |
author | 46% | Patchy. Often an agency name or a desk rather than a person. |
categories | 38% | Not a taxonomy. Publisher section labels, unnormalised across publishers. A hint at best. |
The five things that will bite you
1. Absence is not evidence
If a search returns nothing, that means our sources did not capture an article matching your query. It does not mean the event did not happen. Coverage is broad but not exhaustive, and it varies by publisher and by day. This is the single most common way this data gets misread — especially by language models, which will happily conclude that nothing happened.
2. Country is a guess for about a third of articles
Fifteen percent have no country at all. Of those that do, the fraction derived from an
en-US template default is material and skews towards South Asian, Nigerian
and Filipino publishers. For any country-level count, use
strict_country=true and accept a smaller, cleaner sample.
3. Wire copy inflates counts
One agency story republished by forty outlets is forty articles. If you count articles
to measure how big a story is, you are partly measuring syndication. Count distinct
host values instead — that measures independent publishers.
4. Publication times are self-reported
Publishers backdate, timezone handling varies, and some emit future dates (we clamp
those). Use crawled_at when you need a timestamp that cannot be manipulated,
particularly for polling — it only moves forward.
5. Language is declared, not detected
The lang field comes from the publisher's markup. Multilingual news
agencies commonly declare one language across the whole site, so filtering
lang=en misses their English editions and lang=tr returns
articles written in English and Russian. Verified on aa.com.tr, where every
/en/ and /ru/ article is tagged tr.
If your slice must be linguistically clean, filter by tld or
host as well, or check the text yourself. Text-based detection is on the
roadmap.
6. Coverage is skewed by what the archives reach
Large publishers with clean HTML are over-represented. Sites behind hard paywalls,
heavy JavaScript rendering or aggressive bot blocking are under-represented or absent.
Publishers who block automated crawlers in robots.txt are largely absent.
This is a structural bias in the sources, not something we can correct.
What we filter out, and why that is also a bias
A blocklist removes automated content farms — financial quote generators, timetable templates — that emit thousands of near-identical pages and would otherwise top every publisher ranking. This is a judgement call. It makes the dataset more useful and slightly less raw. If you need the unfiltered stream, the source is public: run the pipeline yourself.
Text extraction failures
Extraction is heuristic. Known failure modes: liveblogs come out as one long
concatenation; photo essays yield captions only; articles split across paginated pages
give you page one; some paywalled pages yield only the free preamble. There is no flag for
these — if your use case is sensitive to it, check text length.
Report an error
If an article is mislabelled, tell us which and how — [email protected]. Country and date errors are the most common, and reports are the fastest way we find systematic extraction bugs.
