Sources

One upstream source, thousands of publishers, no licensing deals. Here is exactly what that means and who is actually in the index.

Where the data comes from

Large-scale public news archives — bulk datasets of news pages that have already been crawled and released as open data, published continuously in hourly batches.

We process each release as it appears. We do not fetch pages from publishers' servers and we hold no content licences. The upstream datasets are public and anyone can build the same thing — the work is in the pipeline, not in the access.

More sources are being added. The API surface does not change when they are: the same fields, the same filters, and the same labelling of which values were inferred rather than observed.

Most active publishers right now

Not a curated list — the crawl's own ranking. Presence here says a publisher was crawled often, nothing about its quality or credibility.

Who is missing, and why

• Hard paywalls. If the article body is not in the HTML, we get the preamble or nothing.
• JavaScript-rendered sites. The archives capture HTML as served. Sites that assemble the article client-side yield an empty shell.
• Bot blocking. Publishers who challenge or block crawlers are absent.
• robots.txt exclusions. Publishers who disallow automated crawlers are absent by their own choice, and we consider that settled.
• Content farms we removed. A blocklist drops automated quote generators and timetable templates. See methodology.

Publishers

If you publish news and want your site out of this index, it takes one email and no argument: for publishers.