no key · no human
Home › What is inside the corpus › Methodology

Methodology

Every step from a compressed crawl archive to a searchable article, with the trade-offs named. If a number here looks suspicious, this page should tell you why.

1. Fetch

Our upstream archives publish a new batch roughly every hour. A cron job checks the index each hour, downloads anything new, and records what it has processed so a restart never reprocesses or skips a file.

One file is about 1 GiB compressed and holds 15,000–20,000 crawled pages. Downloading and parsing takes about 90 seconds.

Files are diffed by the 14-digit timestamp in the filename, not by the sequence number. The sequence number resets when the crawler restarts — it went from 369 back to 0 on 7 August 2026 — and anything keying on it silently reprocesses a month of data.

2. Extract

Each record is parsed for the article body, with boilerplate — navigation, footers, related-article rails, cookie banners — removed. Metadata is read from JSON-LD NewsArticle markup where present, falling back to Open Graph tags, then to standard meta tags.

A page is kept only if it looks like an article: enough body text, a headline, and a plausible publication date. Section fronts, tag pages and paginated listings are dropped.

3. Date

Publication time is taken from the first available of: JSON-LD datePublished (86% of articles), an article:published_time meta tag (4%), a date parsed out of the body or URL (9%), and finally the crawl time (<1%). Each article records which, in date_source.

Future dates are clamped. Publishers occasionally emit a date days or weeks ahead. Left alone, a single article dated next month sits at the top of every date-sorted query forever. Anything more than a day past its crawl time is clamped to the crawl time and labelled crawled_clamped.

4. Language

Read from the publisher's own markup, in order: <html lang>, then JSON-LD inLanguage, then og:locale. A value is present on 99.9% of articles.

Known defect. Present is not the same as correct. Multilingual publishers frequently declare one site-wide language on every page, so their English and Russian editions carry the site's primary language. Anadolu Agency (aa.com.tr) is a clear example: its /en/ and /ru/ articles are all tagged tr.

Detection from the body text, which would fix this, is on the roadmap. Until then read lang as "the language this publisher declares" rather than "the language of this article", and prefer tld or host when you need certainty.

5. Country

The hardest field, and the one everybody else reports as a fact. Resolved by cascade, best signal first:

  1. Country-code top-level domain. .ua → UA. Nearly always right.
  2. Publisher markup. addressCountry in the publisher's JSON-LD organisation block. Reliable when present.
  3. Locale region subtags from og:locale, <html lang> or inLanguage: pt_BR → BR.
  4. Language mapping, where a language maps to one dominant country. Weak, and labelled as such.

The cascade resolves about 85% of articles. Every one records which rung it landed on in country_source.

The en-US problem

A large share of Indian, Nigerian and Filipino publishers ship <html lang="en-US"> as an untouched template default. Taken literally, that inflates the United States and erases those countries.

We keep the value but label it weak_en_us, so you can decide. Passing strict_country=true excludes it. We chose labelling over dropping because some of those articles really are American, and silently discarding them would be its own distortion.

6. Deduplicate

Two passes. By URL, so a page recrawled in a later file does not create a second article — the document id is the MD5 of the URL, and re-inserting is a no-op. And by body-text fingerprint within a batch, which removes about 10% of what survives extraction: wire copy republished under different URLs by the same publisher.

Cross-publisher deduplication is not done. When thirty outlets run the same agency story you get thirty articles, because for measuring coverage that is the honest answer. Group by host yourself if you want one voice per publisher — there is a recipe in RAG. Story clustering is on the roadmap.

7. Filter

A blocklist removes hosts that are technically in the news crawl but are not news: automated financial-quote farms that emit thousands of near-identical pages, and template sites like railway-timetable generators that produce one "article" per route. Left alone, a handful of these dominate the publisher rankings and drown real journalism.

Roughly 40% of crawled pages are discarded in total across all filters.

8. Index

Articles go into Elasticsearch in bulk with op_type=create, so an article that already exists is skipped rather than overwritten — reprocessing a file is safe. Indices roll over weekly and are deleted after 30 days, replicated to all three nodes so any node can answer any query.

Indexing 10,000 articles takes about 12 seconds. The full cycle per file — download, extract, index — runs in roughly 99 seconds, and you can watch it happen on snapshots.

Known limitations

Collected on one page, with numbers: quality.