Index schema
How the search index is built. Useful if you are tuning queries, or if you are building something similar and want the numbers.
Field mapping
| Field | Type | Searchable | Notes |
|---|---|---|---|
title | text | full-text, boost 3 | Standard analyser |
description | text | full-text, boost 2 | |
text | text | full-text, boost 1 | Excluded from responses unless full_text=true |
url | keyword | exact only | Used by /v1/article |
host, tld, country, lang |
keyword | exact, aggregatable | Eager global ordinals for fast faceting |
published_at, crawled | date | range, sort | UTC |
country_source, date_source | keyword | exact | Provenance labels |
sitename, author | keyword | exact | Not normalised across publishers |
categories | keyword[] | exact | Publisher's own labels |
image | keyword | not indexed | Stored only |
The mapping is dynamic: strict: a document with an unexpected field is
rejected rather than silently adding a field. That is what keeps the schema stable enough
to publish on a page like this.
Search behaviour
Queries are multi_match of type best_fields with
operator: and — every term must appear in the article. That is why a
natural-language question returns nothing and two or three keywords work. Scoring is BM25
across the three text fields with the boosts above.
Language-specific analysers are on the roadmap. Today
everything uses the standard analyser, which means no stemming: searching for
election will not match elections. Search for the form you
expect to see in a headline.
Cluster shape
| Property | Value |
|---|---|
| Nodes | 3 |
| Shards per index | 1 primary, 2 replicas |
| Effect | A full copy on every node; any node answers any query |
| Rollover | Weekly, or at 25 GB |
| Retention | 30 days, then deleted |
| Codec | best_compression |
| Index sort | published_at descending |
| Storage per day | ~540 MiB per copy |
Two choices worth explaining
Weekly indices, not daily
Daily indices would give tidier retention, but 30 of them means 30 shards touched by
every unfiltered query. The search thread pool is 7 threads per node, 21 across the
cluster; 30 shards saturate it with a single request. Weekly rollover keeps the shard
count at four or five, and index sorting plus can_match pruning skips the
ones that cannot contain results.
Two replicas, not one
Two replicas is more storage than durability needs. It is a throughput decision: with a copy on every node, all three can serve any query, which is what turns 400 requests per second into the measured 1,247.
Reproduce the measurements
curl "https://freenewsapi.ai/v1/stats?top=100" # every facet, exact counts\ncurl "https://freenewsapi.ai/health" # live document count