Ranked timeline hydration

The Redis-backed Hot and Active paths in FanoutTimelineEndpointService now select ranking columns first, check eligibility in bounded batches in rank order, and load the original full graph only for the selected page. Home and Hybrid opt into the explicit TimelineFilterNote callback contract. Redis keys, contents, cache sizes and score calculations do not change. Local ranked, Global and forum database paths, including active-timeline-candidates.ts, are unchanged.

Dependency map

timeline-filter-note.ts contains both the structural contract and the selected columns. Relation primary keys are included so TypeORM can construct the selected relation objects. The root author remains an inner join; missing authors are rejected just as before. Optional missing relations remain null. Unit mocks strip unselected columns, and the database fixture exercises the real TypeORM projection. Because a field declared on the contract but absent from the SQL list would read as undefined and silently hide or reveal notes rather than fail, timelineFilterShape restates the contract as data through an Exhaustive<T> mapped type: a field added to TimelineFilterNote and not listed there is a compile error. timelineFilterSelection is then derived from that shape, so there is no second list to keep in step. Three unit tests in the ordinary suite (no database required) pin the derivation: every declared field is selected, nothing is selected beyond the contract except the relation primary keys TypeORM needs, and the resulting column set still matches the one the original query used.

The own-note shortcut applies only where the original filter put it. It does not bypass the outer media/reply filters, mute/block/instance checks or viewer visibility. Home's blocked-hashtag exception and Hybrid's different callback are retained separately.

Selection and consistency

The candidate query reads all cached IDs with only five ranking columns. It applies the original window and cursor expressions and sorts with the original comparator. Eligibility queries read between 32 and 100 roots per batch, depending on page size. Rejected candidates and offset entries do not count toward the page. Batches continue until the page fills or all candidates are exhausted; there is no added candidate limit and allowPartial does not stop this walk early.

Ranking, eligibility and full graph hydration share a PostgreSQL REPEATABLE READ transaction on a slave query runner, preserving the repository's read-replica routing. 'slave' is TypeORM's role name for the replication.slaves entries built in postgres.ts; when dbReplications is false no slaves are configured and TypeORM serves the runner from master, which is the path the benchmark and test suites exercise. A deletion or edit committed between those stages cannot mix different note/relation versions into that page. The transaction commits before invoking database fallback and is rolled back and released on errors. It adds transaction commands and holds a database connection while filters run. Replication lag, cancellation and production connection-pool contention were not benchmarked.

Viewer relationship caches, and supplemental lookups made by visibility/packing outside those graph queries, retain their existing behavior; this is not a snapshot of every cache or of the later packing stage. A missing forum relation can still trigger the original visibility lookup. The concurrent-edit fixture specifically verifies the graph snapshot, not globally atomic authorization changes.

No eligible cached notes still invokes fallback even with allowPartial; an offset beyond otherwise eligible cached notes retains the original empty-page distinction. A short page still appends fallback notes in the fallback's order, deduplicates by ID, and applies the existing window check. It does not re-sort or reinterpret the fallback's candidate widening.

Reproduction

The unit suite covers this refactor without extra infrastructure, including the differential comparisons against the frozen baseline service and the projection-coverage tests:

NODE_ENV=test corepack pnpm --filter backend exec vitest run --config vitest.config.unit.ts \
    test/unit/FanoutTimelineEndpointService.ts test/unit/FanoutTimelineService.ts \
    test/unit/misc/note-ranking.ts test/unit/misc/active-timeline-candidates.ts

The measurements and the packed-equality checks below came from a differential benchmark that ran both variants through a real Nest endpoint, PostgreSQL, Redis and packMany, using a temporary database and Redis namespace it removed afterwards. That script is a one-off validation artifact and is deliberately not in the tree; it is kept with its raw output at catodon-plans/ranked-timeline/. Re-running it requires restoring the script, .config/test.yml pointing at test PostgreSQL and Redis, and a role allowed to create databases. The baseline service it compared against is frozen from 26346d97401 in test/fixtures/ranked-timeline-baseline.ts, which the unit tests also import; only its exported class name and an explanatory header differ from the original.

Correctness comparisons check ordered complete packed objects, ties, rank/ID cursors, offsets, windows, missing/future timestamps, filters, sparse pages, Home and Hybrid, actual database fallback and cache/fallback merges. Explicit assertions exclude denied authors, nested quoted authors, instances, private forums and private notes, and ensure authorized DMs and own-note exceptions survive. Hidden quoted text must not appear in the packed output. A concurrent commit deletes one selected note and changes another's visibility and score after candidate selection; the new implementation must return the same graph snapshot as the baseline before that commit. Tests also exercise arbitrary callbacks reading fields outside the narrow contract.

Performance cases use 800 cached roots and a 20-result page: dense eligibility, a nested quote graph, sparse eligibility (one in 31 roots), and every root rejected. The last case disables fallback to isolate the cache scan. Other timing cases use the real Home endpoint and full packing. Each case checks complete baseline/after equality before measuring, and every measured request asserts the expected graph counts so a stale build cannot produce a false success.

Each variant gets five warmups and 30 timed samples per workload, sort and cache condition, with alternating before/after execution order. Ten separate requests per variant/condition use V8 allocation sampling at a 16 KiB interval, including objects subsequently collected by minor/major GC. Allocation values are estimates, not peak or retained heap measurements.

“Cold-memory” clears only CacheService memory caches before each request. Redis, other application caches, PostgreSQL and OS buffers stay warm; these are not cold database or cold process measurements. “Warm” retains caches after warmup. The benchmark measures endpoint .exec through validation and packMany, not HTTP transport, authentication or JSON response serialization. Requests are sequential; production concurrency and other hosts remain unmeasured.

SQL counts include transaction control and packing/cache queries. PostgreSQL transfer is measured with the driver's socket response-byte counters, including protocol overhead, not estimated from selected columns. Full graph counts are root notes returned by graph queries, not all nested entities; narrow rows include both ranking and eligibility passes. ORM hydration time wraps TypeORM's outermost raw-result transformation; process CPU covers the endpoint request. SQL instrumentation and profiling have overhead. Allocation profiling is kept outside timed samples. p50/p95 use nearest-rank quantiles; the raw artifact retains every timed sample and source/build hashes.

Evidence

Measured on 2026-09-20, Node v24.15.0, PostgreSQL 17.9, Redis 6.0.16, Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz (8 threads, 8.3 GB, linux 6.18.33.2-microsoft-standard-WSL2). Baseline frozen from 26346d97401. 30 timed samples per variant/workload/sort/condition after 5 warmups, alternating before/after order, with 10 separate allocation-sampling requests. Raw artifact: ranked-timeline-final.json (local run artifact).

Every workload uses 800 cached candidate IDs and a 20-result page. dense accepts nearly all candidates, nested adds quote/boost graphs, sparse accepts roughly one in 31, and empty rejects every candidate with fallback disabled to isolate the cache scan.

Work performed per request

(Hot/warm shown; Active and cold-memory differ in magnitude, not direction. SQL counts include transaction control, packing and cache queries.)

Endpoint latency, p50 / p95 in milliseconds

Sampled allocations per request

Allocation figures are V8 sampling estimates at a 16 KiB interval, including memory later collected; they are not peak or retained heap.

Reading these results

The mechanism behaves as intended in every workload: full note graphs drop from 800 to the page size wherever a page is produced, and per-request transfer falls by about an order of magnitude. The dense and nested workloads, which resemble an ordinary timeline read, improve on every axis: transfer 3.9 -> 0.4 MB, ORM hydration 266 -> 12 ms, and p50 567 -> 97 ms for dense hot warm.

The sparse and empty workloads are the honest limit of the change. Sparse improves on p50 (hot warm 653 -> 509 ms) because transfer and hydration still fall, though far less than dense. Empty is effectively flat and should not be cited as a gain: p50 moves 660 -> 651 ms while its hot/warm p95 moves the wrong way, 1061 -> 1108 ms, consistent with the added round trips and within run-to-run noise. Both pay in SQL statements, sparse rising from 26 to 49 and empty from 1 to 29, because the walk must examine every candidate before concluding the page cannot be filled. Empty is a deliberately adversarial shape that rejects all 800 candidates with fallback disabled. Adaptive batch sizing was considered and not implemented: no workload regressed on p50, and a larger batch would trade these round trips for exactly the wide hydration the change exists to avoid.

Two costs are real and unmeasured here. The snapshot holds one pooled replica connection for the duration of a request's eligibility walk, which is roughly 2 queries on a dense page but about 25 on a sparse or empty one, and filters await cache lookups while it is held. Under production concurrency this is a different pool-contention profile than the original code, which released its connection before filtering; sequential benchmarking cannot show that. Measurements come from a single shared development machine with PostgreSQL and Redis on localhost, and exclude HTTP transport, authentication and JSON serialization.

Test and check results

Unit and endpoint-level suites were run on clean develop and on this branch through the same isolated harness (a per-run temporary database and a private Redis), then compared by exact test name rather than by totals.

• Unit: 16 failures before and after, the same 16 named tests, none added or removed. The suite comparison was run with 18 new differential tests (3401/3434 -> 3419/3452); three projection-coverage tests added afterwards bring the new total to 21 and were verified in the focused suite, not in a repeated full comparison.

• Endpoint (timelines.ts, renote-mute.ts, note.ts): 26 failures before and after, identical named sets, no regressions. 97 timeline tests pass. renote-mute.ts fails at its first setup step on both, so it contributes no coverage in either direction.

• Standard backend typecheck and ESLint on the touched files report no errors. Eight pre-existing WordFilterService.ts lint errors reproduce on clean develop and are unrelated.

All inherited failures above were reproduced on clean develop before this work and are not attributable to it.