recserve

A recommendation serving system in Go: retrieval, features, ranking and rerank behind one HTTP API, with the item vectors on gRPC shard replicas. In kind at 100 requests/s, through rolling restarts of the gateways and of the replicas and a bundle swap, 15,000 of 15,000 responses were neither failed nor degraded, with stage budgets of 100 ms.

  • Go
  • Recommender systems
  • gRPC
  • Kafka
  • Kubernetes
Chart of the kind check at 100 requests/s. Responses that failed or were degraded: as deployed, 0 of 15,000; with 30 s between DNS resolutions, 815 of 15,000 degraded; with no shutdown delay, 56 failed. Duration of each step as deployed: the gateways roll in 9 to 11 s, the replicas in 38 to 40 s, the bundle swap in 15 to 17 s. Chart of the kind check at 100 requests/s. Responses that failed or were degraded: as deployed, 0 of 15,000; with 30 s between DNS resolutions, 815 of 15,000 degraded; with no shutdown delay, 56 failed. Duration of each step as deployed: the gateways roll in 9 to 11 s, the replicas in 38 to 40 s, the bundle swap in 15 to 17 s.
The kind check: 100 requests/s while the gateways roll, then the replicas, then the gateways swap bundles. As deployed, none of 15,000 responses failed or was degraded; the gateways rolled in 9 to 11 s, the replicas in 38 to 40 s, and a swap took 15 to 17 s. Two of the broken variants, each run once: with gRPC's default of 30 s between DNS resolutions, 815 of 15,000 were degraded; with no shutdown delay, 56 failed with a connection reset.

Key metrics

responses failed or degraded in kind at 100 requests/s, through rolling restarts of the gateways and of the replicas and a bundle swap. The stage budgets there are 100 ms, where the specification has 5 to 15 ms
0 of 15,000
recall@10 of the logistic-regression ranker on 680 held-out users, against 0.026 for retrieval order, measured before MMR. The weight it gives the vector score is negative
0.091
p99 at 400 requests/s after the inner product of exact search moved to AVX2, from 6.90–7.79 ms before, three runs each. At that rate the gateway's CPU per request is not resolved on this machine
4.27–4.64 ms
of responses fell back to the request's own items at 200 requests/s on Valkey and Kafka: the history read ran past its 3 ms budget at the time (5 ms now). In a separate check with the four CPUs saturated, the p99 of the stage of history read and cache lookup was above the 3 ms budget in two of three runs in the round that computes, 4.3 and 5.8 ms; in the other it was 2.9 ms. All these runs carried a Kafka health check that kept about two thirds of a CPU busy
0.2–0.3%

The latency and load figures come from cloud VMs with 4 vCPUs. The kind cluster is one node, which the control plane and the load generator share. The catalog has 25.6k items, where exact search takes about 2 ms on one core, so the shards show the mechanism and are not needed for speed. The kind check and the ranker's figures are in the decision log (D047, D027). While gateways start on a new version, as in a rollout, a replica that holds the version the running gateways serve and the one before it, and was last sent a search or Prepare of the one before, drops the version they serve when a starting gateway has it prepare a third. Their searches then go to the shard's next replica until the replica has prepared that version again; while no replica of a shard holds it, their responses report shards_partial, or vector_error when that holds for every shard. The drop was checked once, by hand, at one replica. In the load sweep (replay, result cache off, three runs per rate), the highest rate at which every run had p99 at most 50 ms and at most 1% of responses degraded was 550 requests/s, p99 4.6 to 5.1 ms, with the history budget then at 3 ms. In the latest fault runs (275 requests/s on one 4-vCPU VM, result cache off, three runs per fault), four faults passed their criteria with no error and a fault-phase p99 of 6.8 to 20.5 ms: a replica killed, a replica paused, a replica whose searches hang, and 50 ms of latency on a shard. Valkey down failed: one run took 7 s to recover with 5 s allowed. The ingest consumer paused was not judged: one of its runs had no valid attempt. The sweep and these fault runs come from different VMs and are not compared. Benchmarks

Problem and goals

recserve is the serving side of a recommender: a request passes multi-source retrieval, a point-in-time feature lookup, model ranking and re-ranking under a latency budget. Serving engineering is the focus: latency, throughput, availability, distribution and observability. The models it serves so far are simple.

The goal is behaviour that a test can state. A response never mixes two bundle versions. With one replica of every shard lost, every response equals the one with all replicas healthy; for a replica that goes silent, this holds within the hedge budget, and once five of its calls in a row have ended at the shard deadline it is left out, and tried again with one search a second, while the other replica of its shard answers in time. With Valkey unavailable, every response is still a 200 that uses the request's own recent items and says so. An event delivered more than once, in any order, leaves every history and every response unchanged.

The scale is small and stated plainly. The catalog has 25.6k items, so neither approximate search nor sharding is needed at this size; the service shards it anyway to exercise the mechanism. Each load test runs on one VM with 4 vCPUs, so the figures show behaviour under load and relative effects. They do not predict the capacity of a deployment on several machines.

Architecture

The gateway (cmd/gateway) loads a model bundle, whose version is the digest of its files, and answers POST /v1/recommend. It assigns the A/B arm, reads the user's history and the result cache from Valkey through a circuit breaker, encodes the user with ONNX Runtime, retrieves from co-occurrence, popularity and the item vectors, drops the items the user has seen, ranks them in the arm that uses the ranker, then applies MMR. Each stage has its own deadline: a stage that runs out is skipped or falls back, and the response lists the reasons. The item vectors are split into shards, which the gateway holds itself or, as in compose and kind, searches on annshard replicas over gRPC: a failed call goes to the next replica at once, a slow one gets at most one hedge, and a replica whose calls end at the shard deadline five times in a row is left out, and tried again with one search a second, while the other replica of its shard answers in time. POST /v1/events produces to Kafka, and the ingest consumer applies each event to the histories idempotently, then commits. In a bundle swap the gateway loads and verifies the new version and has every replica prepare it, one replica per shard being enough to go on, then activates it with one atomic store.

Key decisions

  • D029 A deadline per stage, and a reason for every fallback

    Each stage of a request runs under its own budget. A stage that times out or fails is skipped or falls back: without the stored history the request's own recent items are the history, without a user vector the vector retrieval is skipped, and a ranker that fails leaves the retrieval scores. The response lists the reasons. A caller's own deadline is never reported as a stage's timeout.

  • D041 Failover at once, hedges on a budget

    A shard search asks one replica. When the call fails, the search goes to the next replica at once. When the first replica has not answered after 2.5 ms, the search asks one more and takes the first answer. Hedges come from a token bucket per shard, at most 10 plus one for every ten searches, so slow replicas are not sent twice the load. A replica whose calls end at the shard deadline five times in a row is left out while another replica of its shard answers in time; once a second one search asks it first, and an answer brings it back. The gateway does the hedging itself: grpc-go implements retries and not hedging.

  • D042 A response never mixes two versions

    A request runs on one bundle version and names it in every shard search, so a replica answers from that version or fails. A swap sends Prepare to every replica and goes on when each shard has at least one that holds the version; if a shard has none, the swap is rejected and the active version stays. A replica keeps two versions and never drops the one that the last Prepare to reach it names as served. That is the version the gateway serves, except for about a second after a swap, with several gateways, or after a Prepare the replica ran late.

  • D027 A ranker checked when its bundle loads

    ranker.onnx is a logistic regression over six features. Every call passes 512 rows, so loading checks that each row's score depends on that row only. The bundle carries golden cases with their features, computed by the Python code that computed the training features; the gateway recomputes them with its own code and rejects the bundle if any differs by more than 1e-6. If the ranker times out or fails, the retrieval scores are kept.

  • D047 kind checks behaviour, with budgets of 100 ms

    The zero-error check runs in kind on one node, whose CPUs the control plane, the Pods and the load generator share. With the specification's budgets, then 3 to 15 ms, 3 of 24,000 responses timed out with no rollout at all, the slowest taking 22 ms. The check's budgets are 100 ms, about five times that, so that a degraded response means a failure. With these budgets the check does not see what a rollout does to latency. Three later runs of its rollouts and swap with the specification's budgets had no error and at most 0.133% of a phase's responses degraded, with p99 at most 9.91 ms; they are reported, not judged, and the check keeps 100 ms.

Decision log

Challenges

  • Gateways that kept dialing the old address

    A replica restarted under its name comes back on another address. With gRPC's default of 30 s between DNS resolutions, two of three gateways reached the new address 30 s after the Pod was deleted, though the replica was ready after 1 s, and in the kind check 815 of 15,000 responses were degraded. The gateway now lets a name be resolved again after one second, and all three gateways reached the replica within 2 to 3 s.

  • A hand-written HNSW that failed its latency criterion

    The pure-Go library tried first reached a recall@100 of at most 0.064, so the index is written by hand. On glove it reaches a recall@100 of 0.95 only with M = 32 at ef = 800, with a p99 of 7.66 ms, above the 5 ms of the criterion: the criterion fails as written, and no threshold was changed. Its recall agrees with faiss's within 0.007, and its p99 on glove is 1.0 to 1.3 times faiss's at the same ef. Exact search is the default.

  • A ranker whose weight on the vector score is negative

    On the 680 held-out users the ranker's recall@10 is 0.091 against 0.026 for retrieval order. The weight it gives the vector score is negative: among the candidates, the items a user went on to choose came more from co-occurrence and popularity than from the vector score. Both figures are measured before MMR. In an offline run of the pipeline in process, with no stage budgets and MMR included, on 43,249 test events, recall@10 is 0.0181, and 0.0050 with the ranker forced to fail, which keeps the retrieval scores: a difference of −0.0131, 95% interval over test events −0.0143 to −0.0119. With co-occurrence forced to fail it is 0.0098; with the encoder forced to fail, no metric moves by more than 0.0004. Its effect on users is for an A/B experiment to show.

  • An idle Kafka that used two thirds of a CPU

    In compose an idle Kafka used 639 to 686 ms of CPU per second: its health check started a JVM every 2 s. One unrecorded profile run with that check had a p99 of 10.97 ms, against 6.41 ms with a check that runs every 2 s only until the broker is healthy and every 10 minutes after. The earlier load runs in compose carried that load, and none was rerun.