Algorithm engineering and server-side systems.

Each project below ships its tests in its repository, with its decision log or its pre-registration.

Algorithms
Recommendation, off-policy evaluation and retrieval. Two-tower recall, gradient-boosted ranking and uplift models, labelled by evidence tier; a pre-registered recheck of the Open Bandit benchmark, where the IPS estimate pooled over the two confirmatory campaigns is 0.974 of the on-policy click rate (0.915 to 1.036 over ten-minute blocks): no shortfall shown; and BM25 and dense retrieval in Go checked against Pyserini's published BEIR numbers: BM25 equal to four decimals on six datasets, BGE-base within 0.0006 on six.
Server-side systems
A recommendation serving system in Go: retrieval, features, ranking and rerank behind one HTTP API, with the item vectors on gRPC shard replicas. In kind at 100 requests/s, through rolling restarts of the gateways and of the replicas and a bundle swap, no response failed or was degraded, with stage budgets of 100 ms.
Backend systems
Go limit order book with a write-ahead command log and open-loop load testing: 6.25M to 8.88M commands/s in the core (measured on a noisy VM, where runs of one benchmark differ by up to 1.96 times), p99 1.76 ms at 10k requests/s; 50k requests/s is near saturation, at 41 ms.

Projects

ope-checks

Pre-registered checks of published off-policy evaluation results. The first study retests the Open Bandit benchmark: on its two confirmatory campaigns, with ten-minute blocks as resampling units, the standard estimate is not shown to fall short of the on-policy click rate.

  • Python
  • Off-policy evaluation
  • Pre-registration
  • Block bootstrap
  • Open Bandit Dataset

retrieval-lab

BM25 and dense retrieval in Go, where BM25, exact dense search and fusion are each accepted against a number that Pyserini publishes for BEIR. BM25 equals the published nDCG@10 and Recall@100 to four decimals on six datasets. BGE-base with exact search is within 0.0006 in nDCG@10 on all six, and within 0.0004 on all but TREC-COVID. A pre-registered study on it found that HNSW's loss in nDCG@10 depends on the model at equal ef, and not once each model is searched to the same neighbour recall of 0.95.

  • Go
  • Information retrieval
  • BM25
  • Dense retrieval
  • BEIR

seqrec-lab

A pre-registered study of Transformer sequential recommenders, SASRec, BERT4Rec and gSASRec, with equal tuning budgets, five to ten seeds per arm and full-ranking evaluation on one rented consumer GPU. On MovieLens-1M, Video_Games and Steam, SASRec with full cross-entropy is ahead of BERT4Rec and gSASRec, and SASRec with BCE and one negative trails it by more than 2%. MovieLens-1M's test targets had been read for anchor checks before its final runs. MovieLens-20M has no results yet.

  • Python
  • PyTorch
  • Sequential recommendation
  • Pre-registration
  • Equivalence testing

recserve

A recommendation serving system in Go: retrieval, features, ranking and rerank behind one HTTP API, with the item vectors on gRPC shard replicas. In kind at 100 requests/s, through rolling restarts of the gateways and of the replicas and a bundle swap, 15,000 of 15,000 responses were neither failed nor degraded, with stage budgets of 100 ms.

  • Go
  • Recommender systems
  • gRPC
  • Kafka
  • Kubernetes

matching-engine

A limit order book matching engine in Go: price-time priority, REST and WebSocket APIs, and a command log that replays to the same book.

  • Go
  • Order book
  • WebSocket
  • Write-ahead log
  • Load testing

Earlier projects