seqrec-lab

A pre-registered study of Transformer sequential recommenders, SASRec, BERT4Rec and gSASRec, with equal tuning budgets, five to ten seeds per arm and full-ranking evaluation on one rented consumer GPU. On MovieLens-1M, Video_Games and Steam, SASRec with full cross-entropy is ahead of BERT4Rec and gSASRec, and SASRec with BCE and one negative trails it by more than 2%. MovieLens-1M's test targets had been read for anchor checks before its final runs. MovieLens-20M has no results yet.

  • Python
  • PyTorch
  • Sequential recommendation
  • Pre-registration
  • Equivalence testing
Interval chart of the relative difference in test NDCG@10 from SASRec with full cross-entropy, with ±2% shaded. MovieLens-1M: sampled softmax −4.1% (−5.2% to −3.1%), gBCE −4.1% (−5.0% to −3.1%). Video_Games: sampled softmax −4.8% (−5.7% to −4.0%), gBCE −8.0% (−8.4% to −7.5%). Steam: sampled softmax −1.8% (−2.5% to −1.2%), gBCE −1.3% (−2.0% to −0.7%). Interval chart of the relative difference in test NDCG@10 from SASRec with full cross-entropy, with ±2% shaded. MovieLens-1M: sampled softmax −4.1% (−5.2% to −3.1%), gBCE −4.1% (−5.0% to −3.1%). Video_Games: sampled softmax −4.8% (−5.7% to −4.0%), gBCE −8.0% (−8.4% to −7.5%). Steam: sampled softmax −1.8% (−2.5% to −1.2%), gBCE −1.3% (−2.0% to −0.7%).
Sampled softmax and gBCE on the SASRec backbone against full cross-entropy, as relative differences in test NDCG@10 with 95% intervals. H2b expected both within ±2%; equivalence is not shown on any dataset. On MovieLens-1M and Video_Games every interval lies below −2%, on Steam neither lies wholly below it. On MovieLens-1M the equivalence tests are underpowered.

Key metrics

test NDCG@10 of SASRec with full cross-entropy over SASRec with BCE and one negative, on MovieLens-1M, Video_Games and Steam. On all three, a minimum-effect test shows the gap above 2%
+34.3%, +39.7%, +26.5%
SASRec with full cross-entropy over BERT4Rec on the same three datasets. On MovieLens-1M this confirms the published gap and shows SASRec non-inferior (H1a); on Steam its sign is the reverse of the published gap
+19.5%, +22.8%, +2.9%
sampled softmax and gBCE against full cross-entropy on the same backbone, over three datasets. On none is equivalence within 2% shown (H2b). On MovieLens-1M and Video_Games every 95% interval lies below −2%; on Steam the differences are −1.8% and −1.3%
−1.3% to −8.0%
final runs, each of which read the test data once, in 81.9 GPU hours on an RTX 4090; the 300 tuning runs read it zero times
223

Differences are between the means over seeds of log test NDCG@10, shown as relative differences with 95% Welch intervals, on leave-one-out with ten seeds per arm on MovieLens-1M, six on Video_Games and five on Steam. Holm's correction runs over each family's registered size across all four datasets, so a comparison that the missing dataset could still change is pending, and nothing pending is reported as a result. On MovieLens-1M one BERT4Rec seed sets the seed variance, so every directional and equivalence comparison there is labelled underpowered; its test targets were also read twice for each of four arms in the anchor checks before these runs. MovieLens-20M has no results yet. Results

Problem and goals

BERT4Rec was reported to beat SASRec. Later work traced most of that gap to SASRec's training objective, binary cross-entropy with one sampled negative, and to training length, and found SASRec with full cross-entropy ahead on most datasets. Those studies mostly report single runs, tune with unequal or unstated budgets and use leave-one-out splits, while differences of 1 to 2% between SASRec runs can change winner with the seed.

seqrec-lab reimplements SASRec, BERT4Rec and gSASRec in plain PyTorch and compares five arms on four datasets: SASRec trained with BCE, full cross-entropy, sampled softmax or gBCE, and BERT4Rec, with two floor baselines. Every arm gets the same tuning budget, items are ranked against the full catalogue, and the number of seeds follows a power rule. The study asks which published gaps survive, and what each training objective costs in time to a target quality. Its hypotheses, tests and the 2% smallest effect of interest were frozen before any model was trained on real data.

On the three datasets with results, H1a, that SASRec with full cross-entropy is non-inferior to BERT4Rec, is supported on MovieLens-1M; it is registered on the two MovieLens datasets only. H1b, that SASRec with BCE is more than 2% below the other three, is supported on Video_Games and Steam and pending on MovieLens-1M, where its test against BERT4Rec waits on MovieLens-20M. H2a, that full cross-entropy reaches the target quality first, is supported on MovieLens-1M and Steam and not on Video_Games, where sampled softmax is not separated from it. Of H2b, the clause on BCE is supported on all three, and the clauses expecting sampled softmax and gBCE to match full cross-entropy are not.

MovieLens-1M, Video_Games and Steam have their final runs and confirmatory analysis. MovieLens-20M has none yet, and every comparison on it is pending. The efficiency study of where training time goes, and the conditional question on approximate retrieval, have not run.

Architecture

seqrec data prepare downloads the four datasets, checks each file by SHA-256 and preprocesses them, and src/seqrec/data builds the leave-one-out and temporal splits. The five arms are PyTorch models in src/seqrec/models. Batches run on a rented RTX 4090 on a Ryzen 9 7950X host: seqrec tune runs one Optuna study per arm and dataset on validation data, and seqrec run trains the final runs with the seeds the power rule gives. seqrec test reads the test data once per final run, through the one reader that logs every read. seqrec export writes the batch files with the count of reads, and seqrec analyze applies the tests of the frozen METHODS to them and writes the tables behind docs/RESULTS.md.

Registered rules

  • §7 A 2% effect, tested across runs

    The unit is one training run, scored by the log of its test NDCG@10, and two arms are compared by Welch t-tests over their seeds, with Holm's correction within each family. Tests over users would be overconfident, and differences of 1 to 2% between runs can change winner with the seed, so the smallest effect of interest is 2%; equivalence is two one-sided tests against ±2%. The number of seeds is the smallest from 5 to 10 at which both kinds of test reach a power of 0.8 at 2%, or 10 if none does, from the variance of the first five: five on Steam, six on Video_Games, and ten on MovieLens-1M, where none does.

  • §5 One tuning budget for every arm

    Each arm gets one Optuna study of 20 trials per dataset (10 on MovieLens-20M), all on training seed 0 and under caps fixed in advance: 300 epochs for the SASRec arms, 1,000 for BERT4Rec and 8 hours a run. The best trial is rerun with every seed. An equal wall-clock budget would favour the fast arms and mix the question of quality with that of cost. An arm with more than 20% of its trials stopped by a cap has every limit doubled in its final runs.

  • §11 Test data read once

    The test split is sealed. Only one reader opens it for scores; it requires a phase, final or anchor, and logs each read with the run, the commit and a hash of the checkpoint, and a tuning run that tries to read it fails. The export recounts the reads: each of the 223 final runs read its test data once.

  • §3 Shuffled order as a control

    With each user's items shuffled in training and evaluation, a sequential model should lose more than 2%; without that drop on MovieLens, the order handling and evaluation would be audited before any other result. The unshuffled SASRec-CE is ahead by 428.2% on MovieLens-1M, 67.8% on Video_Games and 75.4% on Steam, and BERT4Rec by 390.4% on MovieLens-1M.

Pre-registration

Limits

  • One seed set the power on MovieLens-1M

    One BERT4Rec seed reached a validation NDCG@10 of 0.1174, against 0.1692 to 0.1740 for the other four, and BERT4Rec holds 98.9% of the pooled seed variance. The registered rule then gave ten seeds and a directional power of 0.005 at 2%, so every directional and equivalence comparison on MovieLens-1M is labelled underpowered, those between arms whose seeds agree closely included. The run finished normally and stays in every analysis; a power computed from the two arms compared is reported beside each directional and equivalence test and decides nothing.

  • MovieLens-1M's test targets were read before

    To check four of the arms against published values, the anchor runs read the MovieLens-1M test targets twice for each of them before the final runs, once on the laptop and once on the rented machine. All four landed inside their bands, and every conclusion on MovieLens-1M about which gaps survive carries this note.

  • Times hold for one kind of host

    Times to target come from an RTX 4090 on a Ryzen 9 7950X host. On small configurations a training step is bound by kernel launches, and time to target there mostly measures launch overhead. On MovieLens-1M, H2a rests on runs that never reached the target: 14 of the 15 runs of the other three arms. On Steam its test against SASRec-BCE rests on five runs that never reached it.