ope-checks
Pre-registered checks of published off-policy evaluation results. The first study retests the Open Bandit benchmark: on its two confirmatory campaigns, with ten-minute blocks as resampling units, the standard estimate is not shown to fall short of the on-policy click rate.
Key metrics
- of the on-policy click rate: the fixed-prior IPS estimate pooled over All and Women's, with an interval of 0.915 to 1.036 over ten-minute blocks. No shortfall is shown, and one of up to 8.5% is not excluded
- 0.974
- distance between the published prior and the item frequencies of the BTS log, on All and on Women's, against noise floors of 0.14 and 0.11: the prior is not the distribution the log shows
- 0.53
- paired change in the pooled estimate when the target is aligned to the BTS log in three-hour periods, with an interval of 0.002 to 0.139 whose lower end is, post hoc, about two Monte Carlo standard errors from 0; on All and on Women's alone the interval contains 0. No shortfall was established, so the registered rule does not read the gap
- +0.072
- the standard deviation of the H3 ratios with ten-minute blocks over the one with rows or impressions as resampling units, on all three campaigns (post hoc)
- 1.31–1.47×
Registered results on All and Women's from one run of commit 11e9340. Intervals are the 2.5th and 97.5th percentiles of 10,000 replicates that resample ten-minute blocks, a length fixed with the Men's results in view, as the registration states. Men's is the development campaign and enters no verdict; its IPS ratio is 0.891 (0.790 to 1.005). The last figure is post hoc. Results in the README
Problem and goals
The Open Bandit Dataset holds one week of logs from two policies on ZOZOTOWN, uniform random and Bernoulli Thompson sampling (BTS), in three campaigns. Its standard benchmark estimates the BTS click rate from the random logs and compares it with the click rate of the BTS logs. On the Men's campaign the estimate is about 0.89 of the on-policy value.
The BTS target is computed from the published prior, which is not the policy that was deployed during the logged week. The study asks how much of the gap comes from that target. It does not ask why the two differ.
The propositions, statistics and decision rules were committed before any outcome of All and Women's was computed; the results on Men's were known. Men's is the development campaign; All and Women's are the confirmatory campaigns, and every verdict comes from one run on them.
Architecture
data.py downloads the dataset archive and the prior file, verifies their SHA-256 and runs the registration's stop checks, which read no click. tables.py sums the rows of both logs by ten-minute block. estimators.py computes the fixed-prior target and the propositions H1 to H7 as functions of those tables, and verdicts.py holds the decision rules. run.py draws 10,000 bootstrap replicates over blocks, each of which rescales the block tables and calls the same estimators, and writes one JSON file per campaign with every statistic, interval and label and the commit that produced it. The post hoc analyses live in obd/posthoc/ with result files of their own, and the figures are drawn from the result files.
Registered rules
-
§4 One campaign to develop on, two for the verdicts
Men's is the development campaign: code is written and checked on it, and its results enter no verdict. Nothing was computed from the files of All and Women's, beyond row counts and checks that read no click, until the code for every proposition had run on Men's and been committed. The confirmatory results come from one run of that commit.
-
§5 Ten-minute blocks as the resampling unit
Lists are reused across impressions for up to a few minutes and clicks cluster over a few minutes, so neither rows nor impressions are independent. A replicate draws 1,008 ten-minute blocks with replacement, and a block carries the rows of both logs that fall in it.
-
H3 Low only if the whole interval lies below 1
For each unit and estimator the label is
lowif the interval of the ratio to the on-policy click rate lies below 1,highif it lies above, and otherwiseno shortfall shown. On All, on Women's and pooled, IPS and SNIPS are allno shortfall shown. -
H4 The gap is read only where a shortfall is found
H4, the primary proposition, replaces the prior with the BTS log's own frequencies in the same period. Its reading of the gap applies only in a unit where H3 found IPS
low. No unit did, so every unit readsshortfall not established; gap not read. -
A7 One feature value written differently in the two files
The two log files hash the user features, and one value of each feature is written differently in them. Amendment 7, fixed before any confirmatory outcome was computed, treats the two as one value where exactly one value occurs only in each file. That matched three of the four features on each confirmatory campaign; the fourth is compared as written, so the registered distances between the users of the two logs, 0.360 and 0.236, cannot be read as a difference in the users.
Limits
-
Not shown is not absent
The pooled interval reaches down to 0.915, so with ten-minute blocks a shortfall of up to 8.5% is not excluded. Post hoc, with three-hour blocks the pooled interval is 0.897 to 1.056. On the development campaign the ratio is 0.891 (0.790 to 1.005), and it enters no verdict.
-
The block length was fixed after the development results
The block length and the thresholds of H1 were fixed with the Men's results in view. The data have no user id, so dependence that reaches beyond a ten-minute block, such as visits of one user spread over the day, is not in the intervals.
-
The main question has no reading
The study set out to measure how much of the gap the evaluation target accounts for. With no shortfall established on the confirmatory campaigns, the registered rule reads no gap, so that question is unanswered. The aligned estimate is not distinguishable from the on-policy value in any unit, and its paired change from the fixed-prior IPS excludes 0 only pooled, where the lower end of its interval is 0.002.
-
What the public data cannot show
The Beta parameters in force at any time, the update interval and which version served a row are not in the data. Whether the recorded propensities of the BTS log are internally consistent is undetermined on both campaigns. All is not a clean hold-out from the development campaign: a check made after the results, without reading a click, finds all 34 Beta parameter pairs of Men's in the prior file among the 80 of All, identical and in the same order, and none of the 46 of Women's.
-
Coverage is counted only on simulated clicks
A separately registered semi-synthetic study (
obd_coverage/) keeps the rows of the logs and generates clicks from a known process, so that the coverage of a nominal 95% interval can be counted. With dependence that reaches one or two minutes, at the strength solved from published statistics of the logs, intervals of the IPS ratio on All and Women's cover 0.85 to 0.88 when rows or impressions are resampled and 0.945 to 0.953 with clock blocks of 5 to 180 minutes; this is an upper reading of short-range dependence: if part of the published variance ratio is a moving click rate, rows and impressions cover more. With added groups of sessions that span an hour, ten-minute blocks cover 0.907 on All and 0.850 on Women's, and with groups that span a day no block length is adequate. Both scenarios were tuned so that the true variance exceeds what ten-minute blocks see, so under-coverage by ten-minute blocks was expected there. On Women's the published widening of the intervals up to 180 minutes is more than the short-range scenario produces, so its ten-minute intervals are not shown to cover. The study does not show that the real logs have any of these structures. -
What the re-scoring labels cover
A further study,
obd_rescore, re-scores two published estimator comparisons againstR_IPS, the IPS of the evaluated target on the whole random log, which is unbiased for the target's value, as well as againstV_on, the click rate of the BTS log. Its registration was committed before any code of the published comparisons was run on the dataset, and both replications arereproduced. On All, with ten-minute blocks and the target applied to its own (item, position) pairs, MIPS (w/ SLOPE) against IPS isflipsagainstR_IPS: it is closer in 91 of 500 runs. On Men's, the interval ofBLUE - IPSover 2,000 block replicates, 0.000107 to 0.000898, excludes 0, so "BLUE remains unbiased" iscontradictedat the full size of the data, with the reward model and the target fixed; on All and Women's it isnot contradicted, which does not show that BLUE is unbiased.flipsdoes not show that MIPS (w/ SLOPE) is worse in general. The labels describe each protocol's resampling, with the references fixed at their full-data values, and rest on ten-minute blocks; the reward model of BLUE is fitted once on the full data.