<?xml version="1.0" encoding="UTF-8"?><rss version="2.0" xmlns:content="http://purl.org/rss/1.0/modules/content/"><channel><title>Aaron Yang</title><description>Articles about the projects on this site.</description><link>https://shyanorn.github.io/</link><language>en</language><item><title>The Open Bandit shortfall was not shown on the two confirmatory campaigns</title><link>https://shyanorn.github.io/articles/open-bandit-shortfall/</link><guid isPermaLink="true">https://shyanorn.github.io/articles/open-bandit-shortfall/</guid><description>A pre-registered check of the Open Bandit Dataset benchmark: the published prior is not the policy that was logged, and on the two confirmatory campaigns the off-policy estimate is not shown to fall short of the on-policy click rate.</description><pubDate>Tue, 10 Nov 2026 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;On the Men’s campaign of the Open Bandit Dataset, the standard off-policy estimate of the Bernoulli Thompson sampling (BTS) policy is about 0.89 of the click rate BTS achieved online. A pre-registered study, ope-checks, asked how much of that gap comes from the evaluation target. On the two confirmatory campaigns, All and Women’s, it found that the published prior is not the distribution the BTS log shows, and that the shortfall itself is not shown: the ratio is 0.948 (0.867 to 1.033) on All, 1.004 (0.917 to 1.099) on Women’s and 0.974 (0.915 to 1.036) pooled. With no shortfall established, the registered rules leave the main question without a reading.&lt;/p&gt;
&lt;p&gt;The rules were committed before any outcome of All and Women’s was computed. Men’s was the development campaign and enters no verdict. Everything below that was made after the results were known is marked post hoc.&lt;/p&gt;
&lt;h2 id=&quot;the-benchmark-and-the-089&quot;&gt;The benchmark and the 0.89&lt;/h2&gt;
&lt;p&gt;The dataset holds one week of logs from two policies on ZOZOTOWN, uniform random and BTS, in three campaigns. The benchmark estimates the BTS click rate from the random log and compares it with the click rate of the BTS log. The BTS target comes from a published Beta prior by Monte Carlo draws, with no updates. On the full Men’s campaign the IPS estimate is 0.891 of the on-policy value. Three public computations arrive at the same figure, two of them with a bootstrap interval for IPS that lies below the on-policy value: 0.815 to 0.966 of it in one, and 0.005612 to 0.006472 against 0.006743 in the other. Those two resample impressions or rows; the third uses the whole random log and no bootstrap. The registration cites the first two, and a correction made after the results records the third.&lt;/p&gt;
&lt;h2 id=&quot;rows-in-these-logs-are-not-independent&quot;&gt;Rows in these logs are not independent&lt;/h2&gt;
&lt;p&gt;Served lists are reused: in each of the six logs, 30% to 37% of complete impressions show the same three items in the same order as the impression before them. The click rate also moves between adjacent ten-minute blocks more than independent impressions would allow. The variance of the difference is 1.85 to 2.45 times as large, a ratio that carries no interval and does not separate clustering from drift.&lt;/p&gt;
&lt;p&gt;The registration therefore resamples ten-minute blocks, the same blocks in both logs. That choice was made after seeing the Men’s results, and the registration says so.&lt;/p&gt;
&lt;p&gt;How much narrower the usual intervals are was measured on all three campaigns only afterwards. In that post hoc comparison the standard deviation of the ratio with blocks is 1.31 to 1.47 times the one with rows or impressions. The unit changes the side of 1 on which the interval ends only on Men’s: 0.790 to 1.005 with blocks, 0.818 to 0.964 with rows, 0.816 to 0.968 with impressions. On All and Women’s the interval contains 1 under each unit, so the result there does not rest on the choice of blocks.&lt;/p&gt;
&lt;h2 id=&quot;what-was-registered&quot;&gt;What was registered&lt;/h2&gt;
&lt;p&gt;Seven propositions, with All and Women’s as confirmatory campaigns and a pooled value (weights 0.54 and 0.46) for the two central ones. Intervals are percentiles of 10,000 replicates over 1,008 ten-minute blocks from a registered random stream. H1 sets the distance between the prior and the hourly frequencies of the BTS log against a noise floor from alternate blocks within the hour. H3 calls the shortfall “low” only if the interval of the ratio lies below 1. H4, the primary proposition, takes the BTS log’s own frequencies in the same period as the target and reads the gap only where H3 found one.&lt;/p&gt;
&lt;h2 id=&quot;what-the-confirmatory-campaigns-show&quot;&gt;What the confirmatory campaigns show&lt;/h2&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;td&gt;&lt;/td&gt;&lt;th&gt;All&lt;/th&gt;&lt;th&gt;Women’s&lt;/th&gt;&lt;th&gt;Pooled&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;H1 distance; noise floor&lt;/td&gt;&lt;td&gt;0.526; 0.136&lt;/td&gt;&lt;td&gt;0.527; 0.115&lt;/td&gt;&lt;td&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;H1 label&lt;/td&gt;&lt;td&gt;inconsistent&lt;/td&gt;&lt;td&gt;inconsistent&lt;/td&gt;&lt;td&gt;&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;H3 IPS over on-policy&lt;/td&gt;&lt;td&gt;0.948 (0.867 to 1.033)&lt;/td&gt;&lt;td&gt;1.004 (0.917 to 1.099)&lt;/td&gt;&lt;td&gt;0.974 (0.915 to 1.036)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;H3 label&lt;/td&gt;&lt;td&gt;no shortfall shown&lt;/td&gt;&lt;td&gt;no shortfall shown&lt;/td&gt;&lt;td&gt;no shortfall shown&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;H4 aligned estimate over on-policy&lt;/td&gt;&lt;td&gt;1.031 (0.932 to 1.128)&lt;/td&gt;&lt;td&gt;1.062 (0.974 to 1.158)&lt;/td&gt;&lt;td&gt;1.045 (0.978 to 1.114)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;H4 change from the fixed-prior IPS&lt;/td&gt;&lt;td&gt;0.083 (-0.019 to 0.182)&lt;/td&gt;&lt;td&gt;0.058 (-0.031 to 0.147)&lt;/td&gt;&lt;td&gt;0.072 (0.002 to 0.139)&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;H4 alignment label&lt;/td&gt;&lt;td&gt;no change shown&lt;/td&gt;&lt;td&gt;no change shown&lt;/td&gt;&lt;td&gt;raises the estimate&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;H4 reading&lt;/td&gt;&lt;td&gt;shortfall not established; gap not read&lt;/td&gt;&lt;td&gt;shortfall not established; gap not read&lt;/td&gt;&lt;td&gt;shortfall not established; gap not read&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;The prior and the log disagree by more than the rule asked for: a distance of 0.53 against a noise floor of 0.11 to 0.14. In the first hour of the week the distance on All is 0.180, 0.126 and 0.088 by position. What the deployed parameters were, and how they were updated, the public data cannot say.&lt;/p&gt;
&lt;p&gt;“No shortfall shown” is not “no shortfall”. The pooled interval reaches down to 0.915, so with ten-minute blocks a shortfall of up to 8.5% is not excluded. On the development campaign the ratio is 0.891 (0.790 to 1.005), and that campaign enters no verdict.&lt;/p&gt;
&lt;h2 id=&quot;aligning-the-target&quot;&gt;Aligning the target&lt;/h2&gt;
&lt;p&gt;With the target set to what the BTS log showed in the same three-hour period, the estimate is not distinguishable from the on-policy value in any unit. The paired change from the fixed-prior IPS is positive in every unit and at every period width. Its interval excludes 0 only in the pooled unit, 0.072 (0.002 to 0.139). On All and on Women’s alone, no change is shown.&lt;/p&gt;
&lt;p&gt;This does not say that the target explains the gap. The rule reads the gap only where H3 found one, and it found none. The registration also names a way the aligned estimate can be too high: if BTS learned from clicks of both buckets, or follows demand both buckets share, the aligned target and the random log’s clicks move together. A calculation made after the results does not support the second case: under demand that both buckets share the aligned estimate is too low within a period, so the direction of its bias is not established. H5 is the descriptive check of that. A target built from the hour after a row gives a higher estimate than one built from the hour before, by 0.099 (0.033 to 0.166) on All and 0.076 (0.022 to 0.130) on Women’s. The registration calls this descriptive, because it cannot tell the two sources apart.&lt;/p&gt;
&lt;h2 id=&quot;what-stayed-open&quot;&gt;What stayed open&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;The recorded propensities (H2).&lt;/strong&gt; Undetermined on both campaigns. The registered statistic should average 1 if the recorded propensities are the probabilities that were used; it is 0.973 over 3 eligible cells on All, fewer than the rule needs, and 0.967 over 7 on Women’s.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The users of the two logs (H7).&lt;/strong&gt; The two files write one value of each user feature differently. An amendment made before any confirmatory result matched that value in three of the four features on each campaign. The fourth was compared as written, so the registered distances (0.360 and 0.236 for the joint distribution) still contain the difference in writing and cannot be read as a difference in the users.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The gap itself.&lt;/strong&gt; Not established on the confirmatory campaigns, so the primary question has no reading.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;five-post-hoc-looks&quot;&gt;Five post hoc looks&lt;/h2&gt;
&lt;p&gt;All five were fixed after the results were known. For the block lengths and the distances the settings and the reading were also committed before they were run. They are descriptive and change no registered label.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Resampling unit.&lt;/strong&gt; The comparison above. One figure in the registration did not survive it: the one-minute variance ratio of 1.2 to 1.5 reported for Men’s lacks a small-sample factor, and with it the ratio is 1.18 and 1.20. The figure entered no rule; the factor changes the ten-minute ratios by less than 0.05.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The unmatched feature.&lt;/strong&gt; On each campaign the BTS log holds one more rare value of that feature (30 and 14 rows), so the amendment’s condition failed. The other value of each log sits on exactly the rows the matched features mark. With those two taken as one value, the joint distance is 0.007 and 0.009 (registered: 0.360 and 0.236), and the change in the aligned estimate under user weights is 0.001 (-0.003 to 0.005) and -0.001 (-0.014 to 0.005) (registered: -0.058 (-0.136 to 0.013) and -0.116 (-0.178 to -0.063)). That the two values are one category is an inference from the rows they sit on.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The propensity statistic.&lt;/strong&gt; Much of its distance from 1 sits in a few hours in which an eligible cell is shown at under a tenth of the rate its recorded propensities imply. On All such hours hold 0.033, where the statistic is 0.027 below 1, and nearly all of that falls on 27 and 29 November, as on Women’s. The hours are picked by the statistic itself; on Men’s, where it is at 1.005, the same selection finds 0.002. The cause is not identified.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Block length.&lt;/strong&gt; The lengths and the reading were fixed before these were run. With clock blocks of 5 to 180 minutes as resampling units, the interval of each H3 ratio contains 1 at every length on All, Women’s and pooled. From ten minutes upward the estimated standard deviation of the ratio there is larger at every longer length, a rise not tested against sampling error: the pooled interval is 0.915 to 1.036 with ten-minute blocks and 0.897 to 1.056 with three-hour blocks, where it rests on 56 blocks, so the 8.5% above becomes 10.3%. On Men’s the standard deviation stays within 3% of the one at ten minutes. The interval of the aligned estimate contains 1 in every unit at every length. The lower end of the pooled paired change lies between 0.001 and 0.004 and is within two Monte Carlo standard errors of 0 at three of the six lengths. The H5 difference lies above 0 with blocks of 5 to 20 minutes on both campaigns; on Women’s the lower end of its interval is at 0 with 30-minute blocks, and the interval contains 0 with 60-minute blocks on both campaigns and with three-hour blocks on Women’s. The analysis does not separate dependence beyond ten minutes from change over the week.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The prior, the week and the hours.&lt;/strong&gt; Three distances that read no click, fixed before they were computed. The frequencies of the whole week lie at a distance of 0.386 (All) and 0.413 (Women’s) from the prior. The hourly frequencies lie at 0.410 and 0.394 from those of the whole week, 3.0 and 3.4 times the noise floor; in the random logs of the same two campaigns, whose policy is fixed, that figure is 0.52 and 0.53. By the thresholds of the registered rule, and without an interval, the hourly frequencies thus differ from those of the whole week. The distances bound the 0.53 of H1 and do not say how much of it is change. In the first hour, three of the six distances from the prior are at most a noise floor computed for that hour and three lie between the floor and twice the floor.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;limits&quot;&gt;Limits&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;The interval method was shaped by the development campaign.&lt;/strong&gt; Block length and the H1 thresholds were fixed with the Men’s results in view.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Dependence beyond ten minutes is not covered.&lt;/strong&gt; The data have no user id, so visits of one user spread over a day are treated as independent.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;One week, one site.&lt;/strong&gt; All is not a clean hold-out from the development campaign. A check of the published prior file, made after the results and without reading a click, finds all 34 Beta parameter pairs of Men’s among the 80 of All, identical and in the same order, and none of the 46 of Women’s. Women’s shares no pair with Men’s.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Two estimators.&lt;/strong&gt; The confirmatory campaigns were run with IPS and SNIPS only.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;One run.&lt;/strong&gt; The confirmatory results come from a single run of one commit. A first attempt was cut off by a host restart before it wrote or printed anything, and the same command was run again at the same commit; the repository records it.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;when-this-does-not-hold&quot;&gt;When this does not hold&lt;/h2&gt;
&lt;p&gt;The statement covers these two campaigns, this target and intervals over ten-minute blocks. In a separate simulation on the rows of the logs with simulated clicks (&lt;code&gt;obd_coverage/&lt;/code&gt;), such intervals cover as nominal when dependence reaches a minute or two and fall short when it reaches an hour or a day, and on Women’s the registered intervals are not shown to cover. It does not say the estimate is unbiased, since the Men’s ratio is 0.891 with an interval down to 0.790, and it makes no claim about the recorded propensities or the users of the two logs.&lt;/p&gt;
&lt;h2 id=&quot;reproduce&quot;&gt;Reproduce&lt;/h2&gt;
&lt;pre class=&quot;astro-code astro-code-themes github-light github-dark&quot; style=&quot;--shiki-light:#24292e;--shiki-dark:#e1e4e8;--shiki-light-bg:#fff;--shiki-dark-bg:#24292e;overflow-x:auto&quot; tabindex=&quot;0&quot; data-language=&quot;plaintext&quot;&gt;&lt;code&gt;&lt;span class=&quot;line&quot;&gt;&lt;span&gt;pip install -e .&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;python -m obd.run all women    # confirmatory campaigns and the pooled unit&lt;/span&gt;&lt;/span&gt;
&lt;span class=&quot;line&quot;&gt;&lt;span&gt;python -m obd.posthoc.run      # the post hoc analyses&lt;/span&gt;&lt;/span&gt;&lt;/code&gt;&lt;/pre&gt;
&lt;p&gt;The first run downloads the dataset archive and the prior file and verifies their checksums. Every result file names the commit that produced it.&lt;/p&gt;</content:encoded></item><item><title>A low PBO can mean the search asked an easy question</title><link>https://shyanorn.github.io/articles/low-pbo-easy-question/</link><guid isPermaLink="true">https://shyanorn.github.io/articles/low-pbo-easy-question/</guid><description>A PBO of 0.05 came almost entirely from the long-only versus long/short choice; within the long-only half it is 0.42, in a post hoc, exploratory analysis.</description><pubDate>Mon, 09 Nov 2026 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;The moving-average search in backtest-lab passed its overfitting check easily, with a probability of backtest overfitting (PBO) of 0.05, well inside the green band. Almost all of that came from one yes-or-no question in the parameter grid: whether the strategy may go short. Every in-sample winner was a long-only setting. Restricted to the long-only half of the grid, the PBO is 0.42, and for the momentum search it rises from 0.12 to 0.72. So which long-only setting wins in-sample says little about which one does well out of sample.&lt;/p&gt;
&lt;p&gt;This analysis was designed after the pre-registered results were in. The project labels it exploratory, with no confirmatory weight (D029).&lt;/p&gt;
&lt;h2 id=&quot;the-setup&quot;&gt;The setup&lt;/h2&gt;
&lt;p&gt;backtest-lab runs a parameter search on a strategy and then asks whether the winner is probably overfit. The demo searches were fixed before any strategy ran on real data (D021): SPY data from January 1993 to October 2026, net of 1 bp fees and 5 bps slippage per side.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Moving-average crossover:&lt;/strong&gt; 8 fast windows × 8 slow windows × &lt;code&gt;allow_short&lt;/code&gt; in {false, true}, 128 settings.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Time-series momentum:&lt;/strong&gt; 17 lookbacks × &lt;code&gt;allow_short&lt;/code&gt;, 34 settings.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;One of the checks is PBO, estimated by combinatorially symmetric cross-validation (CSCV). The history is cut into S = 16 blocks. For each of the &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mrow&gt;&lt;mo fence=&quot;true&quot;&gt;(&lt;/mo&gt;&lt;mfrac linethickness=&quot;0px&quot;&gt;&lt;mn&gt;16&lt;/mn&gt;&lt;mn&gt;8&lt;/mn&gt;&lt;/mfrac&gt;&lt;mo fence=&quot;true&quot;&gt;)&lt;/mo&gt;&lt;/mrow&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mn&gt;12,870&lt;/mn&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;\binom{16}{8} = 12{,}870&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;/span&gt; ways to pick half the blocks as in-sample, the setting with the best in-sample Sharpe ratio is found, and its rank among all settings in the other half is recorded. PBO is the share of splits where that winner ranks at or below the out-of-sample median:&lt;/p&gt;
&lt;span class=&quot;katex-display&quot;&gt;&lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot; display=&quot;block&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;mover accent=&quot;true&quot;&gt;&lt;mrow&gt;&lt;mi&gt;P&lt;/mi&gt;&lt;mi&gt;B&lt;/mi&gt;&lt;mi&gt;O&lt;/mi&gt;&lt;/mrow&gt;&lt;mo stretchy=&quot;true&quot;&gt;^&lt;/mo&gt;&lt;/mover&gt;&lt;mo&gt;=&lt;/mo&gt;&lt;mfrac&gt;&lt;mn&gt;1&lt;/mn&gt;&lt;mn&gt;12,870&lt;/mn&gt;&lt;/mfrac&gt;&lt;munder&gt;&lt;mo&gt;∑&lt;/mo&gt;&lt;mi&gt;c&lt;/mi&gt;&lt;/munder&gt;&lt;mn mathvariant=&quot;bold&quot;&gt;1&lt;/mn&gt;&lt;mo stretchy=&quot;false&quot;&gt;[&lt;/mo&gt;&lt;msub&gt;&lt;mi&gt;λ&lt;/mi&gt;&lt;mi&gt;c&lt;/mi&gt;&lt;/msub&gt;&lt;mo&gt;≤&lt;/mo&gt;&lt;mn&gt;0&lt;/mn&gt;&lt;mo stretchy=&quot;false&quot;&gt;]&lt;/mo&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;\widehat{PBO} = \frac{1}{12{,}870} \sum_{c} \mathbf{1}[\lambda_c \le 0]&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;/span&gt;&lt;/span&gt;
&lt;p&gt;Here &lt;span class=&quot;katex&quot;&gt;&lt;span class=&quot;katex-mathml&quot;&gt;&lt;math xmlns=&quot;http://www.w3.org/1998/Math/MathML&quot;&gt;&lt;semantics&gt;&lt;mrow&gt;&lt;msub&gt;&lt;mi&gt;λ&lt;/mi&gt;&lt;mi&gt;c&lt;/mi&gt;&lt;/msub&gt;&lt;/mrow&gt;&lt;annotation encoding=&quot;application/x-tex&quot;&gt;\lambda_c&lt;/annotation&gt;&lt;/semantics&gt;&lt;/math&gt;&lt;/span&gt;&lt;/span&gt; is the logit of the winner’s out-of-sample relative rank. The thresholds were also fixed in advance (D022): green up to 0.10, yellow up to 0.30, red above. The red line comes from a null experiment in which 200 searches over pure noise gave a mean PBO of 0.488, with 89.5% of them above 0.30.&lt;/p&gt;
&lt;p&gt;In the registered results, PBO was 0.05 for the moving average and 0.12 for momentum. Both strategies were rated borderline overall, the moving average because of its deflated Sharpe ratio (0.945 against a 0.95 threshold) and momentum because of that PBO.&lt;/p&gt;
&lt;h2 id=&quot;what-the-winners-had-in-common&quot;&gt;What the winners had in common&lt;/h2&gt;
&lt;p&gt;A PBO of 0.05 means the in-sample winner nearly always lands in the top half out of sample. What those winners share shows up in this table:&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Measure&lt;/th&gt;&lt;th&gt;Moving average&lt;/th&gt;&lt;th&gt;Momentum&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;Sharpe range, long-only settings&lt;/td&gt;&lt;td&gt;0.49 to 0.80&lt;/td&gt;&lt;td&gt;0.54 to 0.79&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;Sharpe range, long/short settings&lt;/td&gt;&lt;td&gt;0.02 to 0.51&lt;/td&gt;&lt;td&gt;0.01 to 0.53&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;CSCV splits whose in-sample winner is long-only&lt;/td&gt;&lt;td&gt;12,870 of 12,870&lt;/td&gt;&lt;td&gt;12,869 of 12,870&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;On SPY over these samples, which start in 1994, going short has been costly, so the long/short half of each grid is mostly worse than the long-only half. The in-sample winner is long-only in all 12,870 splits for the moving average and in all but one for momentum. A long-only setting then almost always ranks in the top half out of sample, because the bottom half is full of long/short settings. So the low PBO, though computed correctly, measures the choice between long-only and long/short, which the equity premium already suggests.&lt;/p&gt;
&lt;p&gt;Running CSCV again on the long-only half alone:&lt;/p&gt;
&lt;table&gt;&lt;thead&gt;&lt;tr&gt;&lt;th&gt;Measure&lt;/th&gt;&lt;th&gt;Moving average&lt;/th&gt;&lt;th&gt;Momentum&lt;/th&gt;&lt;/tr&gt;&lt;/thead&gt;&lt;tbody&gt;&lt;tr&gt;&lt;td&gt;PBO, whole grid (registered)&lt;/td&gt;&lt;td&gt;0.05&lt;/td&gt;&lt;td&gt;0.12&lt;/td&gt;&lt;/tr&gt;&lt;tr&gt;&lt;td&gt;PBO, long-only half&lt;/td&gt;&lt;td&gt;0.42 (64 settings)&lt;/td&gt;&lt;td&gt;0.72 (17 settings)&lt;/td&gt;&lt;/tr&gt;&lt;/tbody&gt;&lt;/table&gt;
&lt;p&gt;Inside the long-only half, picking the in-sample best is not shown to beat picking at random for either strategy.&lt;/p&gt;
&lt;h2 id=&quot;reading-the-higher-pbo&quot;&gt;Reading the higher PBO&lt;/h2&gt;
&lt;p&gt;A high PBO here does not mean the long-only settings are bad. The project’s methods note says that if all configurations have similar real skill, PBO is high too, and the pre-registration expected this for momentum, whose lookbacks behave alike. Every long-only setting has a positive Sharpe ratio, from 0.49 to 0.80 for the moving average. The best long-only setting keeps a significant deflated Sharpe ratio within its half, about 1.00 for both strategies, with 10.5 and 3.8 effectively independent settings.&lt;/p&gt;
&lt;p&gt;The high PBO does show that the fine choice (fast 10 and slow 200, or lookback 273) finds little support in this exploratory analysis.&lt;/p&gt;
&lt;p&gt;The within-half deflated Sharpe ratio of about 1.00 also needs care. Counting every setting of the grid, as it requires, gives the registered 0.945 and 0.988. The values near 1.00 hold only when the long/short half is left out, and leaving it out is itself a choice made after seeing the outcome. Dropping trials after the fact is one of the things these statistics are meant to penalize.&lt;/p&gt;
&lt;h2 id=&quot;limits&quot;&gt;Limits&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Post hoc.&lt;/strong&gt; The long-only split was chosen after the registered results were in, to explain them. It is kept out of the credibility reports and labelled post hoc wherever it appears (D029).&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;One instrument.&lt;/strong&gt; Everything is on SPY. The gap between long and short reflects one market’s history with a strong upward drift.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No cost of shorting or return on cash.&lt;/strong&gt; Cash earns nothing and shorts pay no borrow fee (D015), which shifts the long/short comparison.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Coarse ranks.&lt;/strong&gt; S = 16 blocks and, for momentum’s long-only half, 17 settings. With few settings, ranks are coarse.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Same data for both questions.&lt;/strong&gt; The halves were defined on the same data the PBO was computed on, so the two PBO values are not independent tests.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;when-this-does-not-hold&quot;&gt;When this does not hold&lt;/h2&gt;
&lt;p&gt;The pattern needs a grid axis whose effect dwarfs the others. When the dimensions of a grid have comparable effects, no single choice dominates the in-sample winners, and the whole-grid PBO measures the selection you care about. The whole-grid PBO is also the right number when the coarse choice is the decision itself: if the question really is whether to allow shorting, a low whole-grid PBO is a fair answer to it.&lt;/p&gt;
&lt;p&gt;The check is quick. Before reading a low PBO as evidence that a parameter choice will hold up out of sample, look at what the in-sample winners have in common. If they all agree on one binary axis, compute PBO again inside that branch, and treat the result as exploratory unless the split was registered in advance.&lt;/p&gt;</content:encoded></item><item><title>Locking the vault means ending a process</title><link>https://shyanorn.github.io/articles/locking-ends-a-process/</link><guid isPermaLink="true">https://shyanorn.github.io/articles/locking-ends-a-process/</guid><description>JavaScript strings cannot be wiped, so Local Ledger ends the processes that held them: the page&apos;s renderer on every lock, unlock and committed password change, and on every lock the process that holds the vault. Tests check that they have exited.</description><pubDate>Mon, 09 Nov 2026 00:00:00 GMT</pubDate><content:encoded>&lt;blockquote&gt;
&lt;p&gt;&lt;strong&gt;Updated 2026-11-22&lt;/strong&gt; for 0.3.0, which moved the vault out of the main process into a utility process that every lock ends (&lt;a href=&quot;https://github.com/sHyanorn/local-ledger/blob/82caeac92348c24e2d7ae8f78a9518e48b2e7398/docs/DECISIONS.md#d047-the-vault-in-a-process-that-ends-at-each-lock&quot;&gt;D047&lt;/a&gt;). The text of 2026-11-09 described 0.1.0.&lt;/p&gt;
&lt;/blockquote&gt;
&lt;p&gt;In an Electron app, clearing state on lock leaves the master password in memory. JavaScript strings are immutable, and nothing in the language can overwrite one. So Local Ledger treats a lock as the end of a process. On every lock, unlock and committed password change, it loads a fresh page in a new renderer process and closes the old one, and the renderer that took the typed password exits. The vault is opened and read in a utility process, the vault host, and every lock ends that process too. Key derivation works the same way: it runs in a worker thread that ends after each derivation. For strings, ending the process is the only reliable way to end their life, and even then the memory becomes unreachable without being erased.&lt;/p&gt;
&lt;h2 id=&quot;why-clearing-state-is-not-enough&quot;&gt;Why clearing state is not enough&lt;/h2&gt;
&lt;p&gt;Local Ledger is a personal finance desktop app. Its data sits in a database encrypted with SQLite3 Multiple Ciphers in the SQLCipher 4 format, whose key is sealed under a key that Argon2id derives from the master password. The natural way to implement “lock” in a React app is to close the database, clear the store and show the password screen.&lt;/p&gt;
&lt;p&gt;That leaves the password where it was typed. It was the value of an input element, then React state, then an argument sent over a port to the vault host. Each of those is a JavaScript string, and strings cannot be wiped. Assigning an empty string to a variable drops one reference and leaves the characters in the heap until garbage collection reclaims them, and V8 may also have copied the string while compacting the heap. Clearing a store changes only what the app can reach.&lt;/p&gt;
&lt;p&gt;Binary secrets are easier, because the app can zero a buffer it owns. The data key, the sealing key, the encoded password and the raw key literal live in buffers the app owns and fills with zeros after use, also when an operation fails or is cancelled. While the vault is unlocked, the data key is only inside SQLite, in the vault host. Copies that the key derivation or SQLite make outside these buffers are not wiped by the app; they go when the host ends. The trouble is the text that has to pass through strings on its way into those buffers.&lt;/p&gt;
&lt;h2 id=&quot;replacing-the-page-process&quot;&gt;Replacing the page process&lt;/h2&gt;
&lt;p&gt;D016 hosts the page in a &lt;code&gt;WebContentsView&lt;/code&gt;. On every unlock, lock and committed password change, the main process loads the page in a new view and closes the old one, whose renderer process then exits. Two views of the same origin get separate processes. A lock replaces the page even when the vault is already locked, which means a password typed into the unlock screen but never submitted does not survive it either.&lt;/p&gt;
&lt;p&gt;Key derivation follows the same rule. D015 runs Argon2id (64 MiB of memory, 3 passes, 4 lanes) in a worker thread that ends after each derivation, releasing its 64 MiB. The worker also keeps the window responsive. On the one machine measured (Electron 45, a 2.1 GHz Xeon with 4 cores), a derivation takes 321 ms; on the main thread it blocked the event loop for 281 ms, and in the worker for 7 ms.&lt;/p&gt;
&lt;p&gt;The steps that run at lock time need the vault to lock, so it also locks on its own: after a set idle time (1, 5, 15 or 30 minutes, 5 by default, with no option to turn it off), when the screen locks, when the computer sleeps and when the window closes.&lt;/p&gt;
&lt;h2 id=&quot;tests&quot;&gt;Tests&lt;/h2&gt;
&lt;p&gt;The end-to-end suite tests the processes. One helper reads the operating system process ID of the page currently shown, another lists the vault host’s, and a third polls until a process no longer exists. The suite records the renderer’s PID before an action that should end the page’s process and waits for it to exit afterwards, across the vault, import, transaction and memory tests. They cover creating a vault, unlocking after a wrong password, locking, closing the window, a lock that arrives while an unlock is still deriving its key, a password typed but not submitted, and a password change. After a lock the suite also waits for the host to exit, and checks that the next page gets a new one.&lt;/p&gt;
&lt;p&gt;Heap snapshot tests look at the main process, which does not restart. The one for transactions plants a known control string in the main process, adds a transaction with a random marker in its description, locks, waits for the host that read it to exit, and takes a heap snapshot. A heap snapshot collects garbage first, so it contains only what is still reachable. The test requires the control string to be found, which shows the snapshot can see such strings, and requires the marker to be absent, which shows that the row’s text is not reachable in the process that outlives the lock. A heap snapshot cannot see the contents of Buffers or memory already freed.&lt;/p&gt;
&lt;p&gt;Since 0.3.0 one test reads memory itself. On Linux, Windows and macOS it reads the memory of every process of the app three times: unlocked, after a lock once the host and the page have exited and the main process has collected garbage, and 30 s later. It counts copies of a typed description, two imported ones, the 40 seeded descriptions, the master password, the sealing key and the data key. It fails if the main process holds any of them, or if any process does after a lock, apart from two kinds of copy in the main process: text typed into the page, and the page of a PDF export. On macOS it reads the pages the system counts as modified or compressed, so its counts there are a lower bound. Run against 0.2.1, where the main process held the vault, the test found there after a lock, on Linux, 38 copies of the imported text, 36 of the 40 seeded descriptions, the master password, the sealing key and the second half of the data key. D047 records that finding and moves the vault into the host.&lt;/p&gt;
&lt;h2 id=&quot;limits&quot;&gt;Limits&lt;/h2&gt;
&lt;p&gt;These come from the project’s own security design, which lists what the app does not claim.&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Strings stay until their process ends.&lt;/strong&gt; The password is a string in the renderer that took it until that renderer ends, and in the vault host, after crossing its port, until the next lock ends the host. Decrypted rows become strings in the host and stay there until garbage collection, or until the host ends. A call under way at a lock keeps its data there until it settles; a host still running one 30 s after the lock is killed.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Freed memory is not overwritten.&lt;/strong&gt; Memory released by a worker, the vault host or a renderer goes back to the operating system as it is. The app does not scrub it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;SQLite’s cache.&lt;/strong&gt; While unlocked, SQLite keeps up to 16 MB of decrypted pages in the vault host. Closing the database frees them without overwriting, since this build leaves SQLite3 Multiple Ciphers’ secure memory option off.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;The main process still receives some text.&lt;/strong&gt; A PDF export hands the page of the month’s statement to the main process, which prints it, and copies of its text stay there after a lock. Copies of a description typed into the page also appear there, though the app’s calls do not carry it; they come from Chromium or from the test driver that types it, and the test does not tell which.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Swap and hibernation.&lt;/strong&gt; The operating system may write any of this memory to disk. The app does not lock pages in memory.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Same-user malware is out of scope.&lt;/strong&gt; While the vault is unlocked, such malware can read process memory or the screen; while it is locked, it can record the password at the next unlock.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A failed password change keeps its page.&lt;/strong&gt; It needs to report the failure, so it clears its fields, but the typed passwords stay in that renderer as strings until the next lock ends it.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;After a lock, once the calls under way have settled, no process that opened the vault or showed its rows is still running. The tests check that the old processes exit, that the planted text is unreachable in the main process, and that a read of every process’s memory finds none of the counted text and keys beyond the two kinds of copy in the main process. On macOS that read is a lower bound, and nothing here promises that released memory was erased.&lt;/p&gt;
&lt;h2 id=&quot;when-this-does-not-hold&quot;&gt;When this does not hold&lt;/h2&gt;
&lt;p&gt;The approach depends on owning the process model. Electron lets the app create and destroy renderer processes. A web page has no such control, so a browser-based app cannot use this method and has to drop references and wait for the garbage collector.&lt;/p&gt;
&lt;p&gt;It also assumes the threat that the project’s model names: someone at the computer after the app has locked itself. Malware already running as the same user is outside that model. If an attacker can read memory while the vault is unlocked, the data is exposed anyway, since it has to be decrypted to be shown.&lt;/p&gt;
&lt;p&gt;The approach also has a cost. Each lock reloads the page and discards its state. For a vault whose lock screen already asks for the password again, that cost is small. An app that is expected to resume exactly where the user left off would need to persist that state somewhere, and that state is what the lock was meant to remove.&lt;/p&gt;</content:encoded></item><item><title>The cryptography was the part that held</title><link>https://shyanorn.github.io/articles/cryptography-held/</link><guid isPermaLink="true">https://shyanorn.github.io/articles/cryptography-held/</guid><description>A self-assessment of an end-to-end encrypted file sharing service: none of its five High findings involved cryptography; four came from the upload lifecycle under concurrency, retries and deletes, and the fifth from cleanup when the database and the storage disagreed.</description><pubDate>Mon, 09 Nov 2026 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;&lt;strong&gt;This is a self-assessment.&lt;/strong&gt; Every finding below comes from the project’s own reviews and tests. No outside party has audited vault-share.&lt;/p&gt;
&lt;p&gt;vault-share is an end-to-end encrypted file sharing service. The browser encrypts a file before upload, and the key travels only in the part of the link after the &lt;code&gt;#&lt;/code&gt;, which browsers never send. In a project like this, the cryptography is the obvious place to look for security bugs. The findings log holds 70 entries, and none of the five rated High involves a cipher, a key derivation or a nonce. Four are about the lifecycle of an upload: what happens when it races a delete, a retry, a timeout or another upload. The fifth came from cleanup when the database and the storage disagreed. A second group of findings had a common shape, a limit that was checked before an attempt and counted after it, which concurrent requests could slip past.&lt;/p&gt;
&lt;h2 id=&quot;how-the-findings-were-found&quot;&gt;How the findings were found&lt;/h2&gt;
&lt;p&gt;The log records the source of each finding. Most came from reviews of milestones M2 to M7 or from an adversarial re-check of the fixes for earlier findings. Others came from a pass against the OWASP Top 10:2025 using black-box and browser test scripts, a production readiness review and a code review, two from a simplification review, one from a second implementation of the share format, one from an end-to-end test, and one came up while fixing another finding. Fixes such as those for F26, F38 and F39 record tests or measurements from before and after.&lt;/p&gt;
&lt;h2 id=&quot;four-high-findings-in-the-upload-path&quot;&gt;Four High findings in the upload path&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;F1, orphaned ciphertext.&lt;/strong&gt; The upload handler published the blob before marking the record. If the record had been deleted or purged in between, the blob stayed on disk with nothing pointing to it, outside every cleanup rule. Each repetition could leak up to the file size limit.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;F2, concurrent uploads replaced acknowledged content.&lt;/strong&gt; A second PUT with the same owner token overwrote the stored blob, and a bad body on a late PUT deleted the blob of an earlier successful one.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;F3, long uploads stored but not acknowledged.&lt;/strong&gt; Only the read deadline was extended for large bodies. The server’s 60-second write timeout cut the connection before the reply reached the client, which then retried and ran straight into F2.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;F26, concurrent uploads to one file.&lt;/strong&gt; Nothing stopped an owner from running many uploads to one record at once. The S3 store buffered each in memory, and 40 uploads to one record took the server from 18 to 341 MiB. On disk, six uploads to one 64 MiB record held 360 MiB, beyond a quota that counted records only. After the fix a file takes one upload at a time, so six uploads to one record left one temp file and five 409 replies. The S3 store now streams each body instead of buffering it, and ten uploads to ten records moved the server from 19 to 20 MiB; streaming caps files on S3 at 5 GiB.&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;F2 and F3 are connected: the timeout in F3 caused retries, and through F2 those retries could replace or delete data the server had already acknowledged.&lt;/p&gt;
&lt;h2 id=&quot;the-fifth-high-finding&quot;&gt;The fifth High finding&lt;/h2&gt;
&lt;p&gt;F47 came from a later production readiness review. It involves no cryptography either, and it has nothing to do with uploads. vaultd created its database when the file was missing, and the cleanup at startup then deleted every stored object that had no record, as an orphan. A lost or replaced data volume, a restore that left the database out, or a second instance pointed at the same bucket therefore emptied the storage within seconds: 300 blobs on disk and 30 objects in an S3 bucket in the reproduction. No request can trigger it, but the loss is every shared file. vaultd now refuses to start when the database is new and the storage already holds content, runs its first cleanup an hour after the start, and deletes at most 100 objects in one run (D032). An existing database pointed at another instance’s storage still loses that content, 100 objects an hour, so the bucket must belong to vaultd alone.&lt;/p&gt;
&lt;h2 id=&quot;limits-that-were-checked-before-they-were-counted&quot;&gt;Limits that were checked before they were counted&lt;/h2&gt;
&lt;p&gt;Two later findings, F38 and F39, came from rate limits and lockouts that each looked correct when read on their own.&lt;/p&gt;
&lt;p&gt;The second sign-in step read its ticket, checked the code, and only then counted a failure. The account’s lockout was asked before the check and told after it. Codes sent at the same moment all passed both gates before any failure was recorded, which gave an attacker who already knew the password many more guesses at the second factor. In a test, 40 wrong codes on one ticket had 7 to 20 of them checked, where the ticket allows 5. For sign-ins, 50 wrong sign-ins sent at once from 50 addresses were all checked where the account allows 20, and 20 bad download signatures at once had 11 to 14 checked where 5 should lock the client out of the file.&lt;/p&gt;
&lt;p&gt;The fix reverses the order. A lockout now admits an attempt before it is checked, and refuses one while the failures already counted plus the attempts in flight could reach the limit. A ticket gives up an attempt in the same statement that reads its count. The tests send the attempts at once and failed before the fix. Afterwards at most 5 codes are checked, whether they arrive on one ticket or over eight, and exactly 20 sign-ins from many addresses, 5 from one client and 5 signatures.&lt;/p&gt;
&lt;p&gt;This is counting code with no cryptography in it, and it sets how many guesses an attacker gets against a six-digit code.&lt;/p&gt;
&lt;h2 id=&quot;why-the-cryptography-held-up&quot;&gt;Why the cryptography held up&lt;/h2&gt;
&lt;p&gt;The cryptographic design deliberately avoided anything new. D002 restricts the project to the browser’s Web Crypto API and mature libraries, on the grounds that hand-written primitives are error-prone. D012 adopts an existing format as well: files are encrypted in the record format of RFC 8188, the &lt;code&gt;aes128gcm&lt;/code&gt; content encoding, which Firefox Send also used. Keeping that layout lets the RFC’s section 3.1 test vector run against this implementation unchanged, with the RFC’s 128-bit key; files themselves use a 256-bit key. The section 3.2 vector carries a key id, which vault-share rejects, so it runs only with the key id removed. Where vault-share differs from Firefox Send, it is stricter. Among other differences, it rejects unexpected header values, which Firefox Send ignores, and an empty file still produces a final record, so a stream cut after its header fails to decrypt instead of producing an empty file.&lt;/p&gt;
&lt;p&gt;The only finding filed under Cryptographic Failures (A04:2025), F20, was a mismatch between the documents and the code. The specification said the server stores only a verifier of the download authentication key. It actually stores the key itself, as Firefox Send does, so a copy of the database could sign download challenges and spend download counts. It still could not decrypt anything, since that needs the secret in the link. The documents were corrected. An asymmetric verifier would close the gap, at the cost of a protocol change.&lt;/p&gt;
&lt;h2 id=&quot;limits&quot;&gt;Limits&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;Self-assessed.&lt;/strong&gt; The findings come from the project’s own milestone reviews, adversarial re-checks of fixes, a test pass against the OWASP Top 10:2025, a production readiness review and a code review. No external audit has been done, and an outside reviewer may well find problems of a different kind, cryptographic ones included.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Severity is the project’s own.&lt;/strong&gt; Each rating follows the impact on the project’s own threat model. A different model would rank some of them differently.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;No cryptographic weakness found.&lt;/strong&gt; One finding, F70 (Low), was at a key derivation’s input: a share password typed in another Unicode form did not open its link. That none was found does not show that none exists. The RFC test vectors show the record format is implemented as specified; they say nothing about side channels or about how the keys are used beyond the file format.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Scanner results say little here.&lt;/strong&gt; gosec, govulncheck, npm audit and Semgrep report 0 issues on the source at commit a277b92; govulncheck on the Caddy binary reports 4 advisories, accepted as F22. Nothing here shows whether the scanners would have caught the findings above.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Known gaps remain.&lt;/strong&gt; A compromised server can serve JavaScript that steals keys, as with most web-based encryption tools, and the server sees each file’s size and access times.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;when-this-does-not-hold&quot;&gt;When this does not hold&lt;/h2&gt;
&lt;p&gt;The pattern depends on two choices made at the start: standard primitives, and an existing, reviewed format with test vectors. A project that designs its own record format, manages nonces by hand, or composes primitives into a new protocol moves the risk back into the cryptography, and review effort should follow it there.&lt;/p&gt;
&lt;p&gt;It also depends on the kind of server. vault-share’s server never sees plaintext, so most of its work is storage bookkeeping, under concurrency or when the database and the storage disagree, and that is where its High findings were. A service whose core logic is cryptographic, such as a key server or a signing service, will have a different distribution.&lt;/p&gt;
&lt;p&gt;For a project shaped like this one, I would spend most review time on the upload state machine, the retry paths, any cleanup that deletes what the database does not list, and any limit that is checked in one place and counted in another. A borrowed, reviewed format keeps the cipher code small and testable, which makes those parts the most likely to be wrong.&lt;/p&gt;</content:encoded></item><item><title>The order book was 2% of the CPU</title><link>https://shyanorn.github.io/articles/order-book-cpu/</link><guid isPermaLink="true">https://shyanorn.github.io/articles/order-book-cpu/</guid><description>The matching core applied a command in 144 to 198 ns, but end to end the order book was 1.4% to 2% of the engine&apos;s CPU, and the latency gains came from a JSON key check and an fsync moved off the matching goroutine.</description><pubDate>Mon, 09 Nov 2026 00:00:00 GMT</pubDate><content:encoded>&lt;p&gt;The order book’s data structure in matching-engine was chosen by benchmark, and end to end it barely shows up. On its own, the matching core applied a command in 144 to 198 ns. Under load the full engine spent 39 to 95 µs of CPU per request, and in CPU profiles of the running engine &lt;code&gt;Book.Apply&lt;/code&gt; accounted for 1.4% to 2% of the engine’s CPU. The two changes that improved latency were a faster check of JSON keys and moving an fsync onto another goroutine, and neither of them touched the book.&lt;/p&gt;
&lt;h2 id=&quot;the-order-book&quot;&gt;The order book&lt;/h2&gt;
&lt;p&gt;matching-engine is a limit order book in Go. Each symbol has one goroutine that owns its book and its command log, so the book needs no locks and replays deterministically (D003). Price levels sit in a hand-written red-black tree with a FIFO queue per level (D004).&lt;/p&gt;
&lt;p&gt;D004 came out of a measurement. Three prototypes, a red-black tree, a skip list and a sorted slice, ran the same model tests and benchmarks. At 10,000 price levels, removing one level and adding one near the top took 250 ns in the tree and 405 ns in the skip list. Across the sizes tested, the skip list’s writes were 26% to 101% slower. The core benchmark, which replays the project’s reference order flow, came out at 144 to 198 ns per command (5.05M to 6.95M commands per second) with up to 1M resting orders in the book.&lt;/p&gt;
&lt;p&gt;Very little of that speed reached a client.&lt;/p&gt;
&lt;h2 id=&quot;measuring-end-to-end&quot;&gt;Measuring end to end&lt;/h2&gt;
&lt;p&gt;The end-to-end benchmark uses an open-loop load generator (D030). Request i is due at t0 + i/rate regardless of the replies, and latency counts from that due time, so a stall is charged to every request it delays. Each rate runs three times on a fresh engine. A run counts only if 99% of the target rate went out, no request failed, and every trade was pushed once and in order.&lt;/p&gt;
&lt;p&gt;Dividing the engine’s CPU time by the requests it answered gives its CPU per request. It was 95 µs at 10k requests/s and fell to 39 µs at 50k/s, because fixed costs such as idle wakeups and GC are spread over more requests. At 50k/s the engine alone takes about two cores.&lt;/p&gt;
&lt;p&gt;The CPU profiles were taken during the optimization work, which ran on a different VM of the same type. That VM was slower than the one behind the headline figures, so shares within a profile can be compared with each other, while absolute times cannot be compared across the two machines. In a profile at 25k/s, taken before any optimization, the CPU went here:&lt;/p&gt;
&lt;ul&gt;
&lt;li&gt;reading and writing HTTP: about 19% in system calls alone&lt;/li&gt;
&lt;li&gt;decoding request bodies: 14%, of which 9% was the check of their keys&lt;/li&gt;
&lt;li&gt;garbage collection: about 10%&lt;/li&gt;
&lt;li&gt;the matching goroutine, book included: about 10%&lt;/li&gt;
&lt;/ul&gt;
&lt;p&gt;Across this profile and the one taken after both changes, &lt;code&gt;Book.Apply&lt;/code&gt; on its own was 1.4% to 2% of the engine’s CPU. The book has since gained its own order index and reuses the nodes of orders that leave it (D038, D039); the end-to-end figures were measured before that and have not been measured again.&lt;/p&gt;
&lt;h2 id=&quot;the-two-changes-that-helped&quot;&gt;The two changes that helped&lt;/h2&gt;
&lt;p&gt;Both were measured on the slower VM, with a full sweep for each step.&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Checking request keys with one scan.&lt;/strong&gt; A request body has to be one JSON object with exactly the expected field names. The original check walked the body with &lt;code&gt;json.Decoder&lt;/code&gt; tokens, and the decoder builds a syntax error after each key and scalar and then throws it away. The new check runs &lt;code&gt;json.Valid&lt;/code&gt; and then scans once for the keys, and a fuzz test holds it to the old check’s answers. It went from 5.79 µs and 79 allocations to 0.51 µs and none. Its share of the CPU profile fell from 9.2% to 1.1%, CPU per request fell 5% to 8%, and 30k requests/s became sustainable on that VM. Before the change, this one input check used almost as much CPU as the whole matching goroutine (9% against about 10%).&lt;/p&gt;
&lt;p&gt;&lt;strong&gt;Running the fsync beside the matching goroutine (D031).&lt;/strong&gt; The command log is fsynced every 10 ms. At first the symbol’s goroutine did this itself and stopped for about 0.8 ms on average, sometimes for tens of milliseconds, while every queued command waited. A helper goroutine now runs the fsync, which is also how Redis, Kafka and PostgreSQL handle their logs. At 10k/s, p90 fell from 0.91 to 0.57 ms and p99 from 5.32 to 3.82 ms. At 25k/s the three runs’ p99 went from 20.0, 36.4 and 215 ms to 19.6, 19.8 and 28.7 ms. CPU per request rose by up to 3%. The crash window also grew: a machine crash can now lose up to one interval of commands, plus any taken while an fsync was still running.&lt;/p&gt;
&lt;p&gt;On the faster VM, the headline end-to-end runs reached a p99 from order to trade push of 1.76 ms at 10k requests/s. 50k/s is near saturation for this setup: the three headline runs had a median p99 of 41.1 ms, and the same sweep on a second 4 vCPU VM had 141.1 ms. At 50k/s the p99 depends on the machine.&lt;/p&gt;
&lt;h2 id=&quot;limits&quot;&gt;Limits&lt;/h2&gt;
&lt;ul&gt;
&lt;li&gt;&lt;strong&gt;One machine.&lt;/strong&gt; The headline end-to-end figures come from one 4 vCPU VM, and the load generator shares those 4 vCPUs with the engine. The D004 prototype benchmarks were run separately, on a VM of the same type. A second VM of the same size reproduced the 10k/s figure (1.75 ms against 1.76 ms) but not the 50k/s one; its reports are not in the repository.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Earlier code.&lt;/strong&gt; The headline end-to-end runs used Go 1.24.7 and predate later changes to the engine’s API code; the repository holds no end-to-end run on that VM after them. The project now builds with Go 1.27.4.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;Two VMs.&lt;/strong&gt; The optimization comparisons ran on a slower VM of the same type. Its absolute latencies are not comparable with the headline end-to-end figures.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A noisy host.&lt;/strong&gt; A 1 ms sleep loop in the load generator recorded pauses of up to 53 ms in one run on the faster VM, and a run on the slower VM recorded a stall of about 128 ms. Anything above p99 depends on single pauses like these. Three runs per rate show the spread; they do not remove it.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;One symbol, one protocol.&lt;/strong&gt; Every end-to-end run used one symbol, JSON over HTTP, and 64 keep-alive connections.&lt;/li&gt;
&lt;li&gt;&lt;strong&gt;A missed target.&lt;/strong&gt; The spec asked for a p99 under 5 ms at 50,000 orders/s, about 71k requests/s in this flow. On this machine the p99 stays under 5 ms only up to 15k requests/s.&lt;/li&gt;
&lt;/ul&gt;
&lt;h2 id=&quot;when-this-does-not-hold&quot;&gt;When this does not hold&lt;/h2&gt;
&lt;p&gt;This is a result about proportions, and the proportions depend on the transport. With a binary protocol, batched submissions or a kernel-bypass network stack, there is less overhead around each request, so the book’s share of the CPU grows. With many active symbols per core, the per-symbol work adds up while some fixed costs stay shared.&lt;/p&gt;
&lt;p&gt;Startup is the one path in this project where the book is a much larger share of the cost. The engine rebuilds each book by replaying its command log, with no HTTP involved. Before D038 and D039, replay ran at 534 ns per command, about 3.7 times the cost of &lt;code&gt;Book.Apply&lt;/code&gt; alone, which put the book at roughly 27% of it. The rest goes to reading, checksumming and decoding each record and rebuilding the order history. A day of load at 10k requests/s would take about eight minutes to replay from the page cache, and a cold start also waits for the disk.&lt;/p&gt;
&lt;p&gt;D004 was also chosen for two properties a throughput profile does not show. A red-black tree has a guaranteed worst case and uses no randomness, so the same command sequence always builds the same tree. A skip list’s seed can be attacked: in testing, crafted prices raised the average lookup steps near the top of the book from 18 to 517. Before spending another round of optimization on a component, profile the whole request path to see how much of the CPU it uses.&lt;/p&gt;</content:encoded></item></channel></rss>