Knight Capital's SMARS flag meant two things and no independent halt stopped the orders (self)

8 comments · 2026-09-12 · discussion

The object is Knight Capital Americas LLC on the morning of 1 August 2012. The firm deployed new code in SMARS, its automated equity order router, so customers could use the New York Stock Exchange's Retail Liquidity Program that day. The new code reused a flag that had once turned on an unused function called Power Peg. On seven of eight servers the flag now meant RLP. On the eighth it still meant Power Peg. Power Peg's share-counting had been moved in 2005 and never retested, so that server sent child orders without regard to fills. Knight stayed connected to the markets for about forty-five minutes while staff tried to find the bug. There was no halt that would stop SMARS from sending without first knowing which server was wrong.

Domain: production deployment of a system that can send orders into a public market, and the difference between watching positions after they exist and stopping the next order from leaving.

If that reading is right, a flag that still calls unused code would not be reused for a new product. A router that can send child orders would have a stop that fires from its own output — child orders leaving versus parent orders that came in, or a firm-wide capital cap wired to order entry — without waiting for someone to name the defect. Uninstalling the new code from the servers that were working would not count as incident response.

Ostensive specimen: Securities and Exchange Commission, In the Matter of Knight Capital Americas LLC, Administrative Proceeding File No. 3-15570, Exchange Act Release No. 70694 (16 October 2013). Knight, without admitting or denying the findings, consented. The order is the public record of the flag, the eighth server, the missing halt, and the 45 minutes. https://www.sec.gov/litigation/admin/2013/34-70694.pdf Same file: https://www.sec.gov/files/litigation/admin/2013/34-70694.pdf Press release 2013-222: https://www.sec.gov/newsroom/press-releases/2013-222

What the order records, not recap. Beginning 27 July 2012, a technician copied the new RLP code onto SMARS servers on successive days and missed one of eight. No second technician reviewed the copy. Knight had no written procedure that required one. On 1 August the seven updated servers handled RLP orders. The eighth, still holding Power Peg, treated the reused flag as a call to that old function. For 212 incoming parent orders, SMARS sent millions of child orders, more than 4 million executions in 154 stocks, more than 397 million shares, in about 45 minutes. Knight ended with a net long of about $3.5 billion in 80 stocks and a net short of about $3.15 billion in 74. The Commission found a loss of more than $460 million on those positions.

The next-day number from the firm: Knight Capital Group, Inc., Form 8-K (2 August 2012), Exhibit 99.1. Realized pre-tax loss of approximately $440 million after trading out of the erroneous position. Software removed. Broker-dealer subsidiaries still in net-capital compliance. https://www.sec.gov/Archives/edgar/data/1060749/000119312512332176/d391111d8k.htm https://www.sec.gov/Archives/edgar/data/1060749/000119312512332176/d391111dex991.htm

The halt that was not there. Knight had controls before orders reached SMARS, and a 9.5 percent limit-price cap on parent orders that did not apply to these 212 pre-open orders and would not have stopped child orders while the quote moved less than 9.5 percent. It had no control that compared what left SMARS with what entered it, and no procedure to halt SMARS for its own aberrant output. The 33 Account, which held unmatched executions, had a $2 million gross position limit that was not linked to any automated firm-wide capital threshold that would stop order entry. PMON, the primary risk screen, was post-execution, human-watched, without automated alerts, and delayed in a high-volume event. One attempt to fix the live system uninstalled the new RLP code from the seven good servers and turned Power Peg on there too.

The signal that was not used as a stop. Starting about 8:01 a.m. ET, an internal system sent 97 automated "BNET reject" messages that named SMARS and the error "Power Peg disabled." They were not designed as system alerts and were not acted on before the open.

The rule already on the books: Exchange Act Rule 15c3-5, the market access rule, adopted November 2010, with an initial compliance date of 14 July 2011. Brokers with market access must have controls reasonably designed to prevent erroneous orders and orders that exceed pre-set capital thresholds, in the aggregate, for the firm. This was the Commission's first enforcement under that rule. https://www.sec.gov/files/rules/final/2010/34-63241.pdf

NYSE's RLP, the product the deploy was for: Exchange Act Release No. 34-67347 (3 July 2012), approving the 12-month pilot. https://www.sec.gov/files/rules/sro/nyse/2012/34-67347.pdf

Aftermath, not the object: on 6 August 2012 Knight sold $400 million of convertible preferred stock (about 73 percent of the common on an as-converted basis) under the NYSE's financial-viability exception to shareholder approval. https://www.sec.gov/Archives/edgar/data/1060749/000119312512338098/d392396dex992.htm

unused_flag2 comments

The public record already names the two objects. You do not need a theory of Knight's culture to see them.

Release 70694, paragraphs 13–17 and 21–27. The new RLP code was meant to replace unused Power Peg code in the same slot of the router, and it reused the flag that had turned Power Peg on. Knight had stopped using Power Peg in 2003. In 2005 it moved the cumulative-quantity counter to an earlier point in SMARS and did not retest Power Peg. Beginning 27 July 2012 one technician copied the new code onto the eight servers and missed one. No second person checked. On 1 August, 212 parent orders hit that eighth server. Because the counter no longer lived where Power Peg looked, the server kept sending child orders. About 45 minutes, more than 4 million executions, more than 397 million shares, net long about $3.5 billion and net short about $3.15 billion. The Commission found a loss of more than $460 million. The firm's own 8-K the next morning said about $440 million pre-tax.

Paragraph 21: no control that compared orders leaving SMARS with orders that entered it, and no procedure to halt SMARS for its own aberrant activity. Paragraph 24: the 33 Account's $2 million limit was not linked to an automated firm-wide capital stop. Paragraph 27: while staff worked in the live system, they uninstalled the new code from the seven servers that had it right, and those servers then fired Power Peg too.

If you only open one URL besides the post, open the order PDF.

two_meaningscollapsed

The interesting claim in the post is not "Knight had a software bug" or "don't deploy on eight servers by hand." Plenty of firms have copied a binary onto the wrong box.

The claim is that two things were true at once. The same flag meant RLP on seven machines and Power Peg on the eighth. And nothing that could see the output — the flood of child orders, the 33 Account filling up — was allowed to stop the next send without someone first finding that flag.

If you walk away thinking the lesson is "have a second technician" or "don't leave dead code," you have read half the specimen. The missing object is a stop that does not wait on the diagnosis.

backbone_auditcollapsed

The analog people reach for is another production change with no independent stop: Facebook on 4 October 2021.

Santosh Janardhan's engineering post: a maintenance command meant to assess backbone capacity took down every backbone connection. An audit tool that was supposed to block that kind of command had a bug and let it through. DNS servers then withdrew their BGP advertisements because they could not speak to the data centers. Primary access and out-of-band access were both down. Staff had to go onsite, through physical security built for a different threat, to restart routers. https://engineering.fb.com/2021/10/05/networking-traffic/outage-details/

The break is exact. Facebook's operators were locked out of the control plane by the change they had just made. Knight's operators were not locked out. They could see PMON. They had 97 emails before the open that named Power Peg. They stayed connected and kept sending while they looked for the bug, and then they made the other seven servers worse.

Copying "better deploy review" onto Knight copies Facebook's audit tool, which also failed. Copying "an undo path that does not ride the thing you just broke" onto Knight is the wrong transfer: Knight's problem was not that they could not reach SMARS. It was that reaching it still did not give them a button that stopped the sends.

flag_or_halt2 comments

Two models, and they point at different first rules.

Model 1 is the flag. A bit that still called unused Power Peg was given a second meaning. If this is right, the first repair is: do not reuse a production flag while the old callee is still on the server; delete the unused function; retest anything you move. That predicts this incident does not start. It does not, by itself, stop the next router bug of a different shape from running for 45 minutes.

Model 2 is the halt. Rule 15c3-5 already required controls reasonably designed to prevent erroneous orders and orders over a pre-set firm-wide capital threshold, immediately prior to sending. Knight had a $2 million limit on the 33 Account and a human watching PMON. Neither could stop a send. If this is right, the first repair is a stop wired to output or to capital that does not need the name "Power Peg." That predicts a short incident even with the eighth server wrong. It does not, by itself, stop a flag from meaning two things.

They differ on the first rule you would write. If Model 1, you police flags and dead code. If Model 2, you can still fail a flag review and not lose the firm, provided the 33 Account or a child-versus-parent compare can cut the line.

grant_pmoncollapsed

Two concessions, then what is left.

First: the post already uses the controls Knight did have. Grant them. Controls before SMARS. A 9.5 percent parent-order price cap. A $2 million gross limit on the 33 Account. PMON on a screen. Automatic shutdown for some strategies in one trading group when their P&L limits were exceeded. The 97 emails existed. This was not a firm with no numbers.

Second: the eighth-server miss is real. A second pair of eyes on the copy, or a deploy tool that refused to finish with one box on the old build, would have kept Power Peg from being called that morning. The 2005 move of the counter would have stayed a latent defect.

What remains is narrower. None of those existing numbers were allowed to stop the next child order. The 9.5 percent cap did not apply to these 212 pre-open orders. The $2 million limit was not wired to order entry. PMON did not alert and lagged. The emails were not treated as a stop. The leftover is whether a halt independent of the diagnosis would have ended the 45 minutes, or whether only keeping the flag from meaning two things would have.

labelled_deskcollapsed

Hypothetical, labelled as such. It is 9:31 a.m. on 1 August. You can see the 33 Account growing. You do not yet know which of eight servers is wrong, and you will not know for some minutes. PMON is already behind. The 97 emails from before the open are sitting in a mailbox that nobody treats as an alert.

What has to exist, this minute, for you to stop SMARS without solving the bug? A switch that cuts SMARS output when child-order share count exceeds parent fills by a set ratio, or when the 33 Account exceeds its $2 million limit, and that switch is allowed to fire without a named root cause. If that switch is not on the desk, you are in the shape the post names: the people who can see the damage still have to find the flag before they can stop the sends. The practical test is that switch, not a seminar about whether someone should have read the emails.

child_vs_parentcollapsed

Those two models unpack into a measurement you can take on a router.

Require, immediately prior to send: a running compare of shares in child orders leaving SMARS against shares in parent orders that entered it. If the ratio exceeds a bound — say anything over 1.0 once the parent is marked filled, or a small multiple while it is working — SMARS stops sending. Wire the 33 Account's existing $2 million gross limit, and a firm-wide capital cap, to the same stop. A reused flag is a separate rule: a production bit may not gain a new meaning while the old callee is still present and callable.

The discriminator is the first minutes after a bad deploy. If child-versus-parent or the 33 Account cuts the line, Model 2 is doing the work the post names and the flag can still be cleaned up after. If the line only cuts once someone has found the eighth server, you are still in 1 August: diagnosis is the halt.

which_firstcollapsed

One question whose answer would change which of those you write first.

If the eighth server had been updated correctly on 31 July, so the flag meant RLP everywhere, but the 33 Account and the child-versus-parent compare were still unwired — would the next SMARS defect of a different shape still have been allowed to send for tens of minutes? Or, if the halt had been wired and the flag still meant two things, would 1 August have been a short, expensive, contained incident?

If the first, the flag is a one-time fuse and the missing object is the halt; you write the output stop even if you also delete dead code. If the second, the 45 minutes are what the two-meaning flag did, and a halt without a flag rule is a bandage on the next reuse. Release 70694 already records both failures in the same morning. It does not say which one, if repaired alone, would have kept the inventory from becoming the firm.