Silent failures in trading automation: the three that cost the most

VigilDesk engineering notes

Every automation failure I have paid for fell into the same category: nothing threw. The logs were clean, the code was correct, and the system did exactly what it was told - with the wrong inputs. Three of these are worth knowing by name, because the fix for each is a few lines and the detection is one cheap test.

1. The guard that reads zero

A margin or drawdown guard is one of the simplest shapes in trading software: read a number, compare it to a threshold, refuse to act if it is too big. It is also one of the easiest things to connect to the wrong source.

I have seen this in production: the rule ran on schedule, the comparison was correct, and the number it read was always 0 - because the snapshot it read from was refreshed by a different code path that had silently stopped updating. Nothing failed. The guard was decorative for weeks.

Detection: assert on the effective value the decision uses, not on the value the config intended. If production reads state.margin_used, the test must assert on state.margin_used - not on a freshly computed copy that happens to be correct.

2. The config that quietly overrides the code

A position-sizing ladder returned 1.5 or 2.0. An int() conversion downstream truncated it to 1. At the configuration value the code was written against, the truncation was invisible; after a config change two months later, every position ran at half the intended leverage. The code was right under a configuration nobody remembered changing.

Detection: log the number that was actually used, every cycle, not the constant it was supposed to come from. leverage=1.0 in a log line on day one beats a month of "the code looks correct".

3. The state that dies with the process

This is the one I care about most, because it is the failure mode of every in-terminal risk limit: the counter lives in memory, so a restart, a recompile, a deploy or a VM migration resets it. The rule "never risk more than X% today" politely hands back the full X% - exactly on the day things are going badly, because that is when restarts happen.

Detection: one destructive test. Let the limit trip, then kill the process with a position open, restart it, and watch what the guard does on its next cycle. If it happily continues, the state was in RAM.

Fix: persist the state - which trading day it belongs to, the anchor it measures against, and the decision flag - and write it after the decision, never before. Derive "today" from the broker's clock rather than the host clock, or a timezone change moves your daily boundary without anything throwing. There is a longer write-up of this one, with the implementation, in the daily-loss-limit note.

The pattern behind all three

Successful silent failures share a shape: the thing that is asserted on is not the thing that production reads. Tests check the intended value; monitors check the process is alive; reviewers check that an assertion exists. None of them check the join between intent and reality.

Three habits, all cheap:

• assert on effective values, from the same object production reads;
• log the effective values, not the configured ones;
• have at least one test that destroys the process rather than calling a function.

Where this shows up in trading specifically

Trading automation concentrates these failures, because the environment is hostile in ways unit tests do not simulate: the process is restarted by the user at 3am, the clock is in a different timezone than the broker, the broker disconnects mid-trade, and the platform may recompile your code without asking. A guard that behaves correctly under pytest can still be decorative under those conditions - and you will not find out until it matters.

Related notes

• Your daily loss limit resets when MetaTrader restarts. Here is the fix.
• What a kill switch should actually do (and the four ways they fail)
• Which day is it? Broker time, host time, and the trading-day boundary

Get the guard

Free, restart-proof MT5 daily-loss guard, source included: https://xuks124.github.io/vigildesk/free.html

Risk disclosure

Algorithmic trading carries both technical and market risk. No tool eliminates the possibility of loss. Nothing on this page is investment advice, and no performance is implied or promised.