In production environments, the hardest problems are rarely the models.
The real challenge is building systems that can support real decisions over time.
Over the past decade, I've worked on machine learning and decision systems in large organizations across retail, consumer, and financial services. The pattern has been consistent. The model is rarely the thing that breaks first.
What breaks first is the system.
The data does not look the way it did in the notebook. The workflow does not match the assumptions in the code. The people using the output are solving a slightly different problem than the one you trained for. And the moment the model becomes part of a real decision, the constraints multiply.
Latency matters. Ownership matters. Trust matters. Incentives matter.
None of those show up in the loss function.
Models are local. Systems are global.
A model solves a local problem.
Given this input, predict this output.
A system has to solve a global one.
Where does the input come from? Who owns the data? What happens when the input is missing? Who decides whether to use the output? What happens when the output is wrong? What happens next?
You can have a very good model inside a bad system and still get bad decisions.
In large organizations, most failures are not technical failures. They are coordination failures, workflow failures, or incentive failures.
The model works. The system doesn't.
Production is where assumptions go to die
Demos are clean.
You control the data. You control the environment. You control the timing.
Production removes all three.
Data arrives late, incomplete, or in the wrong format. Users do things you didn't expect. Other systems change without telling you. The definition of the problem shifts while the code stays the same.
This is why something that looked obvious in a prototype becomes fragile in the real world.
The model didn't get worse. The world got involved.
After enough time in production environments, you start to notice that the real engineering work is not in the prediction. It is in everything around it: the pipelines, the interfaces, the decisions about when to trust the output and when not to.
The model worked. The system around it didn't hold.
Decision systems are harder than prediction systems
The difficulty increases when the model starts influencing real decisions.
Forecasting demand is one problem. Deciding what to order, where to allocate inventory, and how to react when reality changes is another.
Each step depends on the previous one. And each is constrained by something the model does not see: lead times, space limits, vendor agreements, or simply the reality of how the business operates.
You can have a very accurate forecast and still end up with empty shelves, excess inventory, or decisions nobody trusts.
Prediction systems answer a question. Decision systems have to live inside the business.
And that makes them much harder to build.
Why this gap is becoming more visible
Modern AI tools make it easier than ever to build something impressive quickly.
You can get a working model in hours. You can build a demo in a day.
What hasn't changed is the difficulty of making that system reliable enough to support real decisions. If anything, the gap is getting larger. The easier it is to build the model, the more obvious it becomes that the real work is somewhere else.
In workflow design. In data ownership. In incentives. In how the system fits into the way people actually operate.
That layer is still underbuilt.
Why I keep coming back to systems
The problems that held my attention were always the ones closest to real decisions.
Not the model that predicted demand, but the system that determined what to order. Not the algorithm that scored customers, but the workflow that decided what to do next.
That is part of what eventually led me to start Toutami, not because the models weren't good enough, but because the layer between the model and the decision was missing.
Good models matter.
But good systems matter more.
And they are much harder to get right.