The literature on interpretability in machine learning has a by now classic starting point: there is no single definition of interpretability, and its necessity arises when the problem the system solves is incompletely specified with respect to what actually matters (Doshi-Velez and Kim, 2017). In the applied contexts we deal with, an accounting entry, a transport document, a check against public data, this incompleteness is the norm: the system answers, but responsibility for the result stays with a person. It follows that explainability and control are not accessory documentation of the product: they are properties of the product, and they must be engineered like the others, with requirements, with measures and with the cost they entail.
Evidence and inference must be kept distinct
The first form of explainability is concrete and has nothing theoretical about it: visible sources, reconstructible steps and a sharp distinction between what the system observed and what it inferred. A figure read from a document and a figure estimated by a model must look different to the user, because they have a different epistemic status: the first is verified by tracing to the source, the second is judged against an error margin. Blurring them in the interface means transferring to the user an ambiguity the system knew about and kept silent.
Research on explanation has also shown that an effective explanation is selective and contrastive, that is, it answers the question «why this and not that» with a few pertinent elements, not with an exhaustive list of factors (Miller, 2019). For the designer the consequence is precise: the useful explanation is the one that lets the user reach the underlying figure in a few steps, not the one that describes the model's internals.
Uncertainty is designed, and calibrated
A mature system does not hide what it does not know. Confidence thresholds, uncertainty indicators and human-review paths belong to the interface, not to the manual. There is, however, a preliminary requirement that practice often neglects: for a declared confidence to mean anything, it must be calibrated, that is, empirically correspond to the frequency with which the system is right. Modern neural networks are documented to be poorly calibrated, systematically more confident than their results justify (Guo et al., 2017); exposing an uncalibrated confidence to the user is worse than exposing none, because it dresses an impression up as a measurement.
Designing uncertainty also means deciding in advance what happens below a given threshold: the system stops, asks, flags, routes to a person. These are product decisions, and they must be taken before the model exists, because they determine the requirements the model has to satisfy and not the other way round.
Control is an architecture
Model versioning, logging of inputs and outputs, monitoring of drift over time, repeatable evaluations on stable test sets: these are the conditions for reproducing a past result and for understanding why today's result differs from six months ago. Without this instrumentation a machine-learning system is not governable: it works until it stops working and, when it does, nobody can say since when or why. Control instrumentation, it must be said, costs: it lengthens development and does not show in demonstrations. It is the typical investment whose usefulness appears only at the first incident avoided or reconstructed.
Human oversight is a requirement, a legal one too
The principle «the system proposes, a person decides, the decision remains inspectable» is not merely prudential: for systems classified as high-risk it is a legal obligation. The European regulation on artificial intelligence (Regulation (EU) 2024/1689) devotes Article 14 to human oversight, requiring systems to be designed so that they can be effectively overseen, including the ability to interpret their output, not to over-rely on it and to interrupt their operation. The legal wording confirms an architectural point: effective oversight presupposes verifiability. You cannot oversee what you cannot inspect.
What this entails for the designer
More instrumentation work and less stage effect. A verifiable system shows its own limits and, in a commercial demonstration, a declared limit impresses less than a confident answer. It is, however, the only way to bring machine learning into processes where someone answers for the result, and it is the criterion by which Analytiko™ products are designed and evaluated: every output distinguishes the observed from the inferred, declares its margin and preserves the chain that ties it to the evidence.
References: Doshi-Velez F., Kim B., «Towards a Rigorous Science of Interpretable Machine Learning», arXiv:1702.08608, 2017 · Miller T., «Explanation in Artificial Intelligence: Insights from the Social Sciences», Artificial Intelligence, 267, 2019 · Guo C. et al., «On Calibration of Modern Neural Networks», ICML 2017 · Regulation (EU) 2024/1689, Article 14.
Analytiko · 2 September 2026