Is an artificial intelligence model that is right 95% of the time but cannot explain why fit to run a company? My master's research focused precisely on that frontier: explainable AI. The short conclusion: accuracy without explanation is not enough to make decisions.
The context
I compared five models for classifying customer reviews in Spanish: from classical statistical methods to Transformers — the same family of architectures behind today's large language models. The Transformers trained specifically on Spanish won clearly: around 95% balanced accuracy versus 85% for the classical methods, a statistically significant difference across all tests.
So far, the story everyone knows: bigger models are more accurate. The interesting part begins afterwards.
Holding the model to account
A classifier that says "this customer is dissatisfied" is useful. One that says "they are dissatisfied because of the delivery delay" is actionable. To get there, I applied two explainability techniques (SHAP and LIME) that identify which words support each prediction, and I put them to the test: if you delete from the text the words the model flags as decisive, does its prediction change?
With SHAP and LIME, it does — strongly. The explanations pointed to terms that genuinely supported the model's decision: evidence of faithfulness, not ornament.
The surprise came with the third technique: the model's own attention weights, which many tools display as if they were "what the model looks at". Subjected to the same deletion test, they turned out to be indistinguishable from chance. In other words: the flashiest and cheapest explanation to produce was the one that did not withstand scrutiny — a result consistent with what the academic literature has been warning about.
Why this matters outside academia
1. Demand traceability, not narrative. More and more "AI-powered" tools dress up their results with explanations. The right question is not whether the explanation sounds convincing, but whether it withstands a test: does the data point being flagged actually support the conclusion?
2. The same principle applies to your BI. A dashboard is also a model of your company. If a number cannot be traced back to its source —which orders, which dates, which criterion— it is not information: it is decoration. In our projects, every indicator has to be able to answer "where do you come from?".
3. Transparency is a competitive advantage. European regulatory frameworks are moving toward requiring explainability in automated systems. Companies that already work with traceable data will have nothing to rebuild.
The practical rule
Before trusting any system that gives you answers —an AI model or a simple monthly report— ask it the deletion question: if I remove this thing you say is important, does your conclusion change? If the answer is no, you have decoration. If it is yes, you have a tool for making decisions.
Ricardo Borja is the founder of Inteleqta and a consultant in ERP and Business Intelligence. This post summarizes results from his master's research in Quantitative Methods (UBA) on classifying Spanish-language reviews with Transformers and explainable AI.