I contributed technical content and opinions to an article titled “Explainable AI: Why Black Box Models Are a Problem” on the Pure AI web site. See https://pureai.com/articles/2026/05/01/explainable-ai-why-black-box-models-are-a-problem.aspx.

Black box models are problematic because they obscure how decisions are made. In business contexts, AI is increasingly used in high-stakes areas such as credit scoring, hiring, pricing, fraud detection, and supply chain optimization.

When a model produces a recommendation or prediction without a clear explanation, managers may struggle to justify decisions to customers, regulators, or internal stakeholders.

When models function as black boxes, managers may either over-trust them, blindly accepting outputs without scrutiny, or under-trust them, ignoring potentially valuable insights. Both outcomes are costly.

I used AI to generate this decorative image but I wasn’t entirely happy with it. Art is not my strong suit, as the saying goes.

Overreliance can lead to serious errors going unnoticed, while underutilization reduces the return on AI investments. Explainable AI helps bridge this gap.

Explainability involves trade-offs. Highly complex models often outperform simpler, more interpretable ones in terms of raw predictive accuracy. However, the marginal gains in accuracy may not justify the increased risk associated with opacity, particularly in high-impact business decisions.

The Pure AI editors asked Dr. James McCaffrey, a founding member of the Microsoft Research Deep Learning group, to comment. McCaffrey observed, “Many AI models are inherently unexplainable. A prediction result, such as the predicted score of a football game, or a generative result, such as the answer to why some racial groups have high crime rates, is a result of mathematical calculations involving billions of values.”

“To make an AI model explainable, there are two main approaches. First, use AI techniques that are relatively simple, such as decision trees. However, simple techniques can only be used in certain, limited scenarios. Second, apply post-hoc techniques that analyze an AI model. For example, counterfactual methods are what-if analyses that examine how much an input to an AI model needs to change to alter the model’s output.”

McCaffrey cautioned, “I’m mildly pessimistic in the sense that because AI is evolving so rapidly, and the financial gains created by non-explainable AI are so enormous, explainable AI will likely be an optional afterthought unless there is some sort of regulatory pressure to require it.”

When I was a young man, I was absolutely fascinated by old electro-mechanical coin-operated games. The inner workings of these beautiful machines was a wonderful mystery.

This is the “1937 World Series” game, manufactured by the Rock-Ola company from 1937 to 1940. It is one of the most highly sought-after games by collectors. Top examples sell for well over $50,000. I played this game many times in the 1960s because it was in the Disneyland Main Street Arcade, within eyesight of my family home — admission to Disneyland was only 60 cents. Yes, 60 cents!