In a nutshell, engineers and researchers:

a.) Have a tendency to continue investing time, money, and effort into a failing venture simply because they have already invested resources in it.

b.) Have a tendency to believe that a complex solution must be better than a simple solution even in the face of contradictory facts.

c.) When compensation and prestige is tied to being an expert, clarity is a liability, and researchers can become obnoxious jerks.

I spent a decade as a university professor (teaching math and computer science), and two decades at Microsoft Research (working on machine learning). Even though my career was in technical fields, my very first BA degree was in Psychology (from UC Irvine).

Recently, while I was working with tree-based regression (predict a single numeric value) systems, I noticed that in literally all of my experiments, AdaBoost.R2 and Gradient Boost regression with super-simple Extra Tree regressors as the weak learners worked better than AdaBoost.R2 and Gradient Boost regression with extremely-complex standard Decision Tree regressors as the weak learners.

However, all library implementations of AdaBoost.R2 and Gradient Boost regression use Decision Tree regressors as the default. Why? Because of the sunk cost fallacy combined with the complexity fallacy.

In research and engineering environments, team prestige, grant funding, patents, and thousands of commit histories are bound up in the systems people build.

The Sunk Cost Fallacy: Admitting that a random split heuristic (with Extra Trees) performs just as well, or better than, a hyper-optimized greedy tree algorithm requires confronting the uncomfortable truth that months of micro-optimizing split-criterion logic, dynamic programming, and search trees weren’t necessary for most real-world tasks.

The Complexity Fallacy: There is a pervasive bias in computer science and research that more complex algorithms must be superior. A mathematically intricate split optimizer (like XGBoost exact greedy algorithm or second-order gradient expansion) feels “smarter” than literally rolling a pair of dice to pick a threshold, even when the rolling-dice technique gives better practical results.

Standard Decision Trees had a 20-year head start in academia and production codebases before Extremely Randomized Trees were introduced in 2006. By then, the entire ecosystem (libraries, papers, benchmarks, hardware optimizations) was already anchored around greedy splits with standard Decision Tree learners.

On top of all this, there is a huge I-Know-More-Than-You-Do ego effect.

Arguably the most pervasive, unwritten political dynamic in technical organizations is complexity as job security and social status.

In research, the phrase “knowledge is power” takes on a very literal, cynical form. When an engineer or researcher’s compensation, title, and influence are tied to being an irreplaceable subject matter expert, clarity is a liability. If a solution is simple, elegant, and easily understood by a junior engineer or a product manager, it strips away the author’s aura of specialized genius.

This leads to preferring overly-complex techniques that can’t be easily understood by non-experts or even other experts. Put a bit more bluntly, although the vast majority of my colleagues were nice and collaborative, there were a lot of complete jerks and jerkettes too.

All of this is a sharp reminder of how technological defaults are shaped as much by human psychology and historical inertia as by pure engineering merit. A simpler method often works just as well or better because nature and real-world data are noisy, and simple random heuristics are surprisingly robust against noise.

The Elizabeth Holmes Effect