Over-Control and Under-Control

When teams can't sit with nebulosity

Creating value with AI is uncomfortable for individuals because it demands greater capacity for nebulosity.

Stochastic systems resist the clean specifications and deterministic fixes that knowledge workers are trained to expect. This discomfort shows up not just at the individual level but in the collective psychology of teams and companies. They tend to react in one of two ways when it hits.

Discomfort at the level of teams

Over-control: Believing you can eliminate all “bad” model behavior

Suppose an embarrassing LLM output gets escalated within a team. Instead of calmly accepting that this is what stochastic systems do and thinking carefully about how the underlying evals and quality control processes should get updated, there’s a frenzied overcorrection. People start saying things like “This must never happen again”. What follows is a predictable rush to inflate the evals with examples that capture the relevant edge cases. Or to add “hacks” in the LLM’s scaffold to attempt to deterministically catch such behavior. None of this is done carefully because it’s done in a high-stress state. But the team manages to put out the fire for now.

The stochastic system keeps producing embarrassing outputs, albeit each time with a different flavor. The evals are patched haphazardly to account for these embarrassing outputs without much concern for the overall actionable questions that the evals were originally designed to answer. The entire system gradually becomes a mess that’s difficult for the team to effectively iterate on. This creates even more stress for the team.

There’s often some trade-off in any stochastic product in terms of explicitly minimizing embarrassment (i.e. failures) and maximizing value (i.e. victories). One extreme way to avoid a specific type of embarrassment is for the system to never try at all. For example, by punting on many user requests with the response “I can’t help you with this request.” Over time, despite what the team says, the evals increasingly skew towards avoiding embarrassment rather than amplifying customer value. It’s difficult in such cases to point the finger at any single individual. The overall system gradually converges to this equilibrium due to the contributions of every single participant.

Despite the entire system tending towards this extremely risk-averse attractor, the team is constantly asked to create more value for its customers. This only adds to the collective stress. So the cycle continues.

Under-control: Offloading value judgments to the model

The increased nebulosity of AI products forces teams to grapple with what “good” means far earlier in the product lifecycle. Even if the product is entirely owned by a given team, defining and defending what “good” means is hard. It requires working with competing values both within the team and between its external collaborators. Reaching strong inter-subjective agreement may only be possible after substantial inter-subjective disagreement. It may require uncomfortable conversations where specific individuals have to take formal accountability for what “good” means. So people avoid doing it.

LLMs are the perfect accelerant for these avoidant tendencies. For example, it’s common for people to set up LLM-as-a-judge systems to evaluate LLM workflows that they care about, without creating corresponding evals for these judges. This looks rigorous because it’s produced by a “machine” or “algorithm”. But it just sweeps the problem under the rug. The judge needs its own evals but no one recognizes this. One could ascribe this to ignorance. But often, it’s because people are reluctant to invite and metabolize disagreement.

The result is that the team is gradually disempowered to truly take ownership for its product’s quality. They incrementally create a system they don’t understand and can’t control. Nevertheless, they’re constantly asked by their superiors to create more value for customers. It becomes harder and harder to slow down and course correct as the team grows disempowered. They’re in a perpetual state of overwhelm. So the cycle repeats itself.

What this looks like at the company level

Over-control: Preserving familiar structures

Change management is complex, difficult and frightening. Many organizations correctly identify that injecting more nebulosity into their system carries scary risks. But fear of these risks can cause executives to unskilfully grip control too tightly. That is, to create increasingly compelling rationalizations for why necessary change is actually unnecessary. Or perhaps, to pay lip service to the necessity of creating change without actually doing much.

There may be substantial excitement about AI in leadership meetings. But leadership keeps funding the same projects as before. Only this time AI is bolted on top. Strategic planning treats AI as a tool to accelerate existing roadmaps rather than something that may fundamentally change what’s possible or what should be built. The entire leadership team is already perpetually swamped. So their revealed preference is that they don’t have any time to look into this new AI thing. But they also don’t want to give off the impression that they don’t know what they’re doing. Or that they’re not taking this change seriously.

When someone proposes a genuine restructuring around AI capabilities, they get labelled as naive or reckless. Safe incremental improvements get praised instead. These rationalizations sound reasonable precisely because they contain a kernel of truth. Consequently, the company moves incrementally faster along the same workflows that it had before. Model improvements fail to unlock novel possibilities for what the company can accomplish in the world.

But the world isn’t static. The landscape starts shifting as the models get better and competitors start changing their operations. At this point, an intervention that seemed too radical doesn’t seem radical enough. But it goes against the deeply held belief to play it safe. This tension creates a lot of stress, and the cycle continues.

Under-control: Performing competence instead of developing it

It’s scary for leaders to show vulnerability. AI is possibly one of the largest shifts since the Industrial Age. So it’s especially scary for leaders to admit they don’t know how AI works or how their business can create value from it. It’s also scary for many teams to feel that their leaders don’t have all the answers.

Nevertheless, everyone can see the shifting patterns of the world. There’s a feeling that something needs to be done. It’s common for such leaders to deputize subordinates that they can’t properly evaluate, precisely because they don’t know much about AI. This also means that they lack the discernment to distinguish genuine competence from charisma and bullshit. This inevitably amplifies the performative dynamics within the company and within their AI projects specifically.

Mundane AI workflows get rebranded as “agents” to match the broader external hype cycle. Sacred cow AI projects get created. They’re quickly branded as “critical”. Evals and performance metrics for these various AI projects eventually morph to manage leadership’s emotions more than tracking value created for customers. Various cultural structures emerge to reinforce these dynamics. For example, vacuous Slack channels are created to “share AI wins” that serve as opportunities for ambitious people to “manage up” rather than truly disseminating valuable tips and tricks within the company.

Moreover, the “critical” AI projects quickly become boondoggles because they lack authentic strategic direction. Their overall dynamic starts to attract ambitious people seeking visibility rather than people best suited to do the work. The whole thing gradually becomes a political shitshow. Inevitably, things often go wrong with such projects. When it does, there’s a profound absence of leadership to course correct.

Feedback from customers doesn’t reach the right levels of leadership in sufficiently undistorted form. The organization can’t learn because various critical corrective feedback loops start to become too broken. The same ignorance that creates all of these performative dynamics also prevents the organization from recognizing and correcting it. The longer this theater carries on, the worse this feedback loop becomes. So the cycle repeats itself.

What I think is going on

AI places greater demands on an individual’s capacity to relate to nebulosity. A similar dynamic also takes place with the collective psychology of teams and companies. Each of the patterns above is what happens when teams and companies lack sufficient capacity:

• Over-control takes place when teams and companies reactively seek to collapse this inherent nebulosity into something that feels more controllable. They attempt to relate to stochastic software with processes and operations as if it were deterministic software. That simply doesn’t work. Each “failure” to hold down control creates more and more stress until something “breaks”.

• Under-control takes place when teams and companies avoid relating to this nebulosity altogether. They attempt to create as much distance as they can between themselves and this new reality. But this doesn’t work either and creates its own distortions. Similarly, each of these “failures” simply raises the temperature of the system creating more stress until something “breaks”.

The transition from deterministic-first to AI-first software is profound. Creating value in tandem with improvements in model performance requires the cultivation of new capacities. These demands to sit with greater levels of nebulosity get expressed at the level of an individual, team and company. It turns out that there’s already some research from Robert Kegan and others that attempts to articulate how such capacities could be developed. The next few posts will explore this body of work to better understand the “map”, before we dive in with “solutions”.

Acknowledgements

Brian Whetten, Prof John Vervaeke, Charlie Awbury, David Chapman for everything they’ve taught me.

Dan Hunt for helpful discussions, editing and generally co-creating these essays with me.