Psychological Development as the Capacity for Nebulosity

What skills and tenure can’t explain

What do we mean by “development”?

Confronting increasing levels of nebulosity is inherently uncomfortable. Value creation with AI demands confronting increased levels of nebulosity. There’s a temptation to fall into two failure-modes. That is, to either over-control one’s relationship to the situation (i.e. to collapse the nebulosity into false certainty), or to under-control (i.e. to avoid the nebulosity as much as possible altogether). This is true both at the level of individuals and organizations.

The capacity to navigate nebulosity isn’t uniformly distributed. Within a given domain, some people are better at facing it than others. It also doesn’t seem to be distributed by IQ. I’ve met really intelligent people that were just awful at dealing with it, and unintelligent people that tolerated it quite skillfully. This is certainly anecdotal. But something interesting is going on here.

The word “development” evokes many meanings to the average tech bro. For example, professional development, software development, skill-building exercises, etc. That’s not what this essay is about. This essay is about development in the “developmental psychology” sense. It’s about the capacity to make meaning when the rules run out, the world becomes fuzzy and one’s confronted with a slate of difficult choices. It’s about one’s capacity to navigate increasingly complex ways of meaning that can coordinate more perspectives, over longer time horizons, with less distortion.

The capacity to navigate nebulosity is developmental. That is, one’s ability to navigate nebulosity without either rigidifying it into false certainty or dissolving it into nihilism, while still producing a workable pattern for action, is a function of one’s psychological development. This essay’s goal is to clearly point out that adult development exists, and to make its connection with the capacity for navigating nebulosity explicit.

This isn’t a personality trait. Neither is it something one can learn the definition of and immediately apply. It’s a way of making meaning that grows through specific kinds of experience and challenge. Some people have consciously or unconsciously cultivated more capacity than others. One’s psychological development shows up most clearly when they’re embedded in a situation that genuinely resists clean answers.

Everyone’s seen real-life examples of what I’m pointing towards. Why is it that some people seem more “mature” when confronted with a difficult situation, and others don’t? That difference isn’t random.

Development in children

I have a nephew that recently turned one. When he was a newborn, he lived in a small “box” (i.e. crib) in his parent’s apartment. Both his perception and actions were incredibly limited. From the perspective of an adult, his field of action was very concrete and patterned, and not very nebulous at all.

The “box” widened as he grew and learned to crawl and eventually walk. His world became more complex, uncertain and nebulous at each developmental transition. Each stage invited him to master new levels of emotional processing, bodily control and sense-making to cope with this nebulosity.

Pre-school will likely offer him rigid rules to help him understand what it means to be a “good boy”. For example, “Hands are for shaking, not for hitting. Legs are for walking, not for kicking”. Despite seeming concrete to an adult, such rule-sets can seem quite complex for the impulsive brain of a child.

The criteria for being a “good boy” grows increasingly complex with age. For example, what if someone tries to mug him when he’s 16? Or when he’s a legal adult? The previous rigidity from pre-school around never hitting or kicking would seem irrelevant. At that point, the nebulosity of being a “good boy” would invite questions of proportionate force, ethics, etc.

It’s culturally obvious to us that psychological development exists in children. There’s entire pop-culture books on it. Developmental psychology’s less well-known claim is that this process continues well into adulthood. The stages of adult development are less visible but no less consequential.

Development in adults

Kegan’s six stages of development

The developmental psychologist Robert Kegan has created a map of the various stages of adult development. Note that adult development is a topic of active research. It’s far from a settled science. Nevertheless, Kegan’s work on stage theory can be extremely useful in making sense of relational and organizational dynamics. It’s best to relate to this work from the spirit of “All models are wrong, but some are useful”. It’s been my experience that an individual’s participation within a stage may be deeply contextual. For example, someone may tend towards Stage 4 professionally and Stage 3 romantically. Or maybe devolve to Stage 3 based on hunger, fatigue, etc.

Stage 0 is the Incorporative Stage. It’s what infants participate in. Their experience is undifferentiated and they don’t have a “self” to speak of. They’re largely embedded in sensations/reflexes and early attachment. Nebulosity isn’t relevant here because there isn’t really a self to speak of.

Stage 1 is the Impulsive Stage. This typically emerges between ages 2-6. The child is embedded in and identifies with impulses and perceptions. That is, it is the hunger, anger, happiness, etc that it’s feeling. The child lacks the ability to see past their immediate experience, and rules act as an external organizing force to be obeyed or evaded. Nebulosity isn’t relevant here because behavior is largely impulsive.

Stage 2 is the Imperial/Instrumental Stage. This stage often develops between ages 6-15. The child can sufficiently step back from their experience and impulses to coordinate their behavior towards specific goals. The child’s sense of self is organized around needs, interests and concrete outcomes. Therefore, relationships are transactional and other people matter mostly in terms of what they can provide. Nebulosity in this stage is threatening because it challenges the concrete pathways towards win/lose that organize the child’s world. Ambiguous situations lacking an obvious “right answer” that maximizes personal gain feels confusing and illegible.

Stage 3 is the Socialized Mind. This stage often develops post-adolescence. The individual’s sense of self at this stage is deeply shaped by relationships, roles and perceived expectations by their broader social milieu. Individuals at this stage can’t step outside the expectations of their social environment to evaluate them. Nebulosity is tolerable when social consensus exists. However, it rapidly becomes intolerable when consensus of one’s expectations break down, roles become unclear or one’s immediate relationships provide contradictory guidance. This is especially acute when the stakes are high.

Stage 4 is the Self-Authoring Mind. Unlike the Socialized Mind where individuals identify with their social relationships, these individuals identify with their own self-authored value systems. They can distinguish between “what I think” and “what others want me to say, think or do”. This allows them to effectively navigate social pressure by setting positive boundaries. They have greater capacity for navigating nebulosity since their self-authored values provide stable patterns for interpretation. However, this tolerance has limits. Nebulous contexts which genuinely threaten their internal framework, or contexts that are beyond their framework’s ability to solve problems get perceived as personal threats.

Not all adults reach Stage 4. Kegan’s book In Over Our Heads suggests that roughly ~55–60% of adults primarily operate from the socialized order (i.e., not yet self-authoring), about ~1/3 are primarily self-authoring, and self-transforming is rare.

Stage 5 is the Self-Transforming Mind. Unlike the Self-Authoring Mind, the individual can step back from their own value system to see it as one system among many, rather than an absolute truth. They can simultaneously hold multiple systems and perspectives without needing to resolve the contradictions between them. Their own identity and beliefs become objects of reflection and revision rather than objects beyond question. David Chapman describes this as recognizing that systems are “nebulous yet patterned” simultaneously. Individuals at this stage don’t feel the need to reactively collapse nebulosity into premature certainty (i.e. over-control), nor do they feel the need to avoid it altogether via nihilism (i.e. under-control). They can make personal commitments and take action while acknowledging the inherent limitations of any single interpretation.

Realistic example at Google

The description of Kegan’s five stages of development was fairly abstract. Let’s explore how different levels of development may show up in an incident post-mortem at a large tech company like Google.

Premise

• Alice and Bob are both Senior SWEs (L5) on the same infrastructure team managing a caching service.

• Last week, Bob pushed a config change that modified a service’s cache eviction policy and TTL settings.

• This was reviewed and approved by Alice.

• Three days later the service went down due to an unexpected traffic spike from a new client team. Their requests caused the cache to evict too aggressively and it flooded the backend database.

• This caused a 45-minute outage which affected multiple downstream production services and impacted revenue.

• Bob’s change was rolled back and a post-mortem was triggered to discuss the root cause.

Why is this situation nebulous?

Large software projects can be incredibly complex, and the question “What was the root cause?” resists a clean specification. For example, the following competing interpretations about making sense of the incident may emerge during the post-mortem meeting:

• “Bob was careless with the config change.”

• “Alice failed to catch the risky change.”

• “The client team should have announced their load increase.”

• “Capacity planning should have caught this.”

• “Downstream systems should be more robust to such outages.”

• “Our monitoring thresholds are too loose, and should have alerted far sooner.”

Unpacking the patterns of interaction between Alice and Bob

For the sake of demonstration, let’s assume that Bob and the rest of the team are operating at Stage 3 (i.e. Socialized Mind). This follows from Kegan’s research that most adults fail to reach Stage 4. We’ll then vary Alice’s participation between Stages 3 to 5 to see if one person’s developmental stage can shift the dynamics of the entire room, despite Bob being at Stage 3.

Since Bob’s at Socialized Mind, he relies on his social interactions with the group to resolve the nebulosity of “What was the root cause?”. He can’t easily separate “my change contributed to the outage” from “I am a bad engineer and people will see me that way.” Throughout the post-mortem process, he’s very attentive to other’s reactions to interpret how bad this may have been for him. His default approach would be to deflect negative judgement from the group by pointing out that the config change was reviewed and approved. Moreover, perhaps his default would be to over-control by imposing a clear narrative (i.e “not my fault”) onto this nebulosity to protect his social standing, and therefore his sense of self.

Alice at Stage 3

Let’s assume that Alice is also at Stage 3. This implies that like Bob, her sense of self and what she finds valuable is shaped by relationships and perceived expectations. Like Bob, she needs consensus from the group in resolving the nebulosity so that she knows what to think. Therefore, she also hedges heavily, gauge’s the room’s reaction before committing to an interpretation. She feels caught between the tension of maintaining loyalty to Bob (i.e. not throwing him under the bus) versus her loyalty to her overall team (i.e. being honest with what Bob did wrong). Both feel like bad choices without any principled way to choose without group consensus.

Since everyone’s operating at Stage 3, no one’s capable of handling genuine complexity. Bob deflects to protect his reputation and Alice waits for group consensus to organize her actions. Since the group consensus never arrives, the team over-controls to the first narrative that feels socially safe. That is, “Bob will fix it”. Perhaps in the future, a similar incident occurs with a different config change with an eerily similar post-mortem. Bob’s more cautious now and less willing to push for improvements to the overall system. Alice develops resentment that “we never actually fixed the real problem”, and the team habituates an implicit norm of avoiding conflict in ambiguous situations.

Alice at Stage 4

As a Self-Authoring Mind, Alice is guided by an internal system of values. She can clearly distinguish between “what I think” and “what others want me to say”. She can tolerate more nebulosity than Stage 3 because this same internal system stabilizes a pattern of interpretation. But she can’t see beyond her internal system. So her temptation is to confuse seeing a pattern for seeing the pattern, which is exactly what happens. On the other hand, this presents her with a level of clarity that the others in Stage 3 lack.

During the post-mortem she states her analysis directly without worrying about the room’s dynamics. She focuses on systemic improvements rather than pointing the finger at anyone. She doesn’t take Bob’s defensive behavior personally, nor does she feel responsible for managing his emotions. She’s able to coherently defend her case even if the group challenges her interpretation. Her primary tension is between staying true to her internal framework and being aware of the relational cost that acting on “her truth” may entail.

Alice’s clarity allows the team to form a coherent and effective diagnosis, prescription and treatment for the overall system. Perhaps in the future, the system overall is far more robust than it otherwise would have been. But the system remains vulnerable to blind spots in Alice’s narrative. Bob ultimately feels steamrolled and doesn’t push back in meetings anymore even if there’s legitimate concerns.

Alice at Stage 5

As a Self-Transforming Mind, her identity isn’t attached to any specific value system. She can skillfully hold multiple systems/perspectives at once, and doesn’t treat her framework as absolute truth. She’s best able to resist the temptation to over-control and under-control in relation to the nebulosity of the situation.

During the post-mortem she’s genuinely curious about all the perspectives including Bob’s. She asks probing questions to better understand what each perspective was optimizing for as the incident progressed. She’s able to see the incident via multiple frames simultaneously, probes for feedback and is able to see her own view as one among many. For example, “This seems like a testing gap, a monitoring gap and a capacity issue. From where I sit, insufficient testing stands out. But I’ve been burned by that before. What’re all of you seeing?”. The room feels different with her in it. Bob’s unlikely to get defensive because the room’s consensus implicitly shifts from “who is to blame?” to “let’s understand what happened?”. At Stage 3, this allows him to be more honest than he’d otherwise been without feeling threatened.

Alice’s ability to inhabit all of these multiple perspectives without getting too entangled with any of them allows the room to gain more clarity on the situation then they’d otherwise have gotten. She gracefully handles the tension between the necessity of making decisions versus the impossibility of certainty. She can make recommendations for next steps while acknowledging their provisionality.

Perhaps in the future, the team develops a norm of genuine inquiry in post-mortems. Creating safety for Bob allows him to surface warnings earlier next time. This culture of genuine inquiry rapidly improves the process in multiple directions after each incident.

Closing Thoughts

Fault attribution within complex systems is extremely challenging. The inherent stochasticity of AI systems confronts teams with such nebulosity far more often within the process of routine software development. This essay’s scope was to describe the phenomenon of psychological development within adults, and concretely describe why it’s relevant within software engineering. Future essays will ground these ideas more concretely within AI development.

It’s also worth pointing out that adult development isn’t purely individual. Organizations create environments that shape the developmental trajectories of everyone inside them. They establish norms, expectations, and patterns of interaction that either support or hinder people’s capacity to navigate nebulosity. Conversely, the behavior of people within organizational containers shapes the overall norms within the organization.

Some organizations habituate wisdom within their participants. Others engender increased foolishness. For example, how a company/org/team runs their post-mortems gradually cultivates or erodes the developmental capacity of everyone in the room.

The next essay explores what it means for an organization itself to orient itself towards development.

Acknowledgements

Brian Whetten, Prof John Vervaeke, Charlie Awbury, David Chapman for everything they’ve taught me.

Dan Hunt for helpful discussions, editing and generally co-creating these essays with me.