Can my career in AI prove I'm a good person?
Proving Worth versus Providing Value
Or more broadly, is it possible to make lots of money while using my career to serve a meaningful purpose?
I’ve wrestled with this question for years. I’ve spent a decade doing deep learning at Google, worked on LLMs before they were cool, and was a core member of Gemini’s original modeling team. It led to me leaving Google. This essay is my tentative answer.
I’d reached a point where traditional notions of “success” didn’t animate me, but none of the purpose-driven possibilities felt viable. I felt trapped between sacrificing my alignment with purpose, or sacrificing my own happiness. It turns out that this dilemma is what my executive coach Brian Whetten calls The Conscious Business Trap.
I’ve reached a better place now. As Brian taught me, my previous default was to insecurely prove my worth either via material or moral achievements. I now have a practice of attempting to more securely provide specific value that’s intrinsically aligned with me. I was able to do this by practicing what Brian calls the art of Enrollment. Specifically, enrolling people into Win/Win or No Deal relationships with me.
This essay is a case study through the eyes of a fictional character named Rohan. He’s a composite of myself and many well-meaning tech bros that I’ve met at various frontier labs.
Rohan’s journey to a frontier lab
Rohan was a precocious boy that asked lots of questions. He loved it, never grew tired of it, and got good at it. This landed him in MIT, where he studied electrical engineering and computer science.
He had a gift for breaking down complex systems, and seeing solutions where others saw walls. He tried doing some internships at stereotypical Silicon Valley companies that were hot at the time. But it just wasn’t his vibe. He kept missing his crew back at MIT. Unlike the Silicon Valley companies, he felt more seen and accepted by them. He felt far more seen for his intellect back there, than in any typical Bay Area company. He ended up doing a PhD with his favorite MIT professor, and it was the happiest he’d ever been.
Towards the end of his PhD he realized that he liked the people he met in academia, but it was just too slow for him. Many of his smartest friends had started getting jobs doing deep learning at Hooli Research. He was intrigued and got a job there.
Becoming a Research Scientist at Hooli Research ended up being a dream come true. He got to work on everything from speech recognition to medical imaging. He had the resources to make some real breakthroughs. His group was able to run experiments he could only dream of even at a big university like MIT. It felt surreal to get paid so well for a job that was a perfect alignment of intellectual challenge and meaningful impact.
Rohan climbed Hooli’s ranks to become a director within a decade. Scaling up deep learning models was expensive. Access to the most interesting work was becoming political, and Rohan was determined to avoid getting locked out. He’d never been interested in status per se. But he absolutely didn’t want to get locked out of the coolest projects!
Before long, he’d become thoroughly financially independent, had a great boss, and had inspiring reports. Life was good.
Rohan’s moral dissonance
Everything changed when OpenAI launched ChatGPT.
Hooli frantically created the Apollo team, an org dedicated to training LLMs. Rohan’s expertise put him smack dab in the middle of it within Apollo’s pre-training team. The world was still reeling from the implications of GPT-4, and Rohan was no different. However, unlike the public, he could confidently extrapolate where the models would go in a few short years. He felt a deep tension in his belly he couldn’t shake. His once-joyful disposition started becoming dour.
His friends grew concerned when he suddenly announced that he was leaving the pre-training team for the less prestigious post-training team. He lost his reports in the process and became an IC, but he didn’t care. Eventually, staying in post-training became untenable too, so he left Apollo altogether. He couldn’t bear to participate in Apollo itself, but also didn’t want to cut himself off from LLMs. So he started making prototypes with Apollo’s models that leadership found interesting. He was tenured enough that no one really paid attention to him. His leads figured he’d be better off sitting on Hooli’s rooftop doing nothing, than joining OpenAI in strangling Hooli.
His mind kept flashing dark visions of the future. Leaving Apollo gave him time to reflect on how LLMs were likely to impact society. He started reading everything he could on AI Safety, got into TPOT, joined jhana retreats and talked to anyone that seemed to care.
He started getting job offers before long. Neolabs, think tanks and academic labs all wanted a piece of him, and each was more purpose-aligned than the last. Of course, Apollo wanted him back too, but he couldn’t stomach that. Leaving Hooli was tempting, but it was home. It was the only institution that he’d known other than MIT. It provided for his family and gave him extraordinary freedom.
An offer Rohan couldn’t refuse
One day his friend Mario emailed him asking to catch up. They’d worked together at Hooli before the LLM era. Mario had just spent the last few years doing foundational work on LLMs at OpenAI. His email said that he and some co-founders had just broken off from OpenAI to start a new lab called Noetic. They felt that OpenAI was insufficiently committed to building safe and aligned AGI, and was too apathetic about the societal impact from AI. He asked whether Rohan wanted to chat. This seemed like a sign from the universe. Mario was an undisputed genius, and Rohan scheduled a time immediately.
Their thirty minute slot ended up becoming a three hour soul-bond. Their conversation covered everything from the path to AGI, to what a post-AGI future might look like, to the sorry state of Silicon Valley culture, to the best coffee in East Bay. At the end of the call, Mario asked Rohan if he’d become Noetic’s CTO.
Rohan was stunned speechless. Rohan deeply respected Mario, and the mission resonated with him. But Noetic was tiny compared to OpenAI! Let alone Microsoft, Google and the rest of FAANG! Nevermind that becoming CTO would destroy his work/life balance, substantially reduce his immediate comp, and put him back into the hot seat of capabilities research. So he said no.
Noetic ended up growing rapidly quarter after quarter. Mario kept periodically reaching out, and Rohan kept finding new reasons to say no. Each “no” left Rohan increasingly out of sorts. Mario kept increasing the size of each compensation package with each offer. And many of Rohan’s friends had already joined. Some of them had been there long enough that they were already buying mansions in Atherton.
Meanwhile, Rohan grew increasingly listless at Hooli. It’d been more than a year since he’d worked on any project that seemed to actually matter. People would still come to him for his sage advice, and he was more than happy to oblige. This usually involved him poking holes at their ideas, but he was often too disconnected from the projects to offer any real solutions. Although they appreciated the constructive criticism, he started feeling increasingly hollow. It bothered him that he couldn’t find a project to actually get energized with. It bothered him to not have anything meaty to go deep on.
One day, Mario emailed Rohan his manifesto for the future. It was a dense tome of detailed predictions on model performance, market dynamics and geopolitics. It laid out Noetic’s core dilemma, which was that scaling would continue whether Mario wanted it to or not. A conscientious lab like Noetic that fell far behind the frontier would have no influence with world governments. On the other hand, accelerating the frontier seemed bad for safety. Therefore, Noetic would aspire to remain at the frontier while maintaining integrity with its vision for an aligned AGI. Specifically, by turning their proprietary frontier safety research into a competitive advantage.
Mario wanted Rohan’s feedback from the vantage point of a CTO. Needless to say, this immediately nerd-sniped Rohan into finding the manifesto’s many contradictions. His mind started racing, and the two started spending hours exploring the various pitfalls of the manifesto. It was the most alive and purposeful Rohan had felt in years.
At some point, they started talking about how Noetic would resolve specific thresholds in model capabilities. Rohan was particularly worried about an LLM’s capacity to find security vulnerabilities. What would Noetic do if a model accidentally broke out of a sandbox to hack real properties on the public internet? Pausing all deployment indefinitely until they were sure the model was safe would destroy Noetic’s competitive position. It also wouldn’t stop the other labs from continuing to deploy similar models. Mario couldn’t commit to something so extreme, and the tension in Rohan’s stomach started escalating with force. The reality of actually taking responsibility as Noetic’s CTO started crashing down on him. He told Mario to give him a day to think about the offer.
Rohan woke up with a rush of adrenaline. He’d decided that enough was enough. Mario was the most principled CEO he’d ever met. Joining was obviously the right thing to do both for himself and the world. He needed to stop being such a coward! Yet he started spiraling the moment he logged into Gmail to reply to Mario.
He realized he was stuck.
How was Rohan stuck?
Rohan wanted to create exceptional value in the world with his gifts in developing AI. As clichéd as it sounded, he wanted to make the world a better place. But he wanted to do so with integrity.
He wasn’t actually stuck in any objective sense. For example, he was financially independent and could easily get a job wherever he wanted. But when push came to shove, the moment he saw Mario’s email his mind would only present two choices. He could either go All In on Noetic, or Stay Put at Hooli.
Going All In on Noetic was scary because Rohan felt that he’d become complicit for everything Noetic did. He’d be back in the hot seat working on potentially dangerous capabilities, surrounded by institutional pressures outside his control. Not to mention the cut in pay, increase in workload and logistical consequences working that hard on his family.
Mario was indeed a genius, but Rohan was sometimes scared by Mario’s ungrounded thinking. Rohan had built his entire career by being the clear thinker in the room. Joining Noetic felt too much like he was bullshitting himself.
Rohan was scared that his soul would die if he Stayed Put at Hooli indefinitely. He deeply valued making meaningful contributions to the world, and of looking after his family. But his current starvation of purpose was turning him into a soulless husk before his very eyes.
Again, he wasn’t objectively stuck! But every other option found itself getting sorted into this binarized choice. Joining some random small company didn’t make sense financially. Taking a sabbatical felt like running away from his problems. Starting his own lab felt comically grandiose, and joining another frontier lab seemed like it’d recreate the problem in different scenery.
Why was Rohan stuck?
Rohan needed both money and purpose to function properly. The former gave him financial resources, stability and met his needs of feeling special. The latter helped him feel connected to something bigger than himself.
He’d been sort of lucky that for most of his career, Hooli had unconsciously provided an integration of both for him.
He’d spent his entire life internalizing the idea that he needed to prove his worth from intellectual achievement. He’d then learned to use money as a scorecard for his achievement, and therefore his worth, even if he didn’t care about money in and of itself. But this overall process for making meaning stopped working once he became financially independent. Making an incremental $$$ just didn’t seem interesting anymore.
LLMs further scrambled this overall equation. A seasoned systems thinker like him couldn’t unsee the moral weight of making technical contributions to Apollo or any other frontier lab. So he started trying to prove his worth via the purity of his morality. He wasn’t wrong that there were material unresolved questions about the morality of unabated model scaling or Noetic’s governance. But he was wrong about what he was actually responsible for, irrespective of whatever his ego told him.
Going All In appealed to his temptation to substitute material achievement with moral achievement. The cause of “AI Safety” was large and nebulous enough that his ego felt exceptional when he aligned himself with it. Going all in allowed him to demonstrate to himself and the world that he was indeed special, and that he’d used his gifts well. However, the broader moral consequences upon the world of going all in on Noetic were beyond his individual ability to control. So going all in started becoming a litigation of whether he could model Noetic with certainty, which became inextricably linked to his own sense of moral self-worth.
Staying Put had essentially the opposite effect. Staying inert at Hooli absolved him of moral strife, but it also deprived him of the satisfaction of further material achievement. Of course, it’d protect the money he was already making. But he knew he had abilities few people on the planet had. So he experienced substantial guilt for staying idle.
No matter what he did, he couldn’t find a way to integrate his desire for both money and achievement, with a sense of purpose. He could either sacrifice his material and moral security to feel purposeful, or he could sacrifice his alignment with purpose to feel a sense of material and moral safety. The resulting tension kept wreaking stressful havoc upon his body. My executive coach Brian Whetten calls this The Conscious Business Trap.
Why couldn’t he simply “think” his way out?
John Vervaeke’s work on Relevance Realization points out that reality is combinatorially explosive in its intelligibility, and that we unconsciously frame reality based on what’s relevant before we can even consciously form propositions about it. Rohan’s unconscious stance of attempting to insecurely prove his worth created fundamental constraints on how he was able to parse his reality.
Within that overall stance of proving worth, Rohan attempted to navigate his core decision by framing it as a moral problem. My executive coach Brian Whetten defines moral problems as those that can be solved by successfully assigning the roles of rescuer, victim and perpetrator to all the agents within the frame. It’s true that an intelligence explosion from AI will likely produce legitimate victims, but it’s also plausible for it to produce rescuers and perpetrators. Whether Rohan appeared to himself as a rescuer, victim and perpetrator by working for Noetic depending on which N’th order consequences he foregrounded. Every possible assignment of moral roles immediately presented some counter-assignment to him. This instability challenged his sense of self-worth, and caused him to either spiral out or avoid the decision altogether.
He then attempted to resolve the decision by framing it as a technical problem. Brian Whetten defines technical problems as those that have a stable objective. And whose solutions can be produced once the appropriate facts, skills and perspectives are brought to bear upon the problem. This led him to use the AI Safety literature and other works to analyze each company’s governance, market dynamics and overall likelihood of success. A technical framing can shed light on whether a particular solution is good, but it can’t by itself inform whether the framing itself is problematic or insufficient. That is, a technical problem presupposes what its objective should be. It can’t by itself suggest what it ought to be.
Rohan’s technical frame unconsciously framed the goal as proving his worth. But the outcomes through which he was seeking to prove his worth were contingent on many factors well outside his control. There was no stable analysis of facts, skills or perspectives that could give him the certainty that he was seeking. So attempting to solve his decision via technical analysis continued not working. This is also why whenever coworkers asked him for advice, he was often able to tear down their solutions at a technical level. But he didn’t have the skills to consciously change the way they constructed their framing of the problem at any deep level.
His faculty with systems thinking exacerbated the confusion he experienced when attempting to frame his decision as a technical problem. It made salient many, many complex interactions that he had at best partial observability over and absolutely no control over.
Rohan’s subjective experience was then akin to hitting his head on the wall of going All In or Staying Put again and again. This gradually produced what John Vervaeke calls reciprocal narrowing. That is, the systematic diminishing of his agency, and the gradual increase of his stress. He kept trying to solve his dilemma via different technical and moral framings, and it kept not working.
Rohan was stuck because he couldn’t consciously recognize that he’d hit a developmental problem. He kept hitting the impasse again and again because his proving worth kept telling him that he needed a better moral or technical framing.
I define a developmental problem as an impasse that can’t be resolved because of how the person fundamentally makes meaning, and what they find relevant.
Overcoming developmental problems requires habituating new patterns of thought, speech and action to fundamentally shift what the individual finds relevant. These shifts allow an individual to cope with greater levels of possibility, complexity and framings. Greater levels of development make an individual’s relevance realization machinery more visible to itself. Or as Robert Kegan says, it gradually distinguishes what a person is subject to and what they take as object.
We’ve already established that Rohan was not in fact objectively stuck. However, he eventually believed he was stuck because the broader space of actions simply weren’t existentially viable to him. Notice that he didn’t even consciously realize he was stuck within a specific developmental dilemma until his phenomenological experience gave him no other choice.
Moral, technical and developmental problems are recursively related to each other. Moral framings make certain objectives and constraints salient. Technical framings shed light on how one could best achieve them. Subsequent action can then produce feedback which may either update the underlying moral framing, or produce novel technical framings. The problem then becomes developmental when the person gets stuck in some dilemma.
An ensuing developmental shift then changes what the person finds relevant, which opens new vistas of moral and technical framings. The cycle then continues.
In the same vein, it’s also worth pointing out that Rohan’s developmental problems don’t preclude legitimate moral and technical problems when faced with AI. Rohan still needed to contend with very real moral and technical problems. Its just that the ones Rohan was confronted with had an unhelpful framing and left him stuck.
My executive coach Brian Whetten has devised five stages of psychological development based on twenty five years of working with clients. Their formulation has been inspired by work from Jean Piaget, Robert Kegan and Ken Wilber.
As John Vervaeke has described elsewhere, every individual is confronted with a combinatorial explosion of what could be made salient. He’s also argued that whatever is illuminated by the light of consciousness is a function of what the individual values at some deep level. Therefore, it’s part of the human condition to be confronted with a multitude of competing values and interpretations of those values. One could conceptualize the stages of psychological development as qualitative shifts in the individual’s default approach for resolving these competing values.
Taking is the first stage of Brian Whetten’s map. It organizes conflict around the individual’s immediate needs and base drives. The individual essentially becomes the value that’s most salient to them (e.g. hunger, frustration, joy, etc.). It produces Lose/Lose relationships and is developmentally appropriate for small children.
Pleasing captures a person’s ability to participate in a socialized world of shared norms. This stage is analogous to Robert Kegan’s Socialized Mind. Competing values are collapsed and resolved by assigning moral roles (i.e. rescuer, perpetrator, victim). For example, if someone disagrees with you, rather than seeing them as having a different set of values than you, they get assigned with one of these roles. Pleasing generally produces Lose/Win relationships, because the individual suppresses their own needs to feel loyal or worthy of belonging.
Achieving captures a person’s ability to author their own hierarchy of values, which may be distinct from their surrounding socializing forces. This stage is analogous to Robert Kegan’s Self-Authored Mind. Competing values are resolved by separating fact from emotion, and resolving trade-offs between values by framing technical problems. It often produces Win/Lose relationships, since the individual is willing to stand for what they want even if the other side doesn’t achieve their desired outcome.
Giving sophisticates a person’s desire to achieve by making them sensitive to the needs of others, without getting socialized by them. It’s afforded by systems analysis that can see the interpenetrating relationships between competing values. This greater level of cognitive empathy predisposes individuals at this stage towards Win/Win relationships. However, the challenge of seeing complex interpenetrating relationships within a system makes it harder for individuals at this stage to establish clear boundaries between categories and people. It therefore predisposes their Win/Win relationships to be inherently unstable because they’re not able to effectively advocate for their own boundaries.
Receiving captures a person’s ability to consciously see the competing values acting upon them as objects in their consciousness, rather than unconsciously being subject to them. This allows them to notice how their framing and what they find relevant keeps producing the same constricted choices. They’re able to consciously engage in psychological development to shift what they find relevant, so as to find better technical and moral framings for their problems. This allows them to simultaneously optimize multiple competing values within a given frame. For example, to consciously find an integration for money and meaning, even if it’s not immediately provided by their environment.
Receiving is analogous to what Robert Kegan calls the Self-Transforming Mind. Individuals at this stage are able to participate in Win/Win or No Deal relationships which are more stable because of their ability to enforce boundaries via No Deal. Receiving is named thus because the individual is better able to receive nourishment of their own needs, while serving what’s best for the broader system.
Each stage of development is best thought of as a discretization or categorization of some underlying continuum of navigating complexity. A stage is akin to the individual’s default disposition, as opposed to a state that can fluctuate based on stress, external circumstances, etc.
Rohan’s default developmental home was increasingly inside Giving. He genuinely wanted a Win/Win relationship with the world, but was still driven by a need to prove his own worth via his moral purity. This made it difficult to establish clearer boundaries with the world, along with circumscribed responsibilities that he could hold with integrity.
From proving worth to providing value
He was stuck because he kept consciously or unconsciously asking himself what joining or refusing Noetic would prove about him, especially his moral purity. He needed to shift the operative question to something like what specific value he wanted to create in the world via Noetic, that he’d be willing to take formal responsibility for within Noetic?
That is, he needed to shift his framing of his problem from insecurely proving worth towards securely providing value.
This shift in perspective no longer collapsed his choices into the binary of going All In or Staying Put. He could more cleanly stand for his own needs, while looking for an arrangement that simultaneously served Mario, Noetic and the world.
Money could be parsed as a way of sustainably serving the world by meeting his and his family’s needs. The alignment, or lack thereof, of purpose and meaning were no longer litigations upon his character. They could simply provide an overarching orientation, even if he made limited contributions in their direction. He no longer had to take responsibility for every consequence that was outside of his control. Instead, he could view the mantle of responsibility as an initial commitment of the value he’d create. And then stretch himself developmentally to hold more.
He made this transition by practicing what my coach calls Enrollment. Specifically, enrolling people into Win/Win or No Deal relationships with him.
The practice of Enrollment
Enrollment operationalizes the aspiration of shifting towards providing value by having value conversations with people.
Rohan followed these steps:
Articulate a single valuable change he’d like to make in the world, that he’d be willing to take formal responsibility for within Noetic.
Articulate how this might benefit Mario and/or Noetic.
Capture a bundle of such changes, grouped together by theme to make them legible.
Identify an existing role/title within Noetic that might align with this bundle. Incidentally, in the process Rohan realized that he didn’t want to be CTO. He wanted to lead a cyber security red-teaming group instead. Specifically to lead a team with sufficient authority to influence pausing development if they saw a sufficiently dangerous model.
Identify a band of financial compensation that would make participation with Noetic sustainable for him.
Have an initial conversation with Mario to contextualize not just what Rohan was looking for, but why. Most importantly, to use his bundle of responsibilities as a starting point for exploring fit, rather than as some immutable object of negotiation. He then had the option of either eliciting Mario’s feedback then and there, or perhaps scheduling another call once Mario had a chance to mull things over.
In this case, it turned out that Mario already had someone leading a cyber security red-teaming group. It’d be too organizationally complex to add Rohan at that part of the conversation. The broader Rohan was looking for also didn’t feel like a good fit for the governance mechanisms Mario had in mind, although he agreed with them in spirit.
They explored their options, and eventually both concluded that it wasn’t a fit at that time. Both left with a sense of closure that none of their previous conversations had produced. Most importantly, it left Rohan with a sense of spaciousness he didn’t have before, because it wasn’t a litigation of his worth. He realized he could have similar enrollment conversations with any of the people that had been reaching out to him, including Apollo!
The hardest part for Rohan was to bring into consciousness the specific sorts of value he actually wanted to hold responsibility for. But bringing it into consciousness substantially expanded the frame of possibilities for him.
What sort of specific value are you trying to create in the world?
If this essay resonated with you, please don’t hesitate to reach out at varun@doubleascent.com
Acknowledgements
Brian Whetten for everything he’s taught me about growing my own leadership, developmental patterns, enrollment, etc. Essentially everything in this essay came from him.
John Vervaeke whose work on mapping human cognition and exploration of Relevance Realization was instrumental for articulating this essay.
David Chapman and Charlie Awbery for their work on mapping out and developing practices for the interplay between nebulosity and pattern.