The changing form of modern boom-and-bust cycles and the human knowledge graph
We always gotta start with the ML
This blog post comes a week late and is going to be a fairly unedited record of interactions and conversations that have inspired me and things I’ve learned and tried to put into words over the last two weeks. Thank you for reading — please comment here or reach out on Twitter if something speaks to you.
Building resilient products and organizations in high-growth environments
Last week, a senior leader and seasoned Silicon Valley veteran I respect at my company spoke to my team about his firsthand viewpoint over the past few decades with instances of the Gartner hype cycle (e.g. currently, generative AI and LLMs), with a technical talk focus on the data quality and infrastructural challenges we uniquely face in our slice of industry. His central thesis of what makes or breaks product engineering teams in hype cycles was investing in high-quality infrastructure and process early enough such that continued engineering growth is sustainable at a healthy pace, but (especially critical to early-stage startups) not so prescriptively and heavy-handedly that talent and innovation is stifled.
Shreya Shankar just published this incredible blog post highlighting some of the most critical infrastructural bottlenecks and development perception challenges underscoring this period of explosive growth in AI/ML:
If teams don’t start with more realistic expectations—meaning, they don’t plan for bad initial outputs and don’t invest in workflows to continually improve the application based on new data and failure modes observed—the application is likely to disappoint and fail.
AI engineering has to mirror successful paradigms of iteration in traditional ML engineering—otherwise we are doomed, as a community, to repeat the same AI-in-production mistakes that we’ve made in the past.6
Shreya’s graphic does a better job of capturing what comes at the end of the cycle, after the analog to the Gartner hype cycle “plateau of productivity”.
I hope that reality proves more optimistic than my cynical expectations, ultimately reflecting a more successful—and widespread—integration of AI in practical applications.
My less conventional take on this, having worked in environments all across the spectrum of business-product-engineering-ML, is that people and even businesses can have the best of intentions but that those very characteristics of generosity and (for lack of better term) idealistic naïveté are precisely what make them susceptible to being misunderstood, mis-priced and overtaken by fickle, unpredictable economic and societal forces.
I used to approach work and my personal life with the same unbridled excitement and passion. I made rash, emotionally-driven decisions that I stand by in principle but wouldn’t communicate the same way today (e.g. putting my credibility on the table by fighting to extend offers with insufficient data but citing the potential I saw in the candidates; sending long emails at 3am with a crop of follow-up ideas and offering to help implement the next day on tight deadlines, which more often derailed and distracted more junior members of my team rather than inspired and encouraged them; trying to fight brittle employee incentive and retention systems I disagreed with from within, to unwilling ears) and felt the consequences deeply. I am extraordinarily grateful to have met professional mentors early on in my career that have become personal mentors as we’ve moved on from our former companies. Over time, supported by these mentors and advocates past and present, I have stepped into higher-impact, higher-visibility roles and see my purpose in my professional life as to elevate deserving, underserved talent. In practice, this means continuously building up my technical and organizational experience and communication ability to a strength that allows me to step up and into mentorship and advocacy at these levels.
When it comes to building ML products rooted in solid infrastructure, which Shreya describes better than I can, I’ll simply offer a few notes drawn from my time in large corporate platform and infrastructural engineering organizations. In my experience, modern engineering teams will be less than the sum of their parts not necessarily because management is non-technical (can be a correlated symptom, not a cause), but more because management is unwilling to relinquish appropriate control and because engineering teams are understandably wary and untrusting of management based on past experiences. In past industrial revolutions of large-scale agricultural labor, automated assembly lines, and mass-produced quality control, a company’s incremental edge could very well be in defining process to squeeze out every literal mechanical and workforce optimization possible. Human labor was cheaper and more specific. You will see these practices outlined in most MBA case studies on Ford, GE, General Mills, and the like.
The modern industrial revolution is shaped differently because each historical iteration and process improvement adds an additional variable to an exponentially more complex equation of human capital and the nature of work. Machine learning especially is incredibly complex and widely misunderstood; innovating at the cutting-edge is unprecedentedly risky. As Shreya alludes to, the only thing that cannot be automated and abstracted away is human interaction; in my view, genuine and ethical productive organizational psychology and thoughtful balance of expectations and behaviors now define more than ever the core management philosophies that will allow organizations to grow into and stay more than the sum of their parts.
Human knowledge and impact multiplication
A few months ago I started to reflect on what studying reinforcement learning did to my brain, specifically a conscious unraveling and piecing back together of my emotional responses in an learned action-reaction framework. I recently met Abhi, who is incredibly (scarily) smart and building a very impressive blog, and who taught me more about microbiomes (more than 90% of total body serotonin comes from the gut, and appropriately we tried some no-yeast fermented sourdough olive bread) and gene therapy (data problems galore!) in just an hour than I ever could have discovered on my own in ten, even a hundred hours. I also met Cold, who is a proper social media genius and taught me more about TikTok, Instagram, Twitter, Substack and the demographic and platform shifts that defined the rise (and fall) of Vine and Facebook (as it was) than any documentary ever could.
I consistently find myself in complete awe of people like this. They are uber-highly-concentrated nodes in the seven-billion-span human knowledge graph of the world. Thinking organizationally, it’s a worthy effort to not just identify and retain talent like this, but invest more in understanding and developing this talent. Studying the ways we learn, communicate and collaborate is often relegated to organizational psychologists and human resources but it shouldn’t be. As an example, running efficient engineering teams in the modern technological landscape is often majority incentives management (motivation via salary bumps, bonuses and promotions — and in healthy organizations, rewarding good collaboration and stewardship) and in my experience most organizations significantly undervalue strong “soft skill” role models (open communication and desire to educate others).
I’ve been discussing with a number of friends recently what principles underly strong communities and strong organizations. As the Gartner hype cycle and Shreya’s graphic illustrate, we’re in a temporal period where a unique conflux of societal and technological factors mean there is long-term opportunity in establishing strong fundamental values and principles, but also the risk of losing out on peaks (and missing the curve entirely) because of an over-investment (perceived or real) in the “plumbing” and process work that may slow down development in the short term.
All said and done, slow and steady wins the race; the only lasting assets are the human relationships and lived experiences we’ve built along the way (I’ll throw another cliché at you — it’s the journey, not the destination), and there may be no right or wrong outcome but there are core morals and ethics that define us.