The Latent Space Gardener: How to Stay a Relevant Musician When Your Skill Becomes a Prompt
Lessons from a veteran music producer on staying relevant in the age of AI-generated art.
I started this podcast series (Relevance—AI changed Work. How to stay relevant?) because of a series of private messages.
After I wrote about how AI was fundamentally changing the nature of coding, I saw a surge of people reaching out—not publicly, but in my DMs. They weren’t just curious about the tech; they were trying to figure out what these shifts meant for their own lives and careers. They were asking the same core question: How do I stay relevant?
The answer started to crystallize for me during a conversation in a college alumni group. My friend Jay described how he was using AI in his music production workflow.
Now, Jay isn’t an amateur “prompter.” He’s a Berklee-trained engineer and producer who has spent decades in the trenches of the music industry. He lives at the confluence of deep musical theory and high-end software engineering.
I, on the other hand, have been trying to learn the handpan the “old” way. It’s a tedious, manual journey of building muscle memory. When I saw Jay’s workflow, the gap between my manual effort and his exponential output was jarring. I realized I had no idea how far the ceiling had moved.
Music is the “canary in the coal mine” for all of us. Whether you are a coder, a writer, or a designer, the shift Jay is navigating in the studio is the same one coming for your desk. We are moving from a world where we are paid for technical execution to one where we are paid for creative direction.
The Mental Model: Autoregressive vs. Diffusion
To understand how to work with these tools, you have to understand the two different “engines” driving them. Jay broke this down in a way that makes sense for anyone who has used an LLM like Claude or ChatGPT.
1. Autoregressive Models (The Storyteller)
These are the models we are most familiar with in the text world. They work by predicting the next “token” based on everything that came before.
In Music: This is great for structure and anticipation. It understands that after a certain chord, the human ear expects a specific resolution. It builds the “story” of the song linearly.
The Flaw: Just like a “drunk teenager,” it can commit to a path and “hallucinate,” losing the plot if the context window gets too crowded.
2. Diffusion Models (The Sculptor)
This is the tech behind tools like Midjourney, now applied to sound.
In Music: Imagine a block of marble (noise) that is slowly chipped away until a statue (a clean audio signal) remains. This is how AI creates texture and mood. It’s how you get that rich, “shimmering” sound of a nylon-string guitar.
The Flaw: This process creates “denoising artifacts.” In AI images, it’s the “melted hands” effect; in music, it’s a slight digital blur or a loss of “crispness.”
When you understand these two engines, you stop looking at AI as a “black box.” You start seeing the seams. You realize that while the AI is incredible at generating probability (what sound usually comes next), it is still fundamentally missing human intentionality.
The “Missing Layer”: Why AI Can’t Create a Signature
If AI has access to the sum total of recorded music, why hasn’t it replaced the artist? Jay pointed out a fundamental gap: The lack of persistent musical memory.
1. The Amnesia Problem
Current AI models are incredible at the “now.” They can generate a breathtaking 30-second clip, but they struggle with thematic coherence. If you want to create a 10-song concept album where a subtle melodic motif returns in a different key during the climax—the way a composer like John Williams or a band like Pink Floyd does—the AI fails. It doesn’t remember the “soul” of the project once the session ends.
2. Probability vs. Intentionality
AI is a probabilistic engine. It chooses the most likely next note. But art is often about the unlikely choice. Jay shared a moment where he completely overrode the AI’s suggestion. The AI provided a mathematically “correct” resolution, but it felt sterile. Jay chose a “blemished” or unexpected chord that evoked the specific emotion he wanted.
“The question isn’t whether AI can make music. It’s whether it can make music that sounds like you.” — Jay Swami
The Shift: From Musician to “Latent Space Gardener”
If the AI is doing the execution, the human moves into a role that looks more like a Gardener.
In AI terms, “Latent Space” is the vast map of all possible sounds a model can create. As a creator, you are no longer building the seed from scratch; you are deciding which ones to plant, which ones to water, and—most importantly—which ones to prune.
Curation as a High-Value Skill: When an AI can give you 100 variations of a melody in seconds, the person who can identify the one that actually resonates is the one who wins. Value is moving from scarcity of ability to quality of taste.
The “Last Mile” Polish: The AI gets you 90% of the way there in 10% of the time. But that final 10%—the human “swing” in the drums or the specific EQ choice—is where the professional resides.
The Handpan Paradox: Why We Still Do the Hard Things
I asked Jay: Does he still find joy in this? When the “struggle” of technical execution is removed, is the satisfaction fleeting?
Jay sees his role like a CEO. A CEO doesn’t write every line of code, but they steer the ship. The satisfaction doesn’t come from the “grunt work”; it comes from seeing a multidimensional vision come to life.
Paradoxically, using AI hasn’t made Jay stop playing “real” instruments. It’s driven him back to them. Because the AI handles the “boilerplate” so quickly, he can focus his mental bandwidth on the parts that require a true human touch—the soulful guitar solo or the specific “blemish” that makes a track feel alive.
Final Thought: The New Bar for Relevance
The democratization of music is real. But as Jay reminded me, the question isn’t “Can AI make music?”
The question is: What kind of music requires you?
If your value was only in the “execution”—the ability to follow a formula—you are in trouble. But if your value is in your intent, your taste, and your persistent creative vision, then the ceiling for what you can build has just been removed.
Don’t be a “prompter.” Be a Gardener.
Watch the full conversation below to see Jay demonstrate his workflow and hear the live creation of a Punjabi-infused podcast theme.