Mentoring the Machine — Eric Hou, Augment Code
00:00:00.000 | My name is Eric, member of technical staff at Augment Code, and today's talk is about mentoring the machine.
00:00:19.720 | Now, the talk today is a personal story, a glimpse into how we at Augment use AI to build production-grade software,
00:00:26.560 | and how that's changed how we operate both as a team and as a business.
00:00:31.480 | So at Augment, we build for real software engineering at scale in production.
00:00:36.860 | Before Augment, I spent six years building products and standards for the automotive industry,
00:00:41.460 | and my peers and I have created and maintained systems that tens of thousands of engineers have touched,
00:00:46.660 | where not one person fully understands even how 5% of the system works.
00:00:50.300 | So we kind of understand the get-your-hands-dirty kind of work we as engineers have to do sometimes.
00:00:56.240 | Now, that's why we have all the line items that you would expect, but are kind of shockingly rare in today's vibe-coded world.
00:01:01.560 | If you wanted to learn more, please come visit our booth or visit us at augmentcode.com.
00:01:07.840 | Now, let me walk you through our journey, which is broken into four sections.
00:01:12.180 | The first two go through my personal journey as an engineer at Augment,
00:01:15.940 | and how I, as well as other engineers, learn to use agents most effectively.
00:01:20.540 | And the second two discuss the gaps that most organizations, and even our own, face when trying to adopt agentic AI,
00:01:27.260 | and how we can address those gaps to solve both our current problems and unlock new opportunities in our businesses.
00:01:32.920 | So without further ado, let's dive into my own journey to realization,
00:01:37.640 | which actually happened a few months ago as we were first rolling out the Augment Agent.
00:01:42.820 | So picture this.
00:01:44.080 | It's Tuesday morning.
00:01:45.040 | For me, it's about to be one of those days.
00:01:48.300 | You're all probably going to recognize this day.
00:01:50.440 | In fact, most of you have probably lived it many times.
00:01:53.320 | But at the moment for me, just another Tuesday.
00:01:56.280 | Now, it's 9 a.m.
00:01:58.520 | I'm behind on a critical design system component that was supposed to merge last Friday.
00:02:02.460 | The design team's waiting.
00:02:03.960 | Downstream teams are waiting.
00:02:05.420 | I'm feeling the pressure, but I'm determined to knock it out.
00:02:08.460 | So clear calendar, cup of coffee in hand, and my fingers are just hitting the keyboard.
00:02:12.520 | 9.30 a.m., my phone buzzes.
00:02:15.340 | Staging emergency.
00:02:16.280 | The main API endpoint is completely broken.
00:02:19.140 | There's a request format mismatch between our client and server,
00:02:23.140 | blocking all QA testing and blocking deployments.
00:02:26.860 | So the primary's on vacation at this point.
00:02:29.060 | I'm the service secondary, and I'm responsible for the on-call process
00:02:32.120 | and seeing that through.
00:02:33.620 | So my carefully planned day just evaporated like that.
00:02:37.500 | 10.15.
00:02:39.660 | I'm starting to wrap up some service log exploration,
00:02:42.500 | and the new engineer that I'm mentoring slacks me,
00:02:45.780 | hey, when you have a minute, can you help me understand
00:02:48.420 | how our extension system works?
00:02:50.300 | I'm stuck.
00:02:51.300 | Now, if you've been an engineer before, you've been here.
00:02:56.440 | That sinking feeling when your day derails.
00:02:58.360 | You're pulled in three different directions.
00:02:59.780 | You go home feeling like you've accomplished nothing,
00:03:02.440 | even though you've worked for 12 hours.
00:03:03.940 | And you know that when you wake up the next morning,
00:03:06.200 | the on-call remediation is going to put another week or two of work
00:03:10.520 | on your plate.
00:03:11.420 | And if that scenario felt familiar, you're not alone.
00:03:14.760 | This is not just your team, not just your company, or bad luck.
00:03:19.520 | Every single interruption costs us 23 minutes of recovery time.
00:03:23.140 | And as an industry, we're spending two-thirds of our time maintaining code
00:03:27.720 | instead of building new features.
00:03:29.360 | That translates to $300 billion annually spent on context switching and firefighting.
00:03:35.560 | So we've normalized this chaos, and we've accepted that days like this
00:03:39.800 | So just part of being an engineer.
00:03:41.720 | Of course, this is an AI, you know, conference.
00:03:45.500 | So what if I told you it didn't need to be that way?
00:03:48.540 | In fact, what if I told you it already isn't?
00:03:51.680 | I'm going to show you exactly how.
00:03:55.920 | So let's see if I can bring this over.
00:03:58.880 | This is our product, the Augment Extension.
00:04:03.700 | It's got everything you like in your favorite AI coding assistants and more.
00:04:07.240 | But today, we're focusing on the agent.
00:04:12.380 | Now, here what we have is I want the agent to take on a personality.
00:04:21.220 | I want it to go ahead and talk to me about, you know, the AI world fair,
00:04:26.600 | the energy and excitement of San Francisco.
00:04:29.740 | And I'm going to go ahead, give it this prompt.
00:04:32.760 | And here are some guidelines that I'm going to give it.
00:04:35.260 | If you notice what these guidelines are,
00:04:37.220 | they're not telling you exactly what to implement.
00:04:39.080 | They're really drawing the boundaries for the agent itself.
00:04:42.880 | So I'm going to go ahead, press run, and let it run in the background.
00:04:47.700 | And we're going to, in the meantime, go back to the talk.
00:04:51.380 | So this seemingly simple example of working with the agent
00:04:59.920 | has kind of fundamentally transformed how we work.
00:05:02.400 | And a few months ago, it transformed what should have been a terrible day for me.
00:05:06.360 | So to see this in action, let's take a look at my Tuesday a little bit more in-depth.
00:05:12.980 | What actually happened and how this approach exemplifies the changes we've taken at Augment
00:05:16.880 | to integrate the growing capabilities of agents into our team.
00:05:20.160 | So it's Tuesday morning, 9 a.m.
00:05:22.740 | Before I grab my coffee, I start scoping out the design system component with an agent.
00:05:27.220 | And instead of micromanaging, what I'm doing is I'm scaffolding and providing context.
00:05:32.220 | I'm giving AI the outcomes, the context, constraints,
00:05:35.880 | and I'd have it perform the same tasks I'd expect of any other engineer.
00:05:39.860 | And so while AI goes and explores the code base and builds the RFC,
00:05:43.960 | I'm taking my morning coffee break.
00:05:45.560 | And when I return, it has a mostly completed RFC
00:05:48.160 | that follows our architectural patterns.
00:05:49.880 | At 9:30 a.m., my phone buzzes.
00:05:53.480 | The staging emergency is on my plate.
00:05:55.980 | And instead of dropping everything for six to eight hours of firefighting,
00:05:59.280 | I parallelize my work to parse through the noise.
00:06:03.120 | And so I take the component, hand it off to an agent,
00:06:06.680 | and it's working in the background for me.
00:06:08.560 | Two AI agents are working with me to help me parse through logs
00:06:13.600 | and performing a git bisect.
00:06:14.960 | And the Augment Slack bot helps me manage steps
00:06:17.880 | through communications with the teams that are, you know,
00:06:20.480 | annoyed that they can't deploy.
00:06:21.480 | So in this world, I'm not fighting fires anymore.
00:06:25.760 | What I'm doing is I'm orchestrating parallel AI work streams
00:06:28.980 | while I get to focus on the critical path of solving the on-call issue.
00:06:32.920 | At 10:15, the new hire interrupts my on-call flow.
00:06:37.660 | And here, our knowledge infrastructure really starts to kick in.
00:06:41.240 | I direct the new hire to the Augment Slack bot,
00:06:43.840 | which has access to our context engine, our code base,
00:06:46.520 | all of our documentation, linear, et cetera.
00:06:48.880 | Now the new hire can have personalized real-time help
00:06:52.340 | while I can stay focused on on-call response.
00:06:54.960 | By 11:00, I'm evaluating agents' work and coordinating the next steps.
00:07:00.840 | The design system component is complete.
00:07:03.440 | There's a storybook link, and my agents have found the bad commit
00:07:06.240 | and already reverted it.
00:07:07.440 | They've started writing up a post-mortem doc
00:07:09.480 | and already started exploring remediation.
00:07:11.520 | In this world, my role has shifted from implementation to evaluation.
00:07:19.120 | So now, I get ready to manage the deployment of the fix,
00:07:22.680 | and the agents are setting up -- off to tie up some loose ends.
00:07:27.300 | Now it's 12:00, lunch, food.
00:07:30.280 | I go eat while the agents are doing work for me.
00:07:33.360 | After lunch, I complete what should have been impossible.
00:07:37.040 | The Augment agents have executed the entire remediation process.
00:07:40.800 | The problem was with the gRPC library upgrade,
00:07:43.360 | and it touched 12 services, 20,000 lines of code.
00:07:47.440 | It has tests, it has a write-up, and actually,
00:07:50.320 | one of my engineering peers told me that it was quite surprising
00:07:55.200 | and really thanked me for pushing this across the line
00:07:58.320 | when, really, it was all the agents doing the work.
00:08:01.760 | So here, when a normal organization might estimate
00:08:04.480 | as maybe three weeks of engineering work
00:08:07.480 | to upgrade the gRPC services,
00:08:09.600 | is complete, tested, and, you know, almost ready for deployment.
00:08:13.400 | But, of course, it needs one final round of human policy.
00:08:17.400 | So the real transformation here is not just that I've completed this work
00:08:25.160 | in parallel. The real transformation is that I've unlocked time
00:08:28.520 | that I previously did not have.
00:08:30.600 | Now, that's not a dream. That's not a pitch.
00:08:34.120 | This scenario that I just described, all three of these challenges,
00:08:36.840 | was something that I personally had to face
00:08:39.320 | and solved in around half a day of active keyboard time.
00:08:43.720 | Same problems, same complexity, same time pressure,
00:08:47.080 | but instead of it being one of those days, it became a normal Tuesday.
00:08:50.680 | Now, what I just showed you is kind of the crux of how we at Augment work with agents today,
00:08:56.600 | by leveraging its unique strengths while compensating for its weaknesses.
00:09:00.360 | And that can be summed up in one core realization.
00:09:02.840 | To make the most use out of AI,
00:09:06.360 | we need to work with it as we would work with junior engineers.
00:09:09.400 | Not assigning tickets, but mentoring.
00:09:12.360 | Now, I know we've all heard this.
00:09:15.800 | We're all rolling our eyes a little bit.
00:09:17.160 | You know, AI has the intelligence of a junior engineer.
00:09:20.360 | Let's actually break down how this analogy applies,
00:09:24.200 | and more importantly, where it doesn't.
00:09:26.280 | Both AI and new engineers start with no context of your systems.
00:09:30.600 | They lack your organizational context.
00:09:33.160 | And most importantly, they lack years of experience working with systems.
00:09:36.520 | In your systems.
00:09:38.280 | So they're able to implement in isolation,
00:09:40.760 | but they kind of need a structured environment to work in to perform best.
00:09:44.600 | These three pieces make up what we call the context or knowledge gap.
00:09:50.200 | Now, in learning and speed is kind of where they differ really drastically.
00:09:54.760 | A junior engineer learns and executes fairly slowly,
00:09:57.480 | but they can retain and synthesize knowledge.
00:09:59.800 | While, if given the same information,
00:10:02.200 | AI can process it and implement what you want in minutes or even seconds,
00:10:06.440 | but forgets things between conversations.
00:10:09.160 | So for us, that means that AI is effectively a perpetually junior engineer,
00:10:15.160 | but one that can work on multiple tasks simultaneously and incredibly quickly.
00:10:19.560 | So to make the most use out of AI, we must become perpetual tech leads.
00:10:24.120 | We need to become mentors to our AI apprentices,
00:10:27.000 | just as we would become mentors to our juniors.
00:10:28.920 | Now, you might be thinking, this is almost great for individual engineers.
00:10:33.800 | What does this mean for teams, my organization?
00:10:36.680 | And that's the right question to ask,
00:10:38.760 | and where a lot of organizations struggle, including augment.
00:10:41.640 | So we've seen this pattern repeatedly.
00:10:44.120 | Individual engineers can achieve remarkable productivity gains with AI,
00:10:47.960 | but when the teams try to scale, progress stalls.
00:10:51.720 | Even when we first started working on the augment agent, actually,
00:10:55.160 | I remember people were saying, your agent is so good.
00:10:58.920 | How can I get what Eric has?
00:11:00.360 | Now, this is kind of indicative of two bigger problems.
00:11:06.520 | How can I replicate individual success with AI across teams?
00:11:09.960 | And how do we turn team productivity into sustainable business advantage?
00:11:14.200 | What's actually blocking real organizations from using AI effectively?
00:11:19.000 | This answer turns out to be fairly simple.
00:11:23.560 | It's not a new blocker.
00:11:28.760 | It's the same problem that makes new hires take six months to ramp up in your standard org,
00:11:35.240 | and why four out of every five engineers across our industry cite context deficit as the biggest blocker.
00:11:42.120 | So the core problem here is context.
00:11:45.160 | And we've had this problem for decades, even without AI in the mix.
00:11:48.360 | And so a paradox kind of arises in our industry.
00:11:52.760 | How can we hope to solve the knowledge infrastructure problem when it's still this bad for human teams?
00:11:58.520 | And how can we scale AI beyond an individual when we don't have the requisite knowledge infrastructure to do so?
00:12:07.560 | This doesn't mean writing more docs.
00:12:09.160 | This doesn't mean doing knowledge reorgs.
00:12:10.840 | This doesn't mean, you know, completely rebuilding your organization for AI.
00:12:14.920 | In that world, humans are serving AI, not the other way around.
00:12:19.240 | It means kind of choosing the right tools and systems that can institutionalize knowledge infrastructure for you.
00:12:26.120 | So here's how to get started.
00:12:29.800 | Companies that successfully use augment and other AI tools tend to follow a fairly similar pattern to get started,
00:12:36.840 | which we've distilled into three steps.
00:12:38.680 | The first step is knowledge gathering.
00:12:41.720 | Start by exploring your existing knowledge bases.
00:12:44.200 | What do you have documented?
00:12:45.800 | Map out your key knowledge sources: Notion, Google Docs, GitHub, etc.
00:12:51.320 | Fill in the critical knowledge gaps, specifically around meetings and decisions,
00:12:57.000 | with meeting intelligence tools to capture that knowledge that would otherwise be lost.
00:13:02.920 | In fact, actually, most of the meetings that I personally attend outside of engineering nowadays start and end with a granola AI recording,
00:13:10.440 | and it comes with basically a list of tasks that we can directly put into our task tracker at the end of it.
00:13:17.240 | And finally, begin integrating data sources using things like MCP and augment native audit integrations
00:13:22.120 | to create the beginnings of your knowledge infrastructure.
00:13:24.760 | Step two is starting to gain familiarity with your tools.
00:13:30.200 | This refers to both you gaining familiarity with the tools,
00:13:33.080 | but also letting the tools gain familiarity with you and your organization.
00:13:37.800 | More broadly, introduce these tools across your teams and enable them to explore the strengths and weaknesses of AI in your specific contexts.
00:13:46.200 | This is where you build up the muscle of working with AI and start teaching your platform of choice about things like coding patterns,
00:13:53.880 | architectural decisions, business logic, etc. Step three is leaning in. Expand the successful patterns you've discovered.
00:14:02.600 | And you can, at this point, start to entrust more complex tasks as you've built up trust and as your confidence in these systems grow.
00:14:10.600 | Share your successful memories and task lists across teams. This is where compound learning starts to really take off.
00:14:17.480 | When people were asking me about how can I get Eric's agent, we have a feature called memories, and I basically just shared that file with them.
00:14:25.000 | This is where, again, compound learning can take off and knowledge and individual successes can start to multiply and spread across your organization.
00:14:34.760 | So while us as engineers are working with AI systems by providing missing structure and guidance,
00:14:43.160 | successful organizations as a whole are enabling AI systems by institutionalizing their knowledge infrastructure.
00:14:48.760 | So now, if these things are possible now, how has that actually changed the way we operate at Augment?
00:14:58.360 | What future is actually available to us? Let me bring you back to the agent here and show you.
00:15:06.200 | So I have a development environment up here. So this is on the real Augment code base. This is our dev version of our build.
00:15:15.560 | You can see in the top, that's the extension development environment. And hopefully, when I type at, I can -- okay, personalities -- AI engineer world fair Augie.
00:15:26.360 | Awesome. You can see it even created a little icon. It's a little rough, but there it is.
00:15:33.560 | What is your favorite city? Let's see what it says. Awesome. There we go.
00:15:44.600 | Easy question. It's absolutely hands down San Francisco. I mean, are you kidding me?
00:15:50.840 | The city is the epicenter of the AI revolution. So that's awesome. You can see that, you know, as I was giving this talk with just a simple prompt, we were able to create a new personality.
00:16:01.880 | This kind of exemplifies some of the agentic personality stuff that was talked about earlier.
00:16:06.920 | But this, you know, really starts to change when, you know, if I can give a talk and also implement a feature,
00:16:16.600 | it really starts to change how we think about the economics of developing software.
00:16:24.680 | See, once we solve the knowledge infrastructure problem, everything starts to change.
00:16:28.840 | When information transfer becomes instant and scalable, we unlock AI's true economic potential,
00:16:36.280 | parallel exploration of your business. The traditional approach to building software starts with designing,
00:16:42.200 | then building, then testing. And each iteration locks us out of potential decisions at every single step.
00:16:47.640 | But when knowledge infrastructure exists, prototyping is cheap, and building takes fewer resources,
00:16:53.000 | we can do something drastically different. Instead of guessing at what might be the best approach,
00:16:58.040 | we can rapidly prototype, iterate, test, and then converge on a real decision based on real metrics
00:17:03.720 | and by putting our products in front of people. At Augment today, we constantly have prototypes floating
00:17:10.040 | around. We have a prototype of a VS Code fork in case we need it. Augments itself became -- or sorry,
00:17:17.080 | Agents itself began as a prototype as well. And many of the features in our product that our users love
00:17:22.600 | began as a prototype when an engineer at Augment just decided, hey, I'm going to try this and with an agent.
00:17:29.400 | And by trying multiple approaches simultaneously, again, we can quickly converge with real data on the best
00:17:36.680 | approaches that we should actually invest in productionizing without arguing, you know,
00:17:41.320 | in a room talking to each other about what might be best. As an engineer, we've all had to justify
00:17:48.040 | a design decision to leadership, complained about tech debt, or cursed our past selves for doing something
00:17:53.640 | in a particular way. And as leadership, we all wish we could go back and redo some critical decisions
00:17:58.280 | or enable our teams to do more strategic work instead of constantly throwing fires at them to put out.
00:18:03.720 | But with parallel exploration, we can turn these wishes from retroactive to proactive. By measuring
00:18:09.880 | and testing divergent approaches from the start, we can start making decisions better informed by data.
00:18:15.240 | And when we can measure hypotheses of designs, prototypes, and architectures early on and validate them,
00:18:22.120 | we reach a fascinating conclusion. If we use AI effectively to augment our organizations,
00:18:29.000 | we can make the creation of software more of a science, not less. And that begins with all of our engineers,
00:18:36.440 | organizations, teams choosing the best tools for our jobs that most effectively allow us to mentor our machines.
00:18:44.840 | Thank you all so much for your time. If what we talked about today resonates with you,
00:18:49.640 | please visit the Augment booth on the expo floor. Go to Augment.com, try us out for free. And Remote
00:18:56.120 | Agents is out this week. Let it parallelize your work for you. Thank you so much.