Vibes won't cut it — Chris Kelly, Augment Code
00:00:00.000 | Thanks for coming by the way and for sticking around for a little while. If you aren't prepared,
00:00:22.240 | I hate to break it to you, but this time next year half of us won't even be here anymore.
00:00:27.000 | That's basically if you listen to whatever the hype is about AI and AI coding.
00:00:35.000 | You know, there's lots of fanfare, no disrespect to very intelligent people that made these quotes.
00:00:41.000 | But I think they're probably wrong, not because I don't think AI coding is going somewhere important,
00:00:47.000 | but probably because they haven't actually touched a production system in a very, very long time.
00:00:52.000 | And so maybe generating code at 30% isn't really what they think it is.
00:00:58.000 | Because really AI code is still code.
00:01:02.000 | One thing they don't really recognize in this space is that they're working in very, very,
00:01:08.000 | very large code bases that have basically every decision that's ever needed to be made about that code,
00:01:13.000 | about that architecture, about that infrastructure has already been made for them.
00:01:18.000 | So if I'm generating 30% of my code against, you know, let's say they're doing thousands of lines a day
00:01:24.000 | against millions of lines of new code, of existing code, there's not a lot of wiggle room for what that code can do or should do.
00:01:31.000 | Right? Like it's just another button.
00:01:34.000 | If you've ever, no offense, anyone from meta here before I start ripping on meta?
00:01:37.000 | No, no meta. Great. I'll rip on meta for a minute.
00:01:40.000 | If you've ever talked to a meta engineer, they will talk about how they built a button in the ads platform for six months.
00:01:46.000 | Like that's what they worked on for six months. That was their job.
00:01:49.000 | So like there's very little definition about like what this can, needs to do that has wiggle room that AI really struck, can influence.
00:01:58.000 | Next AI is still writing code. This is code we have written for 50 years.
00:02:03.000 | Same programming languages. Nothing is different there.
00:02:05.000 | That code still needs to run in production somewhere.
00:02:08.000 | And if you're not familiar with, if you haven't run a large production system,
00:02:11.000 | even if you write a great line of code in complex systems, things fail.
00:02:16.000 | Like complex systems have emergent behavior that don't show up in just single lines of code.
00:02:20.000 | And so we still have to run this stuff.
00:02:22.000 | And so who's going to fix it? Who's going to examine it?
00:02:25.000 | Who's going to understand those nuances when, when, if you don't have software engineers?
00:02:30.000 | So I think we're still going to have software engineers.
00:02:32.000 | And then also history repeats itself.
00:02:34.000 | This is not the first time I've been told that my career is over.
00:02:38.000 | I'm getting in on the, on the years at this point in time,
00:02:42.000 | but anyone around like the DevOps transformation from 15 years ago, cloud,
00:02:47.000 | all of the, all the sys admins that I know that were racking boxes and booting kernels,
00:02:52.000 | all got pay raises and all work on much more valuable things.
00:02:56.000 | Now they're much happier doing the work they're doing.
00:02:58.000 | So this isn't new.
00:02:59.000 | This is just a different level of abstraction, right?
00:03:01.000 | Like tractors didn't get rid of farms.
00:03:03.000 | They just got rid of farmhands and horses.
00:03:05.000 | Yeah. Like, yes, there will be change in an industry for certain,
00:03:08.000 | but tractors didn't get rid of farming.
00:03:10.000 | We still have to farm.
00:03:12.000 | So vibes, anyone here a vibe coder?
00:03:17.000 | It's okay if you are, no shame.
00:03:19.000 | Anyone? And who here thinks call it would call themselves a software,
00:03:22.000 | a professional software engineer?
00:03:24.000 | Great.
00:03:25.000 | So vibe coding, if you're not familiar with the term,
00:03:27.000 | I think everyone in this room is, but I'll do a quick summary is basically letting the AI write all of the code.
00:03:32.000 | and think through the code and not really examining the code at all.
00:03:35.000 | Just does it do what I need it to do?
00:03:37.000 | And then in that case, like, I just keep going and let it keep going.
00:03:42.000 | Keep going. Don't examine the code and edit it that way.
00:03:45.000 | But that's not what we're talking about.
00:03:47.000 | I'm talking about how can we write code that's for production.
00:03:50.000 | And when I say production, I mean, you have four nines.
00:03:54.000 | You even know what four nines means.
00:03:56.000 | That's production code.
00:03:57.000 | You have thousands of users, gigabytes of data.
00:04:00.000 | We're talking software that runs the internet today.
00:04:03.000 | Vibes don't cut that.
00:04:05.000 | Because there's a lot of, like, nuances on what goes into code.
00:04:09.000 | So first, let's clear up one misconception.
00:04:12.000 | Code is not the job.
00:04:14.000 | In the same way that blueprints are not the job of an architect.
00:04:17.000 | That's an artifact of being an architect.
00:04:19.000 | The artifact of being a software developer is code.
00:04:22.000 | Sure, yeah, I have to output some code.
00:04:24.000 | But I make thousands of decisions about what my software is supposed to do.
00:04:28.000 | What kind of -- what I'm writing.
00:04:30.000 | What packages I'm bringing in.
00:04:31.000 | So let's stop conflating generating code with the art and the craft of doing software engineering.
00:04:38.000 | Those are different things.
00:04:39.000 | And so, yes, LLMs are great at producing code.
00:04:43.000 | But are they -- is that the same as writing production software?
00:04:47.000 | So a very smart person that started Stack Overflow, Jeff Atwood, rest in peace Stack Overflow, said the best code is no code at all.
00:04:56.000 | And I think that's true because every line of code comes with a burden, right?
00:05:00.000 | I have to maintain that code.
00:05:01.000 | I have to debug that code.
00:05:02.000 | So every line of code I generate, I have to be responsible for.
00:05:06.000 | And so we've been -- we spent so much time thinking about, like, how much code can AI generate?
00:05:11.000 | Who cares how much it can generate?
00:05:13.000 | The more it generates, the worse off I end up being -- worse off the system ends up being.
00:05:17.000 | We want to put as little code as possible in there.
00:05:19.000 | Because all codes have trade-offs, right?
00:05:22.000 | Like, we recognize this package has this performance characteristic.
00:05:26.000 | This methodology has this.
00:05:29.000 | The best example I can give you -- I'll try to get through so you can all get back to snacks --
00:05:33.000 | is the difference between, like, a monolith, a microservices architecture, and an adventure system.
00:05:39.000 | How many decisions go into those kinds of architectures and the choices you have to make to do the same thing?
00:05:45.000 | Like, if you've ever -- if you built a flight booking system in all three of these, you have thousands of individual decisions
00:05:51.000 | that have to get made.
00:05:53.000 | LLMs don't make decisions.
00:05:54.000 | They generate text.
00:05:55.000 | They generate patterns.
00:05:56.000 | And at some scale, at some point, there's no pattern for my software.
00:06:02.000 | Anyone here run a piece of production software that's kind of a bit of a snowflake?
00:06:06.000 | That's a bunch of, like, idiosyncrasies in it that, like, well, only Bob knows how that works and only Jane can fix that.
00:06:12.000 | She wrote it six years ago and hasn't been on the project.
00:06:15.000 | And that is software.
00:06:16.000 | So at some scale, pattern matching doesn't work anymore because all of the nuances that go into that software can't be pattern matched against.
00:06:24.000 | So when the software is going down -- this is my, like, trauma of, like, carrying a pager -- when the software goes down at 2 in the morning,
00:06:31.000 | vibes aren't going to fix the bug.
00:06:33.000 | Like, someone has to diagnose that problem.
00:06:36.000 | So what is the work of software engineering?
00:06:39.000 | For me, that's changing software safely.
00:06:42.000 | That's been my job for 20 years is how can I make changes to software, whether that's adding new functionality or changing existing code.
00:06:49.000 | And how do I do it safely so that software doesn't go down so my users get the thing that they're getting, the widget ships, that, you know, data is secure.
00:06:58.000 | So how do we -- how have we done that so far in the industry?
00:07:01.000 | We've done lots of different things to solve that problem.
00:07:03.000 | One, there's just my own knowledge.
00:07:05.000 | I have to, like, learn a shit ton about a code base before I can make changes safely.
00:07:09.000 | We use virtual control to do that.
00:07:11.000 | We know testing.
00:07:12.000 | You know, like, that's why we write tests, to catch -- if, like, if I change this, did this thing break over here?
00:07:17.000 | We use type systems.
00:07:18.000 | We use deployment strategies.
00:07:20.000 | Can AI start to help us?
00:07:22.000 | Can context that an AI has about and can understand more of the code base help us?
00:07:28.000 | Probably.
00:07:29.000 | You know, at Augment, we believe that context is, like, the most important part of all AI generation in code.
00:07:34.000 | So, yeah, we think we can solve that problem.
00:07:36.000 | That doesn't change that I still have to care about production.
00:07:41.000 | So let's just assume that we're writing code.
00:07:44.000 | Like, okay, the world -- the future is here.
00:07:47.000 | I have to use AI.
00:07:49.000 | The thing I find the most interesting about this space, I've been in a long -- in this -- in DevTools for a long time, is professional software engineers are the last people I see adopting AI.
00:08:00.000 | And I've never seen that before.
00:08:02.000 | Like, I've seen version control systems change.
00:08:04.000 | I was at GitHub in early days.
00:08:06.000 | So, like, massive jumps to get the cloud transformation.
00:08:10.000 | Pick your -- pick your innovation.
00:08:12.000 | And developers are like, hell yes, give me that new thing.
00:08:15.000 | I don't know why.
00:08:16.000 | Well, I have some hypotheses.
00:08:18.000 | But I don't fully understand why software developers are like, I'm not touching code.
00:08:22.000 | I'm not touching AI.
00:08:23.000 | It's like, it can't do what it's going to do.
00:08:25.000 | So I want to talk about that for just a couple minutes.
00:08:29.000 | So a few years ago, AI coding was mostly just a pile of bricks.
00:08:34.000 | Like, it kind of worked, but really didn't do much.
00:08:38.000 | About a year ago, you know, years and change when Sonic 3.5 came out.
00:08:42.000 | That's really when we saw a massive explosion in AI coding because the quality substantially jumped.
00:08:49.000 | And then four weeks ago, if you weren't watching the news, literally every AI coding tool said, like, agents are the future.
00:08:57.000 | We did.
00:08:59.000 | Very proud of that.
00:09:00.000 | Very proud of that.
00:09:01.000 | But like, now agents are everywhere, right?
00:09:03.000 | And so this transformation has happened very, very quickly.
00:09:06.000 | And so I want to talk about how we can talk -- do software engineering with this new future that we're seeing.
00:09:14.000 | So how do you build software that's easy for AI to write?
00:09:19.000 | A couple tips for you.
00:09:21.000 | Have some documented standards and practices, right?
00:09:24.000 | Like, what do you use?
00:09:25.000 | Every code base I know is in some sort of flux, right?
00:09:28.000 | So like, are we using this package or this package?
00:09:30.000 | Okay, well, document that.
00:09:32.000 | Let the AI know that this is the next direction of your code base.
00:09:35.000 | Have reproducible environments, right?
00:09:38.000 | Like, can you easily spin up a developer environment?
00:09:41.000 | Is your developer environment very bespoke and unique?
00:09:44.000 | You want to have it reproducible.
00:09:46.000 | We want to have easy testing, right?
00:09:48.000 | Like, can you run your tests locally?
00:09:50.000 | Can -- is that fast?
00:09:52.000 | You know, that's pretty standard stuff.
00:09:54.000 | Establish clear boundaries of what you're going to do.
00:09:57.000 | You're never going to give AI the idea of, like, extract this module using the Strangler pattern.
00:10:02.000 | What does that even mean to an AI?
00:10:04.000 | Like, that's a whole -- you have to give clear boundaries of what you're trying to build and how to get the AI to do it.
00:10:10.000 | And lastly, like, have clearly defined tasks and work.
00:10:13.000 | Because AI is going to be as -- you know, I wouldn't give any engineer on my team, whether they're senior, staff, junior, just a vague task of, like, could you make this button do something different?
00:10:24.000 | Like, I see too many engineers prompting in that way.
00:10:27.000 | And what's interesting about this to me is, like, this just sounds like software engineering.
00:10:32.000 | Right?
00:10:33.000 | Like, if we don't -- ideally, your software engineering stack looks -- has these qualities.
00:10:38.000 | And if it doesn't, you're like, our productivity sucks.
00:10:40.000 | Because we have, you know, bespoke testing infrastructure.
00:10:43.000 | We have bespoke developer environments.
00:10:45.000 | You have to give AI the same tools that engineers need.
00:10:49.000 | Because it's doing exactly the same job.
00:10:51.000 | It's writing code.
00:10:52.000 | I've never one-shotted a piece of code in my life personally.
00:10:55.000 | Like, I've always made mistakes in the code I write.
00:10:58.000 | I run a test.
00:10:59.000 | It fails.
00:11:00.000 | I've got a linter.
00:11:01.000 | It fixes it.
00:11:02.000 | Whatever the thing is.
00:11:03.000 | But we've had this expectation that AI can, like, write perfect code.
00:11:07.000 | And I don't know why.
00:11:08.000 | And so when you're thinking about adopting AI as a software engineer, you have to make sure
00:11:12.000 | that your systems work like you would expect any other engineer to work.
00:11:17.000 | Because that's how it's writing code.
00:11:21.000 | The next thing is code review is by far the most important skill.
00:11:25.000 | I think we've probably forgotten that skill as an industry.
00:11:28.000 | We probably should have been interviewing for code review and not, like, here's this esoteric
00:11:32.000 | leak code problem that you can solve.
00:11:34.000 | But, like, can you read somebody else's code and comment on why it's good or bad?
00:11:38.000 | I think this is going to become far more important as agents are writing more and more code.
00:11:43.000 | And our code review tools today, frankly, suck.
00:11:46.000 | Like, I'm getting a list of changed files.
00:11:48.000 | It's lexicographically sorted.
00:11:49.000 | So, like, great.
00:11:50.000 | File A changed.
00:11:51.000 | Let me read what changed in file A.
00:11:53.000 | What changed in file B?
00:11:54.000 | That's not a way to think about how software changes.
00:11:56.000 | It's in the order of the files.
00:11:58.000 | And so I think we're going to see a pretty big explosion on the way code review can happen.
00:12:03.000 | And that's the skill we need to be interviewing for.
00:12:05.000 | You need to be, like, kind of brushing up on.
00:12:09.000 | All right.
00:12:10.000 | I'm running quick on time.
00:12:12.000 | So I do want to give you a couple highlights of, like, if you are one of those software engineers
00:12:16.000 | that's like, I don't, you know, I'm not, I don't trust AI.
00:12:18.000 | I need some tips.
00:12:19.000 | I want to leave you a few of these.
00:12:21.000 | Feel free to take a screenshot.
00:12:22.000 | One, the most important thing I can imagine is AI talks like a human, but it's actually a machine.
00:12:29.000 | I had this interaction with AI the other day where it was like, you know, I was yelling at it because that's what you do at AI when it doesn't do what you want it to do.
00:12:36.000 | And it's like, oh, I'm sorry.
00:12:37.000 | I just scanned that file.
00:12:38.000 | I didn't read it.
00:12:39.000 | I'm like, the fuck does that even mean?
00:12:41.000 | What, how does a piece of software scan a file?
00:12:45.000 | It just, like, skimmed it?
00:12:46.000 | Like, that's not a thing in software.
00:12:48.000 | It's because LLMs are trained on all the data in the world.
00:12:52.000 | Right?
00:12:53.000 | There are thousands of emails that it has read that said, like, oh, sorry, I didn't read your document thoroughly.
00:12:57.000 | I actually just skimmed it.
00:12:58.000 | It's like, oh, that's what I should say when someone's yelling at me about not reading a file.
00:13:02.000 | Right?
00:13:03.000 | And so we have, we have to distrust some of the things that LLMs are saying they're doing because it's not actually doing that.
00:13:09.000 | Don't forget, it's just generating text.
00:13:11.000 | It's not always doing exactly what it's outputting in that text.
00:13:15.000 | So keep that in mind when you're reading through LLM output.
00:13:20.000 | Let's see, what other quick tips can I give you?
00:13:24.000 | Sometimes code is just different.
00:13:26.000 | It's okay if the LLM outputs code differently than you would.
00:13:30.000 | I can't expect it to produce the code that I would exactly.
00:13:33.000 | In the same way, like, the person that sits next to me also writes code a little differently than I do.
00:13:37.000 | And so just accept that that's okay.
00:13:40.000 | And if you want to force it to write code like you, you can spend that energy, but know the difference.
00:13:45.000 | Is the code better or is it just different?
00:13:47.000 | And so let go of some of that, like, this is how, you know, we, this is why we have linters.
00:13:53.000 | This is why we have rule systems.
00:13:54.000 | That's why we have style guides.
00:13:56.000 | So we can stop arguing about, like, is that how you define a function?
00:14:00.000 | Or is this how you define a function?
00:14:02.000 | Let some of that go if you can.
00:14:05.000 | Write a rules file.
00:14:07.000 | You know, give it, tell it what you want it to do.
00:14:09.000 | Have a file.
00:14:10.000 | I always start all of my projects with, like, here's the stack I'm using.
00:14:14.000 | Here's the, like, guidelines I want you to use.
00:14:16.000 | And that's always ends up being part of the context I send with the LLM.
00:14:20.000 | And then lastly, I like to define, create, refine loop.
00:14:24.000 | Expect, create a document of some kind.
00:14:27.000 | Have the LLM help you generate it.
00:14:28.000 | Like, I'm doing this thing.
00:14:30.000 | Write a markdown file that would, like, lay out the plan.
00:14:33.000 | Here's my plan.
00:14:34.000 | Save that as a markdown file.
00:14:35.000 | Then use that again in your context.
00:14:38.000 | Have it create the thing.
00:14:39.000 | Have the agent run against that file.
00:14:41.000 | You know, you can make your edits to it.
00:14:42.000 | And then refine it.
00:14:44.000 | Then you're going in with, you know, code completions or whatever.
00:14:47.000 | Like, just tweak the things that you want to tweak.
00:14:49.000 | And just do that loop over and over and over again.
00:14:51.000 | Make a plan.
00:14:52.000 | Have it create it.
00:14:53.000 | And then just make your tweaks.
00:14:54.000 | And you'll get a lot more comfortable.
00:14:56.000 | One, how to prompt the LLM to get the code you want to do.
00:14:59.000 | As well as, that's just a very efficient way of coding.
00:15:03.000 | And if you've let go of that code has to be how I would write it.
00:15:07.000 | Not just good functional code.
00:15:09.000 | Then you can get a lot more productive in that way.
00:15:12.000 | Thanks again.
00:15:14.000 | I blasted through that.
00:15:16.000 | I'm happy to talk.
00:15:17.000 | Augment is in the expo hall.
00:15:18.000 | I'll be around there to chat or over here.
00:15:20.000 | I hope to see you here next year.
00:15:22.000 | Hope we're all here next year.
00:15:23.000 | I think the jobs aren't going anywhere.
00:15:25.000 | But happy coding.
00:15:26.000 | Really appreciate it.
00:15:27.000 | I'll see you next week.
00:15:28.000 | We'll see you next week.
00:15:28.000 | We'll see you next week.
00:15:29.000 | We'll see you next week.
00:15:30.000 | We'll see you next week.
00:15:31.000 | We'll see you next week.
00:15:32.000 | Bye.