back to index

The AI emperor has no DAUs why most devs still don't use code AI: Quinn Slack


Whisper Transcript | Transcript Only Page

00:00:00.040 | QUINN SLACK:
00:00:14.060 | QUINN SLACK: Great to see you all.
00:00:15.060 | Again, I'm Quinn Slack.
00:00:16.260 | I'm CEO and co-founder here at Sourcegraph.
00:00:18.480 | And yes, I was at Bleach Report.
00:00:20.400 | I think I learned I'm better coding than doing sports.
00:00:23.120 | So I'm in the right role here.
00:00:25.240 | I want to talk to you about why most devs still do not
00:00:30.220 | use code AI.
00:00:31.160 | It's probably a mind-boggling fact to all the people in this room,
00:00:33.620 | but it's true.
00:00:35.020 | And we'll have a good discussion about that.
00:00:37.300 | I'm a coder.
00:00:37.960 | I code all the time.
00:00:39.200 | Been building Sourcegraph, which is code search and code AI
00:00:42.780 | for devs since 2013.
00:00:44.780 | I merged four PRs earlier today.
00:00:47.340 | But I want to talk mostly about the product and what to build
00:00:50.520 | and how to get it in the hands of users.
00:00:52.560 | Because if you don't do that right,
00:00:54.280 | then all the code that you write is for nil.
00:00:57.500 | So three points I want to make.
00:01:01.380 | One is that most devs don't use code AI.
00:01:05.880 | I want this to scare you.
00:01:07.640 | I want to scare you in this talk.
00:01:09.740 | And then I want to share some of the mistakes we made
00:01:12.040 | and some of the tips from our experience doing all the things
00:01:15.880 | wrong and then trying to figure out,
00:01:17.440 | how do we do it right at Sourcegraph
00:01:19.540 | as we've been building coding?
00:01:21.840 | MATT DAVIS: All right.
00:01:23.060 | First, just show of hands here.
00:01:24.440 | Who uses-- who in the last week has
00:01:27.420 | used any kind of code AI tool while coding?
00:01:31.540 | All right.
00:01:31.940 | Man, I wish the whole world was like this room.
00:01:34.520 | But we are in the heart of it in San Francisco.
00:01:37.400 | And the whole world is very different.
00:01:39.320 | Most devs do not use code AI.
00:01:41.060 | And I'm going to give some numbers here.
00:01:42.780 | I'm privy to a lot of private information and stuff
00:01:44.900 | that I've heard as well.
00:01:46.180 | And all of this is in line with what I've been hearing.
00:01:49.500 | So first number-- oh, man, this is a great statistic.
00:01:52.160 | 92% of devs use code AI tools at work.
00:01:55.560 | Wow.
00:01:56.040 | That means these tools must have tens of millions of users.
00:02:00.440 | This was a study from GitHub.
00:02:02.460 | 500 people.
00:02:03.040 | Well, it turns out this is only in the US.
00:02:05.520 | And they had a very broad definition of what it means
00:02:07.760 | to use a code AI tool.
00:02:10.000 | So here's a case where the hype kind of outstrips
00:02:12.660 | the reality, a common pattern.
00:02:15.660 | This is an infographic from GitHub.
00:02:17.220 | 70% of developers see a benefit to using AI coding tools at work.
00:02:22.180 | Surely if there's a benefit, they must all
00:02:23.900 | be using these things, right?
00:02:25.880 | I mean, if only it worked that way.
00:02:28.520 | All right, some numbers for the broader universe.
00:02:31.500 | There are 1.3 million paid subscribers to GitHub Copilot.
00:02:35.500 | This was what Microsoft reported in January 2024.
00:02:38.860 | 1.3 million is a lot.
00:02:40.820 | But if you're just paying for something,
00:02:42.500 | are you actually using it?
00:02:43.500 | And also, 1.3 million is not a lot
00:02:45.620 | compared to all the people in the world that write code.
00:02:49.820 | And there's actually a way to get an even finer point
00:02:52.500 | on that number.
00:02:53.260 | GitHub released a study saying,
00:02:55.700 | if you look at the fine print, that in this time range this year,
00:02:59.980 | 935,000 devs received a suggestion, not even accepted it.
00:03:04.580 | So there's going to be some that saw that little ghost text
00:03:07.240 | in their editor, never actually accepted it.
00:03:09.440 | And also, that's yearly active users.
00:03:11.380 | I don't know of any companies that go and cite yearly active users.
00:03:14.560 | Surely the monthly active users and daily active user figures
00:03:18.280 | are much lower than that.
00:03:20.460 | So these numbers are pretty--
00:03:22.980 | you can fill a city with these people.
00:03:25.320 | But it's not that many people when you compare it to the 26
00:03:28.060 | million professional developers in the world.
00:03:31.920 | And I think this is an undercount.
00:03:33.640 | I will say that, how do you get this number?
00:03:35.200 | I googled a bunch of different stats,
00:03:36.860 | and they all were around this.
00:03:38.800 | So you know, that's the vibes citation there.
00:03:41.540 | There's a lot more people that touch code in some way or another.
00:03:44.800 | I-- again, it's really hard to get a number on this, but I'd say
00:03:46.980 | probably 100, 150 million people in the world touch code in some way.
00:03:51.560 | And that includes students and other people and so on.
00:03:54.580 | So, you know, 935K, best case, what is that, an MAU?
00:03:59.400 | That's tiny compared to the number of people that interact with code.
00:04:02.900 | So, this is a scary thing.
00:04:04.640 | My best case estimate is that around 5% of professional developers use code AI.
00:04:09.500 | These are people that are paid a lot of money to write code.
00:04:13.000 | And they do it all day, every single day.
00:04:15.300 | And all of us here in the room, we know that there's this amazing new alien technology
00:04:19.820 | that's dropped on Earth, and that it changes how we code.
00:04:22.760 | And 5% are using it.
00:04:24.880 | That's crazy.
00:04:25.920 | That means in a room of 20 people, 19 are not using this.
00:04:29.100 | It just absolutely boggles my mind.
00:04:31.260 | And that's, you know, a huge problem.
00:04:33.000 | And it's an even bigger problem, because if you look at what they're actually using,
00:04:35.920 | the vast majority of code AI use is just autocomplete.
00:04:39.320 | It's the ghost text.
00:04:40.420 | I mean, yeah, that's good.
00:04:41.600 | But anyone who has used anything beyond that knows that AI can do so much more when coding.
00:04:46.860 | It can write entire files, it can answer questions, it can fix bugs, and all of that stuff.
00:04:51.280 | So this usage is absolutely tiny.
00:04:53.360 | And we need to realize that we here in this room, we are all the freaks.
00:04:58.180 | And for us to be successful requires us to change a lot of minds out there in the world.
00:05:04.020 | So how do we change those minds?
00:05:06.580 | Well, you know, we've been building Kodi.
00:05:08.880 | Kodi is a code AI tool.
00:05:10.480 | It does all of these things.
00:05:12.080 | And so we encounter a lot of reasons why people are hesitant at first.
00:05:15.280 | And I've taken the reasons from our own internal tools and ranked them.
00:05:19.800 | Why don't most devs use code AI?
00:05:21.480 | The first one is just, nah, they don't have a good reason.
00:05:25.360 | There are some people in the world, believe it or not, that have not used ChatGPT,
00:05:29.180 | that have not heard about it, even devs.
00:05:31.420 | There's a lot of devs who are kind of grumpy, and they say, well, it's not perfect, I tried it,
00:05:36.280 | and it gave me a wrong answer.
00:05:37.860 | Yeah, no shit, but there's a lot of other people that have figured out how to get the right answer out of it.
00:05:42.880 | A lot of people say, I don't need it, it didn't help me that much.
00:05:46.760 | You know, again, you've got to figure out how to make it useful for you.
00:05:50.280 | A lot of people's company hasn't adopted it, it's too expensive, or, you know, one thing we're seeing a lot less of now is these security, privacy, and legal concerns.
00:05:57.440 | I think it's a situation where enough people are using it so that if it's illegal to use code AI, then we're all fucked, so it's mutually assured destruction.
00:06:06.720 | But there's a lot of reasons why people don't use it yet.
00:06:11.620 | And this should really scare us.
00:06:13.980 | Because there's a lot of companies that are relying on a lot of people at a lot of enterprises using code AI.
00:06:22.580 | There's a lot of people's retirement accounts, or option trading accounts, or whatever, that are also relying on that.
00:06:28.640 | And if you think about how technology makes money, I mean, you know, the money that gets deposited into your bank account, your paycheck,
00:06:36.080 | where do those dollars come from?
00:06:37.580 | It takes, you know, kind of a long route.
00:06:40.380 | Just to explore the market a little bit here, you know, you've got these foundation model companies like OpenAI and Anthropic at the top.
00:06:48.380 | And then you have the AI infra companies.
00:06:52.480 | And the AI infra companies, you know, they will send, if they're serving and doing inference, they'll send some of the money back to the foundation model companies.
00:07:00.180 | In some cases, they're the same companies.
00:07:01.580 | And, you know, what drives usage of that?
00:07:04.580 | You have some experimental usage.
00:07:06.080 | People being the looky-loos, going and trying it out.
00:07:08.380 | That will turn into some experimental revenue.
00:07:10.680 | But all of that can happen without any real, actual usage.
00:07:15.080 | But that gives the perception of, holy shit, this stuff is hot, this stuff is working.
00:07:19.180 | And then you start to get some real usage.
00:07:21.180 | Some people using it, you know, maybe paying 20 bucks a month on their credit card.
00:07:25.880 | You get some devs using it at work.
00:07:27.880 | And then the holy grail, the way that software makes money, is from enterprises where there's recurring revenue from real usage.
00:07:33.180 | That first-year contract, it doesn't matter, it's got to be JP Morgan pays you a million dollars the first year, and then they renew for two million dollars the next year.
00:07:33.180 | And all of the money in our paychecks ultimately comes from that.
00:07:38.780 | If you're working at an AI infra company, if you've raised money to go and sell some AI product, it's ultimately because someone out there thinks that five years from now, JP Morgan is going to be paying you two million dollars a year.
00:07:44.280 | So that's the only thing that matters.
00:07:57.380 | Everything else is downstream of that.
00:07:58.380 | And it turns out there's just not that much of that going on.
00:08:01.380 | And also, this is an even more lopsided pyramid than I've depicted here.
00:08:04.480 | We got NVIDIA and chip makers at the top and so much investment riding on it.
00:08:10.480 | So, you know, this is where we ultimately need to get to.
00:08:13.480 | We need this person to be using our software.
00:08:15.480 | And maybe it's not the coolest thing out there, but that is the reality of how technology makes money.
00:08:20.480 | And I want to put some numbers to this.
00:08:22.480 | I estimate, based on my internal information, that it's not the coolest thing out there.
00:08:28.480 | I estimate, based on my internal information, some of the information that GitHub has shared at this conference, that the total recurring revenue from code AI usage is around 300 million ARR.
00:08:41.280 | Now, look, that's a lot of money.
00:08:42.580 | If you had a company that was doing that, then it could go public on its own.
00:08:45.780 | So that's good.
00:08:47.080 | But how much of that money actually goes to these other companies?
00:08:51.380 | Not that much.
00:08:52.880 | From our own data at Sourcegraph, where we spend a lot of money on AI inference, we spend less than 10% of our revenue.
00:09:01.680 | So if you take that number -- and by the way, we're not even doing it in that optimized of a fashion.
00:09:05.680 | If you take that 300 million number, 10% of it, that's $30 million going back up to these foundation model companies and AI infra companies.
00:09:13.680 | It's a long way from where we are today, that amount of revenue, to them making the kind of revenue that's going to justify these massive multi-billion dollar valuations.
00:09:22.480 | Also, another way to look at it is Salesforce.
00:09:25.280 | Salesforce's annual revenue is $36 billion.
00:09:29.280 | All of this stuff, all this hype, all this usage of code AI, it's amounting to a tiny, tiny fraction, 1/120th of Salesforce.
00:09:38.280 | So, we have a long way to go.
00:09:42.080 | Usage needs to grow a lot.
00:09:43.280 | Or, I could be wrong, and maybe the doomers are right, maybe not for the reason they thought.
00:09:48.080 | Maybe we should never have done this, and this is all just going to be one massive hype bubble, and it's going to pop, and we're all going to be miserable.
00:09:53.280 | But I think, because I use code AI every day, obviously, and you do too, this stuff is real.
00:09:58.880 | It's just, it's really early.
00:10:00.480 | And usage needs to grow a lot.
00:10:02.680 | So, if you're building a product, keep that in mind.
00:10:05.480 | This is what I'm betting on.
00:10:07.880 | And let's talk about some of the lessons from our experience building Codi at Sourcegraph.
00:10:12.680 | I wish I had this mindset going into it.
00:10:15.280 | Just quick background to establish, you know, why would you even listen to me?
00:10:18.880 | Maybe I should have done this in the first slide, but, hey, here we are.
00:10:23.080 | Sourcegraph has been around, as Brittany said, for 10 years.
00:10:25.680 | We started out with code search, and then we found out that if we, we had this tool that all the devs used,
00:10:32.280 | and had all the code in a company, and it turns out that was amazing context to build a code AI on top of.
00:10:38.080 | And so, we took that, and we built Codi.
00:10:40.280 | It's got really great autocomplete and inline edits, generating unit tests, and chat.
00:10:45.880 | And chat is where we really differentiate, because that's where you can make the best use of context.
00:10:49.680 | So, you can figure out why is this broken, or how do I change this, or where should I start on this?
00:10:54.880 | That's where we excel.
00:10:56.880 | We've got a lot of really big customers.
00:10:59.480 | We have four of the top five biggest banks in the country.
00:11:01.480 | We have most of the FAANG or Mañana companies, or whatever, and a lot of other great customers, including some that are presenting today.
00:11:09.280 | And we are the number two code AI company in revenue, second only, of course, to GitHub Copilot.
00:11:15.480 | But because we're number two, and because we're a startup, and we're scrappy, we try harder.
00:11:21.080 | All right, so what have we learned?
00:11:24.680 | One is that hype fools everyone.
00:11:26.680 | Hype fools us.
00:11:27.680 | Hype fools all of you.
00:11:29.080 | Hype even fools your customers.
00:11:31.680 | And the second thing is that AI code completion, the thing that autocompletes the rest of the line, or the next few lines,
00:11:38.680 | that's like the freakish kind of feature that comes along like one time in a hundred years.
00:11:43.280 | And we kind of got spoiled.
00:11:46.280 | It is so perfect.
00:11:47.880 | So I'll get to that.
00:11:48.880 | But first, just, you know, on the hype, how do you get away from being fooled by the hype?
00:11:54.280 | I'll share a few tips that have worked for us.
00:11:57.280 | The first, and this is the most important thing, if you're building a product, and you are not using it every single day,
00:12:01.680 | it is not going to make it.
00:12:02.880 | There is zero hope.
00:12:03.880 | If you're building it on a team, and the people all building it don't use it every day,
00:12:08.480 | your customers will not.
00:12:10.280 | But they might say it's really awesome, and that's the dangerous thing about this hype.
00:12:14.280 | Customers don't know what they want.
00:12:16.280 | Every customer for about a year said, I want fine-tuned models.
00:12:20.280 | They said that all the time.
00:12:22.680 | Our salespeople would say, hey, we need to build fine-tuned models.
00:12:25.280 | Everyone was talking about it.
00:12:26.880 | GitHub Copilot has it as a bullet.
00:12:28.480 | You know, all of our competitors have it as a bullet that's like italic, and it's like coming soon.
00:12:31.880 | It's some bullshit thing.
00:12:32.880 | Really, what they just want is they want it to work well, and they use some of these terms.
00:12:37.280 | So don't listen to them, because what they're saying is probably downstream of them sitting
00:12:42.080 | at, you know, some conference like this and getting some cool ideas.
00:12:44.080 | They want the product to work.
00:12:46.080 | And if you can't describe why it works and why it's great without using the word AI, then
00:12:50.080 | you're probably not going to make it.
00:12:53.080 | So this is a really existential question.
00:12:54.880 | So we make a product that makes developers a lot more productive, like 20% or 30% more productive.
00:12:59.880 | If it's so damn good, why are we selling a product?
00:13:02.880 | It's like the people that have stock picks and sell newsletters.
00:13:05.880 | Why don't we go and buy a software outsourcing firm and prove it and monetize our product by actually
00:13:11.880 | capturing that value directly?
00:13:12.880 | That's a damn good question, and I wish that we had done that a year ago, and we're looking
00:13:17.680 | at doing that now.
00:13:18.680 | And if you don't have the confidence to go and do that, to put a few million bucks behind
00:13:23.680 | that if you've raised a bunch of money, then I'd say your product's probably not ready yet.
00:13:28.680 | And then finally, I think we've all seen this.
00:13:30.480 | This is actually not even one of the worst ones.
00:13:32.480 | This is a tweet that's got a lot of activity, right?
00:13:36.480 | Just keep in mind that that does not translate into DAU, so don't feel shitty about yourself
00:13:40.880 | all the time.
00:13:41.880 | I wish that Elon would add something like this that would actually tell you the DAU of the products
00:13:46.180 | that are getting all this hype.
00:13:47.480 | But until he does that, you can imagine it.
00:13:51.380 | All right, I talked about autocomplete being this freakishly good feature.
00:13:55.480 | So here's how we think about features at Sourcegraph, this kind of for box.
00:13:59.680 | You want to be in the top right, and that's where it's an AI feature that doesn't take a ton of time
00:14:04.880 | to see is that correct, and it's really easy, and it's used often.
00:14:09.380 | AI autocomplete is great because it's literally every keystroke, and you can glance at it in a few seconds
00:14:15.180 | or even milliseconds, 100 milliseconds, you can figure out, is this correct?
00:14:18.680 | So that's a really good property of a feature.
00:14:22.380 | And it just so turned out that the first feature of code AI happened to be smack dab in the top right.
00:14:27.580 | It's pretty amazing.
00:14:28.580 | But most other code AI features do not have this level of product market fit, and we need to realize that.
00:14:33.780 | Now, we can work on it, but you know, where's chat?
00:14:37.780 | Not used nearly as often, and it's harder to vet that long response edits, you know, similar.
00:14:43.580 | I know a lot of people love doing the inline edits like, you know, Meta K, Option K, and Kodi and Cursor and things like that.
00:14:49.780 | It's great, but we've got to be prepared for what actually works being a very different form from what exists today.
00:14:55.980 | Then there's a lot of other stuff, like the agentic stuff.
00:14:58.580 | I mean, look, it's obviously the future, but it's just not there yet.
00:15:01.780 | Who here has used a code AI agent to actually merge a PR in the last week?
00:15:08.980 | All right, cool.
00:15:09.980 | Thank you for helping us, you know, push the world forward, but it's not there yet.
00:15:14.980 | And, you know, other features.
00:15:16.980 | You really got to go for that top right and drive any feature you make to the top right.
00:15:20.980 | And if not, it's just not going to make it.
00:15:23.980 | So what are we doing at Sourcegraph to address this?
00:15:28.180 | Well, we are searching for that next great code AI modality.
00:15:31.180 | What's the next autocomplete?
00:15:33.180 | There's some ideas like the next edit suggestion or like Copilot++.
00:15:36.980 | You know, I'm skeptical of that.
00:15:38.980 | Chat.
00:15:39.980 | Steve Yegge, who is at Sourcegraph, he just wrote this great blog post about chat-oriented programming.
00:15:44.980 | Chop, where it turns out there's a lot of devs who have a chat session running in like Kodi or even ChatGPT or something,
00:15:51.980 | like all the time while they're coding, and it's just one ongoing conversation.
00:15:55.180 | It's a totally new way of coding.
00:15:57.180 | And what's weird is it's like the Gen Z's who do it, and it's like the 50-year-old, you know, disgruntled programmers who do it.
00:16:03.180 | So I love that it's got kind of, you know, some usage in both generations.
00:16:07.180 | There's new ways that people can be using code AI that we have not even thought about, and it's so early to the previous point.
00:16:14.180 | We always got to remind ourselves to build the manual and explicit thing first.
00:16:18.180 | We got spoiled by autocomplete.
00:16:19.580 | It's automatic, and, you know, it just like triggers every keystroke.
00:16:24.780 | No.
00:16:25.780 | First, you've got to make something work in manual and explicit mode.
00:16:28.780 | If you have chat, you've got to make it so people manually at mention the context they want
00:16:32.780 | before magically inserting the context.
00:16:34.780 | If you've got an agent, put it in the editor and make it work in the editor so that if it's wrong,
00:16:39.780 | the dev can just change it right in their editor instead of having to go into code spaces or some other totally different UI.
00:16:45.980 | So make it manual and explicit first and really easy for the dev to go and fix it.
00:16:51.180 | And then you can add the magic.
00:16:52.180 | Probably, you know, you're going to be bogged down enough with stuff to make it a great product,
00:16:57.180 | and you won't actually get to adding the magic for a while.
00:17:00.180 | So, you've got to remove the hype.
00:17:10.380 | So, you've got to remove the hype.
00:17:11.380 | Do this so aggressively, and here's an example from Sam Altman.
00:17:16.580 | This is a tweet, I don't know, like a year ago when ChatGPD had probably tens of millions of DAU.
00:17:22.580 | So, he's right.
00:17:24.180 | We know he's right, and if he's saying this and his product has way more validation than your product,
00:17:28.580 | well, you probably should be saying it like ten times more and ten times more intensely.
00:17:31.580 | Also, it's kind of cool marketing, and it, you know, makes you seem cool.
00:17:36.580 | Yeah, you've got to become, here, it's covered up.
00:17:42.180 | You've got to become a DAU yourself, or kill the product.
00:17:45.180 | And then the last thing is just that, as an ecosystem, all these foundation model companies,
00:17:51.180 | all these AI infra companies, all these AI applications, we all live or die together.
00:17:55.580 | Anthropic and OpenAI and Mixtrel and Fireworks and all these great companies that we use
00:18:00.180 | are not going to get paid if we're not in business making a shit ton of money on coding.
00:18:04.380 | So, we all need to work together and realize that we all have so much to benefit
00:18:09.580 | from this stuff actually being used, from turning down the hype, and building great products
00:18:13.580 | that get actual devs using it all the time.
00:18:16.580 | And that starts with you, if you're the one building the product.
00:18:18.980 | So, thank you, and if you want to reach out to me, there's my contact info, and happy coding.
00:18:25.980 | We'll see you next time.
00:18:42.180 | We'll see you next time.
00:18:43.180 | Bye.