back to indexCopilots Everywhere: Thomas Dohmke and Eugene Yan

00:00:00.000 |
I'm delighted and honored to welcome a very special guest, Thomas Domke. 00:00:18.360 |
Thomas has been fascinated by software development since his childhood in Germany, 00:00:23.260 |
and he's built a career building tools, dev love, and accelerating innovations 00:00:32.840 |
where he has overseen the launch of the first at-scale AI developer tool, GitHub Copilot. 00:00:39.240 |
So please join me in welcoming to the stage, Thomas Domke. 00:01:06.340 |
Many people have shared their own takes on the Copilot origin story. 00:01:11.480 |
So, but what was your personal experience seeing it in GitHub? 00:01:20.420 |
It's lockdown here in San Francisco, in Seattle, everywhere where GitHub engineers are sitting. 00:01:25.640 |
So, like all of you probably were on a Zoom call. 00:01:29.100 |
One of us had early access to a new model that OpenAI had just released in preview, 00:01:38.580 |
And, you know, one had, Uge, I think, had the keyboard, the leader of GitHub Next at the time. 00:01:44.240 |
And we were dictating prompts and asked the model to write some code. 00:01:49.460 |
And I think the first aha moment that I had is that you could ask it to write JavaScript code 00:01:54.460 |
and put the curly braces in the right places and whatnot. 00:01:57.320 |
And you could write it, ask it to write Python code. 00:01:59.720 |
And the model, in a way, you know, it doesn't work like a compiler. 00:02:06.540 |
Or you could also argue it knows them exactly like we'd know it. 00:02:09.560 |
So, that was probably the first moment we kept building, we kept exploring the model 00:02:15.100 |
and then decided we built this autocompletion copilot that, you know, was the first copilot. 00:02:19.100 |
And we built it all, you know, while being remote, while being on lockdown. 00:02:23.240 |
So, if your investors tell you today you need to be in a room and in front of a whiteboard, 00:02:29.800 |
you can innovate if you want to while being in your home offices around the world. 00:02:35.560 |
I think the next moment was that we shipped a preview to our internal engineers, 00:02:43.600 |
And the NPS survey with those engineers was through the roof, I think, 72. 00:02:50.120 |
And typically, our early stage products, especially, you know, with the large language model 00:02:53.880 |
and, you know, all the hallucinations and the UI wasn't really figured out yet, it's much lower. 00:02:59.540 |
So, that was kind of like a holy shit moment that we had. 00:03:03.300 |
And as the product then shipped in mid-2021, and, you know, COVID was still going, 00:03:08.580 |
we started looking at telemetry, and the team came and says it writes about 25% of the code 00:03:22.820 |
And it turned out, you know, that was actually right. 00:03:25.300 |
And by now, it's about half, you know, the code that's written. 00:03:29.100 |
Some languages like Java even have a higher acceptance rate and more lines written. 00:03:33.160 |
And so, I think those kind of, this journey that we went through over the first two years 00:03:37.940 |
really was like one moment after another where we saw the future of AI long before ChatGPT 00:03:47.380 |
And now it's available to everyone here as well. 00:03:49.180 |
So, I think Copilot started as an autocomplete IDE, and now it's all over GitHub. 00:03:56.040 |
What do you do to make Copilot and integrate it across all of GitHub? 00:04:03.840 |
I think the first thing is to think about, you know, what do I do as a leader, as the CEO 00:04:09.240 |
And it's really about constantly reconfiguring our approach. 00:04:12.920 |
So much of, you know, the AI world is changing almost daily. 00:04:17.140 |
And there's, you know, some news on the information elsewhere every morning. 00:04:24.440 |
I have my features all laid out, and it's worked through the backlog. 00:04:27.840 |
It's really like operating as agile as possible, even as we are, you know, a 3,000-person company 00:04:33.140 |
as part of, you know, one of the largest companies on the planet. 00:04:36.840 |
The second is that we really try to meet, you know, the developer where they are. 00:04:41.800 |
We say, you know, we're not trying to build an AI engineer. 00:04:45.120 |
We are trying to build AI for engineers, a human-centric approach. 00:04:48.680 |
You know, that's where the name, what the name Copilot ultimately visualizes. 00:04:53.420 |
But also, you know, we're trying to make the developer's lives better because we are developers 00:04:58.760 |
ourselves, and every productivity improvement we can find ultimately helps us at GitHub to 00:05:05.960 |
So, that really is the approach, like, looking at what can we do next to make, you know, 00:05:10.840 |
our work a little bit easier of building more features for Copilot. 00:05:15.700 |
You're trying to meet the developer where they are. 00:05:17.340 |
So, for now, we've been bringing the AI to the IDE. 00:05:21.920 |
Are we going to try to bring the developer, the IDE, closer to the AI? 00:05:27.300 |
You know, the idea of bringing AI into the IDE, or really into ghost text, you know, auto-completions, 00:05:36.500 |
It was a way of saying, okay, the model's not always going to be perfect, but neither are 00:05:42.020 |
Like, whether you have auto-completions in your Google Docs, or in your email, or in your 00:05:45.840 |
editor, in the old IntelliSense way, as you're typing, it cannot know what you want it to type. 00:05:51.260 |
And so, you're used to adjusting your typing, and then you find this moment when you press 00:05:56.320 |
And even without auto-completions, if you're thinking about what developers do in the editor, 00:06:00.920 |
while they write code, and the best developers write a lot of code before they get stuck, 00:06:04.540 |
and the newbies, and those that rarely write code like I, you know, get stuck more often. 00:06:09.820 |
And then you, you know, control-tap or command-tap into your browser, and you open Google or Stack 00:06:19.760 |
And what you do there is you find code, and you argue with other developers, and then you 00:06:23.780 |
copy and paste that code into your editor, and you modify that as well. 00:06:27.920 |
So, it's kind of like, in a way, Stack Overflow has as many hallucinations as the model that might 00:06:33.720 |
have, and not because the answers are bad, but because the world is changing so much. 00:06:38.040 |
You know, I code a little bit on iPhone projects in Swift, and there's always a new Swift version 00:06:42.700 |
after WWDC, or a new Xcode version, and so things have changed of how you use the APIs. 00:06:49.840 |
That really, the crucial thing here was, we didn't, you know, in the world, you know, 10 years 00:06:56.720 |
We would just call it, you know, smarter auto-completion. 00:07:02.900 |
The core feature of Copilot is that it helps developers to stay in the flow to get the job 00:07:08.680 |
done, and not be in this constant distraction between the editor and the browser. 00:07:13.900 |
And I think, a few months ago, you wrote this post about workspace. 00:07:22.020 |
And maybe for folks who are not unfamiliar with it, what is workspace? 00:07:25.940 |
Yeah, so, you know, you already mentioned auto-completion. 00:07:33.860 |
So, early 2023, we had a chat and GPT-4 to copilot in the IDE as a separate sidebar window. 00:07:43.520 |
And it has reg and all the information, the context available in the IDE. 00:07:47.520 |
But ever since, we've been thinking, how can we make the developer flow even easier? 00:07:52.960 |
It takes a GitHub issue or just a task, an idea that you write down on GitHub.com. 00:07:58.100 |
And it helps you then, as part of your code-based repositories, to figure out how to implement 00:08:03.000 |
It bridges from the issue, you know, from the task description into the pull request, into 00:08:09.300 |
And the magic behind this is that, A, the human is still in the center. 00:08:12.080 |
So, every step of that way, you know, writing a specification, analyzing the current repo, 00:08:17.320 |
the current behavior, and then using your description to figure out how do you modify this. 00:08:21.080 |
And then writing the plan, which shows you how to change all the files to the implementation, 00:08:28.080 |
The human can interact, can change those bullet points, can change the code. 00:08:32.080 |
And what that really does is it gives you a pair programmer that helps you to explore 00:08:39.800 |
Because the challenge we all have as engineers is that as soon as you get moved onto a new 00:08:44.840 |
project or you want to, you know, modify an open source project or you're just, you know, 00:08:49.580 |
coming back from vacation, you're trying to remember what is implemented where in your 00:08:53.680 |
thousand-plus files, that is navigating the code base is the first challenge you have, 00:08:58.080 |
figuring out what's the current behavior and what's the new behavior. 00:09:01.180 |
So, you're having an AI native, a co-pilot native developer environment that helps you 00:09:05.460 |
along that journey that you're naturally also doing in your IDE. 00:09:10.060 |
It's not about, you know, building an autonomous agent. 00:09:13.180 |
I'm sure you've heard a lot about that in the last three days. 00:09:15.880 |
It's about building agents that helps us as humans to solve a task and learn along the 00:09:21.000 |
way as we figure out, oh, you know, there's this test file that I also have to modify if 00:09:26.340 |
I love the point that you mentioned, which is not building autonomous agents and also helping 00:09:31.280 |
So, how should none developers use Workspace? 00:09:35.560 |
And in fact, you know, once we announced this last year at GitHub Universe in November, I 00:09:40.560 |
think the first email we got with feedback was from a program manager or product manager saying, 00:09:44.560 |
"This is awesome because now I can not only write, you know, a user story or a work item, 00:09:49.560 |
I can also see what that would mean to implement in the code base. 00:09:53.840 |
And in many ways, you know, the biggest challenge we have today is can we be as specific as 00:09:58.120 |
possible when we write down a task, you know, as product managers or as engineers ourselves. 00:10:02.120 |
You know, often everything is obvious until it is not. 00:10:06.400 |
And then, you know, you kind of need to size the task, right? 00:10:11.400 |
Like, how long will it take and the mythical man month? 00:10:14.680 |
I think the pragmatic engineer had that a couple of weeks ago is still true. 00:10:18.640 |
Most estimates are half the time that the job actually takes. 00:10:23.180 |
And so, we're really bad at estimating how much time it takes to get work done, whether 00:10:27.720 |
it's encoding or whether it's building houses or roads or infrastructure. 00:10:31.620 |
And so, Workspace helps you with that as it helps you to figure out what I just described. 00:10:36.020 |
Is it actually specific enough to write the code for that or to even figure out what the plan 00:10:41.720 |
Can you share a bit about your vision on how you think we will build and code in natural 00:10:47.900 |
language and how it will help us collaborate better, devs and PMs, coders and non-coders, 00:10:56.900 |
For me, you know, the very first thing when you say natural language is, and I have it on 00:11:00.940 |
my t-shirt here, Copilot Speaks Your Language, is because chat, these large language models that 00:11:05.200 |
we're using today in GitHub Copilot and many other AI applications are the same models that 00:11:11.560 |
They speak almost any language, any major human language. 00:11:15.460 |
And so, whether you want to explore coding in English and you don't understand the concepts 00:11:20.040 |
of true/false, Boolean logic yet, or whether you want to learn that in German, in Hindi, 00:11:25.560 |
in, you know, Brazilian Portuguese, in Spanish and Chinese, you can do that now. 00:11:29.760 |
And if I, you know, look at kids today in school, most of them are growing up with mobile phones. 00:11:35.840 |
You know, when you go into a restaurant here in San Francisco or in Seattle or elsewhere in the world, 00:11:40.360 |
at night, you probably see a family with little kids where the kids have their phone because the parents want to enjoy five minutes on their own. 00:11:46.260 |
And then as then kids grow up, you know, they see Super Mario or Minecraft and they get into gaming and that naturally, 00:11:55.960 |
Copilot enables that and it enables that in the language that the kids grow up with, which, you know, for the majority of the humans 00:12:02.060 |
of the, on the, on the, on this planet is not English. 00:12:07.760 |
It also democratizes access for those that don't have parents at home that have a technical background 00:12:12.360 |
or that don't have parents at home that have infinite patience, but most parents do not. 00:12:18.860 |
At some point you're just done, you know, with explaining the world to your kids and you just want to, you know, 00:12:22.260 |
switch on the TV and, and watch a, watch a Netflix show. 00:12:28.060 |
If you look into the professional context, one of the biggest challenge we have is, you know, 00:12:31.260 |
if you would join my company or I join your company tomorrow, 00:12:35.260 |
the biggest challenge we have is what's all the institutional knowledge? 00:12:39.260 |
You know, and what we don't like as humans is ask a thousand questions. 00:12:43.160 |
Especially if you're a new employee in a, in a big company, you're like having this anxiety in your head 00:12:48.360 |
that everybody else thinks you are, you are dumb. 00:12:50.260 |
You know, why did you get hired in the first place? 00:12:52.060 |
So a copilot also democratizes access to all the information and companies. 00:12:55.960 |
And I think that is going to be changing how we work and not only for developers in the workforce, 00:13:05.160 |
I guess the next thing I want to ask is maybe a little bit more unhinged. 00:13:08.660 |
Speaking of agents, in your opinion, what makes an agent or a copilot? 00:13:14.660 |
What's the definition, what, what's your definition of an agent? 00:13:18.260 |
I think an agent, you know, is like an AI dishwasher. 00:13:22.060 |
You fill it with, you know, the dishes and you let it, let it do its thing. 00:13:26.060 |
And then at the end, just to take the output and you put it back into the shelves. 00:13:30.260 |
And today we have, you know, we called it, used to call it bots, you know, or CI/CD. 00:13:37.260 |
And in many ways that's an autonomous agent, right? 00:13:38.760 |
You push your pull request and you run your CI/CD GitHub Actions or a similar product. 00:13:44.060 |
And many compute primitives that we have today are agents as they get a job done on their own. 00:13:50.760 |
My monitoring, you know, to figure out if GitHub up or down is somewhat autonomous. 00:13:54.760 |
Hopefully pages somebody without us hearing from you that you cannot access your repository. 00:13:59.760 |
So, you know, I think in many ways what we are building is still tools that help us to get the job done. 00:14:06.260 |
And there's many jobs that developers have to get done. 00:14:08.260 |
Many jobs that now AI engineers need to get done. 00:14:12.260 |
All the things that are also still true, you know, even though you can automate things with large language models. 00:14:17.260 |
And a lot of work in software engineering is bogging us down. 00:14:22.260 |
A lot of boilerplate, a lot of security compliance. 00:14:25.760 |
You know, that Friday evening when you want to, you know, enjoy the barbecue because the sun is out and instead you have to update all your log4j dependencies, right? 00:14:33.760 |
Like security tooling, in fact, you know, is creating more work. 00:14:37.760 |
It's actually a tool that tells you that the dishes are dirty and then you have to do the dishes yourself today. 00:14:45.760 |
It just adds stuff to our backlog while we actually want to work on the creative side and we want to build new features. 00:14:51.760 |
You want to build innovative product that are creative things. 00:14:54.260 |
And I think many software developers do not understand themselves as a production worker. 00:14:58.260 |
They understand themselves as artists, as creators. 00:15:01.260 |
But, you know, our companies, our governments, you know, the world is requiring us to do a lot of other work. 00:15:06.260 |
And then we need AI tools, auto-fix, you know, things that scans not only for security issues but then fixes those security issues. 00:15:14.760 |
We need those pieces supported by AI so we have more time for the things we do want to do and AI takes over the things we don't want to do. 00:15:29.260 |
I mean, I want to have these agents that burns down all my security backlog. 00:15:33.260 |
As in any company, the challenge is that I have way too many of these items and there isn't really a book you can buy that tells you as an engineering manager of how to balance those two things. 00:15:45.760 |
You cannot do all your work into security compliance, accessibility and whatnot and you cannot put all your work into innovation because your customers will lose all your trust the moment you have a security issue that threatens their data. 00:15:59.260 |
And as such, you have to balance those two things or you find AI agents that brings the work down. 00:16:04.260 |
And I think as any, you know, leader of a software development company, I always want to go faster. 00:16:09.260 |
I always want to get that feature done faster and I'm sure it's, you know, the same for you folks at Amazon. 00:16:15.260 |
When I have an idea and I ask my folks how long will it take to implement that, the estimate I'm getting is like I'm scratching my head and I'm thinking I can do that myself faster than waiting for my team to do it. 00:16:28.760 |
The truth is that there's so many other things in the process these days that we need to find new abstraction layers that help us to get control over our development lifecycle again. 00:16:41.760 |
Do you have any advice for devs, both new and experienced, on how they should navigate this new world of tools, this new world of abstractions, in what some say is the biggest technology innovation since the internet? 00:16:56.760 |
I think, you know, the most exciting thing about this new technology is, and you saw it hopefully over the last three days at this conference, is that we are moving into a new world of software development. 00:17:09.760 |
And there have been multiple step functions, you know, over my life. 00:17:16.760 |
I remember getting my Commodore 64 on a PC in the 90s. 00:17:22.760 |
And, you know, internet open source before the internet was buying CDs and DVDs in bookstores. 00:17:27.760 |
The internet came, you know, SourceForge, and then GitHub came. 00:17:30.760 |
All of a sudden, developers started collaborating. 00:17:33.760 |
And every time we had those step functions, software development got more exciting. 00:17:37.760 |
And I think, you know, we are again at that step function. 00:17:45.760 |
And I think, you know, it's really like, you know, for me as the CEO of GitHub, I don't get to touch code often. 00:17:52.760 |
And so when I get to touch code on a Sunday afternoon, I don't want to spend all my time of updating all my dependencies. 00:18:16.760 |
I'll be back home and we'll be back home and we'll be back home.