back to index

The AI Evolution: Mario Rodriguez, GitHub


Whisper Transcript | Transcript Only Page

00:00:00.000 | I don't know about you, but I've been having a blast at this conference.
00:00:18.000 | I've been enjoying a lot of great conversations about how to build AI products and how to kind of advance humanity forward at the same time.
00:00:26.000 | This show is hard, like it's not easy to do.
00:00:30.000 | So I really enjoy kind of being part of this community and kind of sharing kind of the best practices, the failures and everything else.
00:00:37.000 | I also want to give a shout out to Ben and Swix.
00:00:40.000 | Let's give it to them for an amazing day one.
00:00:45.000 | I will tell you one thing, putting together conferences is also not an easy job.
00:00:49.000 | It's kind of a lot of hard work and a lot of people end up criticizing and things go wrong.
00:00:54.000 | It's kind of like a wedding at times, you know?
00:00:57.000 | But they have not been or anything like that.
00:01:00.000 | Okay.
00:01:01.000 | So who am I?
00:01:02.000 | My name is Mario Rodriguez and I am VP of product at GitHub.
00:01:06.000 | And I oversee what is called our productivity suites.
00:01:10.000 | That includes repositories, so where you store your code, our pull request product issues, projects, our mobile app, our CLI.
00:01:20.000 | And then kind of since January of this year, I had the privilege of also leading the co-pilot team and the AI strategy at GitHub.
00:01:29.000 | I'm not one of those that invented co-pilot and put it on Twitter or anything like that.
00:01:33.000 | But I am kind of overseeing what that strategy is at the moment and thankful for everyone that made it happen before.
00:01:42.000 | So I actually want to start with a little bit of what I call insider info.
00:01:47.000 | Like if you were not at GitHub at that moment, you would not have known this.
00:01:51.000 | But I would say a catalyst, at least in my opinion, for co-pilot's existence came around August 2020.
00:01:59.000 | And there was a paper circulated at GitHub.
00:02:02.000 | Uga, Albur, and a couple of other people, probably more than a couple of other people, ended up writing that paper.
00:02:09.000 | And it was called this: An Automated AI Programmer: Fact or Fission?
00:02:14.000 | Again, this is 2020.
00:02:16.000 | This is before the hype of ChatGPT and everything else that you know about.
00:02:20.000 | This is before the existence of being able to actually have this conference.
00:02:24.000 | And one of the things in my opinion that that paper had was what I call polarity.
00:02:31.000 | And polarity for me is not a choice because you can choose A and B.
00:02:35.000 | Polarity for me is if those choices are interconnected and related as well.
00:02:40.000 | So there were a lot of people that said that it was fiction.
00:02:43.000 | For example, is this fact or fiction?
00:02:45.000 | You probably will say it's fiction.
00:02:47.000 | If you ask my, you know, four-year-old or three-year-old, they'll tell you it's fact.
00:02:51.000 | They love this thing.
00:02:52.000 | They play with it and all of those type of things.
00:02:54.000 | It's kind of scary that the dinosaur is with the kids there, but it's a good dinosaur, right?
00:02:58.000 | So it kind of had this thing of, like, it's a fact or it's a fiction.
00:03:03.000 | And we went through a lot of conversations about what is this product really going to be like.
00:03:09.000 | And the product kind of failed at the beginning.
00:03:12.000 | We were trying it first in the PR flow, as an example, and it was not good in there.
00:03:17.000 | But then something happened, and we started kind of seeing traction.
00:03:21.000 | And that traction ended up with us shipping the product in 2021.
00:03:26.000 | And that was the first AI at-scale AI programmer in the world.
00:03:31.000 | And it has continued to grow since then.
00:03:34.000 | We were the first co-pilot before everything happened.
00:03:37.000 | And today, if you actually ask out there, we get used by over 20,000 organizations.
00:03:45.000 | There's one million-plus developers using the product as well.
00:03:49.000 | We'll release new stats very soon on that, too.
00:03:52.000 | We got a lot of studies of what we're doing and what's the impact that it's having.
00:03:57.000 | You see things like 46% of the code-written.
00:03:59.000 | That's really just tab-tab and kind of completions.
00:04:02.000 | Coppilot does a pretty good job at times, depending on the language, on multi-line completions overall.
00:04:07.000 | So we were the first one, and we kind of are extending that.
00:04:11.000 | So if I ask you -- you know, Coppilot is also a business.
00:04:15.000 | There was a recent report that, you know, we're losing money, and that's not true.
00:04:19.000 | But regardless of that -- so how much do you think of ARR Coppilot has today?
00:04:23.000 | And ARR is kind of like the annualized revenue that we have.
00:04:27.000 | So raise your hands if you think it's over 25 million in AR.
00:04:33.000 | Okay?
00:04:34.000 | Fair amount.
00:04:35.000 | I'm going to bring you to the board meetings.
00:04:38.000 | What about 50 million AR?
00:04:40.000 | Raise your hand if you think of that.
00:04:42.000 | Okay?
00:04:43.000 | Pretty good, too.
00:04:44.000 | So you're mostly right.
00:04:45.000 | Coppilot today is over 100 million AR product.
00:04:48.000 | That's what it is today.
00:04:49.000 | And can I give you the latest numbers?
00:04:51.000 | Because the SEC will go after me.
00:04:53.000 | But it's a pretty darn successful business.
00:04:56.000 | You know, we builders dream about building things that have that impact.
00:05:02.000 | Forget about actually even the money that it brings.
00:05:05.000 | Because at the end, we really actually are making Coppilot for you.
00:05:09.000 | And when I say for you, I mean the essence of GitHub is developers.
00:05:13.000 | It's AI engineers, you're a developer too, but you kind of specialize on that end.
00:05:18.000 | But the essence of us is developers.
00:05:20.000 | That's why we wake up every single day is to kind of bring you Coppilot so you could actually
00:05:25.000 | enjoy the work you do overall.
00:05:27.000 | And based on yesterday's conversation, maybe Lindy, Flo, Lindy and Copilot maybe can, you know,
00:05:33.000 | help at the end Milton as well.
00:05:36.000 | So, like I said, you know, we're building Copilot for the sake of developer happiness.
00:05:41.000 | You know, for me, the definition of success of Copilot is this.
00:05:46.000 | It's not lines of code accepted, although that's a pretty good metric overall.
00:05:51.000 | It is fundamentally, we are erasing all the boilerplate that you're creating, all the things
00:05:57.000 | that kind of don't keep you in the flow.
00:05:59.000 | And what I want you to experience of Copilot is the feeling of flow at the end.
00:06:04.000 | It's not only about conversational.
00:06:06.000 | Now we have inline chat.
00:06:07.000 | There's other modalities that will end up trying.
00:06:10.000 | The chat box is not all.
00:06:12.000 | You know, and in code completion kind of we have proven that as well.
00:06:15.000 | But, you know, this is what generates happy developers from -- in my end is kind of that
00:06:20.000 | flow.
00:06:21.000 | And happy developers write better software.
00:06:23.000 | You know, Milton doesn't write a lot of good software overall.
00:06:26.000 | All right.
00:06:27.000 | Let's keep it going.
00:06:28.000 | So, more insider information.
00:06:29.000 | Like, what made it successful?
00:06:30.000 | If you ask me, what made Copilot V1 successful?
00:06:34.000 | And that is, I would say, the code completion product that you use today.
00:06:37.000 | And there's four key components on it.
00:06:40.000 | Number one, it was the ghost text.
00:06:42.000 | Believe it or not, that completely changed the game when we were trying the codex model.
00:06:47.000 | And codex was the first model that shipped with Copilot.
00:06:51.000 | When we were trying that model, having the ability to actually generate either from natural
00:06:56.000 | language -- you know, you could just put a comment and have it generate -- or just normal
00:07:01.000 | as you're typing, having something in there to remind you, is this what you want or not.
00:07:05.000 | Having that in the UX was one of the key elements.
00:07:10.000 | The second thing is it had to be fast.
00:07:12.000 | It's not like -- you know, people don't like -- if I want to keep you in the flow and you're
00:07:15.000 | waiting seconds for that, that's not a good experience, right?
00:07:18.000 | So, what do we want?
00:07:19.000 | We want to make it fast.
00:07:20.000 | So, we worked a lot -- you know, just recently, by the way, we switched to GPT 3.5 turbo behind.
00:07:26.000 | So, we're no longer using codex.
00:07:29.000 | But, you know, codex is a very fast model.
00:07:31.000 | So, you end up with less than 100 milliseconds in latency many of the times.
00:07:35.000 | Then, the other thing is, you know, the codex model was like anything we have seen before.
00:07:40.000 | You know, hats off to open AI on that.
00:07:42.000 | But that model really changed the experience that we were having.
00:07:46.000 | We were always tinkering with AI at GitHub.
00:07:49.000 | But once we actually pair ghost text with low latency with an amazing model, then that's
00:07:55.000 | when the magic happened.
00:07:57.000 | And then, at the end, you know, there's people that say prompt engineering is not a true thing.
00:08:01.000 | At least from where I'm standing, prompt engineering is a very true thing.
00:08:04.000 | So, if you're really gifted at that, I have a job offer for you at the end of this.
00:08:09.000 | Because, like, it takes a long -- I could create a demo for you all probably within 30 seconds.
00:08:16.000 | Make an AI product that actually gets used by millions of people that you could sell out
00:08:21.000 | there to companies.
00:08:22.000 | That takes a lot of hard work.
00:08:24.000 | And it takes a lot of engineering.
00:08:25.000 | It takes a lot of people like you continuing and innovating as well.
00:08:29.000 | All right.
00:08:30.000 | So, let's go to the next one.
00:08:32.000 | The main point that I really want to make at the end over here is UX matters in what we're
00:08:36.000 | doing.
00:08:37.000 | Don't think that you're just going to get something out there.
00:08:39.000 | The UX that you end up choosing, the modality you end up choosing with that AI application
00:08:44.000 | can either make or break it.
00:08:45.000 | And we have seen that a lot as well.
00:08:48.000 | All right.
00:08:49.000 | So, what other things?
00:08:50.000 | The other thing that I want to tell you is once you actually hit a little bit of that
00:08:53.000 | product market fit, get used to a very fast pace.
00:08:56.000 | If you think right now you're fast pace, things go kind of to the other side once you're having
00:09:01.000 | success with the business in AI.
00:09:02.000 | There's a lot of questions coming at you.
00:09:04.000 | A lot of things that you have to handle both internally and then externally.
00:09:07.000 | So, some things that are also learnings from inside.
00:09:10.000 | Number one is, and you probably know this, syntax is not software.
00:09:13.000 | Just because you could read Java doesn't make you a Java developer.
00:09:16.000 | So, on that end, same thing for AI.
00:09:19.000 | Just because AI can actually understand some of the syntax, it does not make it a developer.
00:09:24.000 | You have to do a lot of work from a semantic perspective for that AI to end up generating the
00:09:29.000 | right code.
00:09:30.000 | So, just remember that.
00:09:31.000 | Syntax does not suffer if you're going to be in this space.
00:09:34.000 | The other stuff is you need a global presence if you're going to have a global product.
00:09:38.000 | We have deployments all around the world in order to keep that 150 milliseconds many times.
00:09:44.000 | So, we are in Japan, we are in Europe, we are in North America in multiple data centers.
00:09:50.000 | In order to guarantee that first time to buy it overall is very, very fast.
00:09:55.000 | The other stuff is set up scorecards for quality.
00:09:58.000 | You could mess up your deployment very quickly.
00:10:01.000 | Like, you think you actually have something that it works, and then you go to production,
00:10:07.000 | meaning your offline evaluations.
00:10:08.000 | Right?
00:10:09.000 | So, in your offline evals, everything is working.
00:10:11.000 | You go to prod, and your online evals kind of tank.
00:10:14.000 | So, be ready for that.
00:10:16.000 | Scorecards are very, very helpful.
00:10:18.000 | Because what sometimes works in offline is not really going to work in online.
00:10:22.000 | So, if I have one thing to tell you at scale is set up your offline and online evaluations.
00:10:28.000 | So, you could actually end up iterating quickly.
00:10:31.000 | Because you're going to have to increase the ability for you to ship.
00:10:34.000 | So, what else?
00:10:36.000 | The other thing that we have learned, and there's a saying that says,
00:10:39.000 | "Trust grows at the speed of coconut, but falls at the speed of coconut."
00:10:45.000 | And really what I want to tell you on that is, once you go into companies,
00:10:49.000 | so if you're going to take your product and go to B2B overall,
00:10:52.000 | you're going to have to care about three things.
00:10:54.000 | Number one is security.
00:10:55.000 | Please don't store anything at rest.
00:10:57.000 | Usually, that's not going to be a good thing for you or any of your prompts or any of the customer data.
00:11:01.000 | Please don't trade on their data without their approval and all of those types of things.
00:11:06.000 | You're not going to get far.
00:11:08.000 | The second thing, there's a lot of legal things that you have to care about.
00:11:11.000 | You know, for example, for us, we provide indemnification to our customers just in case there's anything happening.
00:11:17.000 | So, you know, and as you could probably imagine from the legal perspective today,
00:11:21.000 | there's a lot of regulations and things that we're talking about with many of the countries.
00:11:25.000 | And the last thing is what we call responsible AI.
00:11:28.000 | You could see, I think yesterday's -- and someone told me this -- but I think yesterday's open AI demo failed,
00:11:35.000 | but it mainly failed on that.
00:11:37.000 | It failed on their ability to actually have, you know, not security, but trust and trustworthiness into what they're doing and the responsible AI aspect of it.
00:11:48.000 | So, the same for us.
00:11:50.000 | We invest a fair amount in making sure that the AI is doing the right thing for the user and it's not harming the user.
00:11:57.000 | So, I encourage you to continue to think from that ethics perspective because it's really important.
00:12:03.000 | We're creating this to advance, you know, human progress forward, not really taking it back.
00:12:07.000 | All right.
00:12:08.000 | So, now I want to kind of explore -- I'm going to shift it from, okay, some lessons learned into a little bit more about the future.
00:12:17.000 | So, from a future, what gets me out of bed?
00:12:20.000 | Well, fundamentally transforming how the world builds software together.
00:12:23.000 | But what does that kind of mean?
00:12:25.000 | Like, where's Copilot going?
00:12:27.000 | There's a talk by Brett Victor that is called "The Future of Programming."
00:12:32.000 | And I want you to -- I want you to watch it because I think, you know, it happened a long time ago, but really talks about the struggles that we have in software going forward that are still present there.
00:12:42.000 | In the first prompt, actually -- well, we'll put it like this.
00:12:47.000 | The first prompt that I want to give you is the following.
00:12:49.000 | What does it look like to move from procedures to goals and constraints?
00:12:53.000 | So, what does that mean?
00:12:54.000 | Well, coding today is procedures, text files, and sequential programming.
00:12:59.000 | This is how you do things.
00:13:01.000 | You know, our CEO went and programmed an OctoRK demo, a snake demo.
00:13:06.000 | So, if a CEO can actually program, that's always great, right?
00:13:08.000 | Although, in a technology company, hopefully all of them can.
00:13:12.000 | But it's this, right?
00:13:14.000 | Like, he did it in JavaScript.
00:13:15.000 | I was kind of trolling him a little bit on, hey, can you just do it in TypeScript and can we have Copilot do that for you?
00:13:21.000 | But it's just procedures, text file, and sequential.
00:13:27.000 | And is that really what AI should help you with -- should be doing going forward?
00:13:32.000 | I think no.
00:13:33.000 | I think a future of Copilot really goes on operating on goals and constraints.
00:13:40.000 | And then the REPL, the actual programming environment that you have will need to change.
00:13:46.000 | This is what we want to do.
00:13:47.000 | We just don't want to play to the status quo of what it is today.
00:13:50.000 | We want to have AI actually change it and kind of do a step up going forward.
00:13:55.000 | So, what do I want the future to be?
00:13:58.000 | I want the future to be create me an app, use Remix, Redis, and Postgres, have it mash the state machine, go and use for this UI library, and then host it in my cloud.
00:14:10.000 | And then once we get there, I think you're going to kind of be unstoppable.
00:14:14.000 | You're going to be able to go and say, you know what, I'm going to go in and evaluate all of the vector databases, all of the embeddings kind of models, and everything else that I have to do to actually make this AI product work.
00:14:25.000 | But that's what the future looks like for me, it's kind of the semantic workspace at the end.
00:14:30.000 | So, two, what it is to have reasoning on code.
00:14:34.000 | So, what would be for the AI to reason on code?
00:14:37.000 | Well, our brain has the ability to actually do reasoning very well, by the way.
00:14:41.000 | And it does that through summarization many times.
00:14:44.000 | So, here's an example.
00:14:46.000 | Let's try this.
00:14:47.000 | So, imagine you need to add a feature flag in a code base you're not familiar with.
00:14:50.000 | What would you search for?
00:14:52.000 | Like, just ask yourself that right now and try to see what your brain gives you back.
00:14:57.000 | Okay.
00:14:58.000 | Congratulations, you just did rag on your brain.
00:15:01.000 | That's what you did.
00:15:02.000 | You went and you ended up putting a bunch of things together and into a query string.
00:15:08.000 | And let me tell you, your brain is freaking phenomenal at that.
00:15:12.000 | The AI that we have today, not even close.
00:15:15.000 | If not, I could retire probably.
00:15:17.000 | So, but your brain is really good.
00:15:18.000 | So, we're going to have to think about what it actually means to actually do summarization
00:15:23.000 | that fast with high quality to advance this going forward.
00:15:27.000 | So, here's another one, another test.
00:15:29.000 | So, this, many of you, if not all of you, are going to see a blue car.
00:15:33.000 | There's actually not a shade of blue in there.
00:15:35.000 | There is -- the car is gray.
00:15:37.000 | But your brain putting blue there because it knows that it needs to -- you know, it thinks
00:15:41.000 | that it's overexposed and it adds a little bit of that for you.
00:15:44.000 | You're used to strawberries being, you know, red, et cetera, et cetera.
00:15:48.000 | So, it adds that.
00:15:49.000 | So, this is what it might look like in the future as you reason through code.
00:15:54.000 | And Amelia, who is a speaker that is coming, I think, two after this, did this project while
00:15:59.000 | she was at GitHub.
00:16:00.000 | But think about a little bit more about what does it mean to actually have these broken down
00:16:05.000 | into many parts and then be able to reason and summarize them very quickly.
00:16:10.000 | So, I'm very excited.
00:16:11.000 | We're doing a lot of -- I would say the GitHub Next team is doing a lot of innovation in that
00:16:15.000 | space as well.
00:16:16.000 | So, think about, you know, a model, architectural, advancing in rags, and then visualizations
00:16:22.000 | to help you with reasoning on code.
00:16:25.000 | What's problem number three?
00:16:27.000 | So, what does it look like to create software together with a co-pilot and others?
00:16:32.000 | A lot of the UX that you have today was not meant to have an AI collaborator.
00:16:39.000 | That's why we're all trying to put a sidebar with a chat thing on it.
00:16:44.000 | It's because, like, the paradigms today, that's kind of what they could afford.
00:16:47.000 | But that's not what the future should be if AI is really going to be with you helping you
00:16:53.000 | get something done.
00:16:55.000 | So, the way that developers collaborate haven't really changed in a long time.
00:17:00.000 | So, what I want to -- what I want to get to at the end is something like this.
00:17:04.000 | So, I'm getting an issue, and it's assigned to you.
00:17:06.000 | Actually, this is more what it looks like today more than later.
00:17:09.000 | So, you get an issue assigned to you.
00:17:11.000 | You meet to chat and discuss.
00:17:12.000 | You go with co-pilot.
00:17:13.000 | You open a PR.
00:17:14.000 | I don't know how many check suites you have.
00:17:16.000 | We have a lot at GitHub.
00:17:17.000 | It's kind of love and hate.
00:17:18.000 | And then you merge and deploy before Friday.
00:17:21.000 | The majority of you do not want to actually deploy on Friday.
00:17:23.000 | Although, if you are at Vercel, they tell you that they're the best cloud to do it.
00:17:27.000 | So, you YOLO to production with them any time you want, okay?
00:17:32.000 | Tell Guillermo that's the best way to do it.
00:17:35.000 | Okay.
00:17:36.000 | So, what would that UI in the future look like if it is actually designed for both human and
00:17:42.000 | AI collaboration?
00:17:44.000 | What would allow you to do if you could then steer the AI and even the other humans in it
00:17:53.000 | to collaborate on that code?
00:17:55.000 | And how can you do it so things that you need to ignore are ignored and things that need
00:17:58.000 | to be verified are verified?
00:18:00.000 | These are the things that we're trying to tackle at the moment.
00:18:03.000 | It's not only about code completion.
00:18:04.000 | It's not only about chat.
00:18:06.000 | It's not about having, you know, training your own models.
00:18:09.000 | It's really about advancing how we code and how the world codes together.
00:18:14.000 | So, I'm going to speed up because I promised Ben that I was going to end up very quickly.
00:18:19.000 | But what we really want to do with that is kind of make meeting transparent along the
00:18:23.000 | way.
00:18:24.000 | And what -- this ends up going.
00:18:27.000 | I cannot give you a sneak peek yet, but I just want you to think about an immersive experience.
00:18:33.000 | And this is just kind of the beginning of that.
00:18:35.000 | An immersive experience that you could go in and out together with AI and other humans.
00:18:40.000 | So, we're excited about that.
00:18:41.000 | I'm not going to do prompt four, but security is top of mind for all of us.
00:18:44.000 | What would it look like?
00:18:45.000 | So, you could imagine.
00:18:46.000 | I could do five, six, seven, eight prompts overall on how to continue to push, you know,
00:18:52.000 | what we do every day forward.
00:18:53.000 | So, the last thing that I want to leave you with is GitHub today is no longer the home of
00:18:58.000 | open source and it's no longer the best version control system on earth.
00:19:01.000 | We have worked a lot to actually make it into a platform.
00:19:04.000 | We have an AI, you know, CI/CD system with actions.
00:19:08.000 | We have packages.
00:19:09.000 | We have copilot.
00:19:10.000 | We have code spaces.
00:19:11.000 | So, it's really an, you know, end-to-end platform.
00:19:14.000 | And we're going to infuse that platform with AI.
00:19:17.000 | And with that, I want to end it.
00:19:19.000 | And thank you all for entertaining this.
00:19:21.000 | And have an amazing conference.
00:19:23.000 | Thank you.
00:19:24.000 | Thank you.
00:19:25.000 | Thank you.
00:19:26.000 | Thank you.
00:19:27.000 | We'll be right back.