This is a tragedy that we see more and more as a lot of building blocks are starting to get figured out. I'm really excited for this next section. So we'll be doing a fireside chat with Andrew, and Andrew probably doesn't need any introduction to most of us here. I'm guessing a lot of people can begin some of his classes, even with Sarah, or deep learning.

But Andrew has been a big part of the LinkedIn story. So I met Andrew a little over two years ago at a conference, and we started talking about LinkedIn. And he graciously invited us to do a course on LinkedIn deep learning. I think it must have been the second or third one that they ever did.

And I know a lot of people here would probably watch that course or have started on LinkedIn because of that course. So Andrew has been a huge part of the LinkedIn journey. And I'm super excited to welcome him on stage for a fireside chat. So let's welcome Andrew in.

Thank you. You've obviously touched and thought about so many things in this industry, but one of your takes that I cite a lot, and probably people have talked about, is your take on kind of like talking about the agenticness of an application as opposed to whether something's an agent.

And so, you know, as we're here now at an agent conference, maybe we should rename it to an agentic conference, but would you mind kind of like clarify that? Yeah, I think it was like almost a year and a half, two years ago that you said that, and so I'm curious if things have changed in your mind since then.

So I remember that, and so I remember an agent. I mentioned that Harrison and I spoke at a conference in a year and over a year ago, and at that time, I think both of us were trying to convince other people that agents are a thing you should pay attention to.

And that was before maybe, I think it was mid-summer last year, a bunch of lawmakers got an agentic term and start to see that sticker everywhere there was. So I heard this question, I think about a year and a half ago, I saw that long people are arguing, this is an agent, this is not a different, you know.

My friends, this is a chain of conference that's an agent. And I felt that the response had an argument, but that we would succeed that as a community would just say that the decrease from something is agentic. So, and then we just say that if you want to go on an agentic system, move autonomy or a lot of autonomy is all fine.

No need to spend time arguing, this is truly an agent. Let's just call all these things agentic systems with different degrees of autonomy. And I think that actually, hopefully, producing a lot of autonomy people, at least spend time arguing some as an agent. And this is common for agenting when they're young.

And I think that should run down. Where on that spectrum of kind of like a little autonomy to a lot of autonomy do you see people building towards this? Yeah. So, my team, your team uses mangrove for a hardest problem during complex values and so on. I'm also seeing tons of digital opportunities that, frankly, are fairly linear of workflows or linear with just occasional cyber actions.

So, one of the businesses, there are opportunities where you're running down with people looking to form a website, doing web search, checking something within the piece to see if it's a compliance issue or if there are, you know, some machine, some service up to. And it's kind of a, well, take something, copy-paste it into a web search, or something different.

So, in business processes, there are actually a lot of fairly linear workflows or linear with very small reasonable educational branches. Usually, you couldn't do a failure with some of the agentless workflow. So, I see a lot of articulation, but one challenge I see businesses have is it's still pretty difficult to look at, you know, some stuff that's being done in US and figure out how to turn it into a agentic workflow.

So, what is the granularity? You know, we should try to bring down the school into micro tasks. And then, you know, after you build your initial prototype, if it doesn't work well enough, and which of these steps do you work on to improve performance? So, I think that whole bag of skills on how to look at a bunch of stuff that people are doing, break into sequential steps, where are the small number of branches, how do you put in place evals, you know, all that.

that skill set is still . And then, of course, there's a much more complex . I think you heard a bunch of very complex groups that's very valuable as well. But, I see much more in terms of number of opportunities, the amount of value. There's a lot of simpler efforts that I think are still being used as well.

Let's talk about some of those skills. Like, so, you've been doing deep learning. I think a lot of courses are in pursuit of helping people kind of like building. And so, what are some of the skills that you think agent builders all across this sector should kind of like master and get started with?

Well, it's a good question. I wish I could answer that. I've been thinking a lot about this, actually. I think one of the challenges, if you have a business process, you often have people who comply with people in the job, whatever, these steps. How do you put in place the, you know, through the MacGraph type integration, you want to see if you have CPP also some of that too, to adjust the data.

And then, how do you process or process multiple steps, be able to build this into a system? And one thing I see a lot is putting in place the record key balance framework to not only to understand the performance of the overall system, but to trace the individual steps.

You can pull in on what's the one step that is broken, what's the outcome that's broken to work on. I find that a lot of teams probably way longer than they should, just using human devalance. Every time you change something, you can sit there and look at a bunch of awkward scenes, right?

I see most teams probably slower to put in place and develop, systematic development is ideal. But I find that having the United States or one to two days in the project is still really difficult. The skill teams, the teams are still learning skills, they often, you know, go down the line now, it's very recent, like a few months, trying to improve one for more.

And then more students will say, you know what, I don't think this could ever be made to work. So just don't, just find the different variables as well. I wish I knew, I wish I knew, I knew more efficiently, to get this almost tactile knowledge. Often you're there, you know, look at the output, look at the trace, look at the length of the output, and just try to make a decision, right?

It is a lot harder to mix, and that's still very difficult. And is this kind of like tactile knowledge mostly around LLMs and their limitations, more around like just the product framing of things, and that skill of taking a job and breaking it down, that's something that we're still getting accustomed to.

I think it's all the way well, actually. So I feel like over the last couple of years, AI tool companies have created an amazing set of AI tools. this includes tools like, you know, that graph, but also, how do you, I guess like, how do you think about RAN?

How do you think about the chat box? Many, many different ways of approaching memory. What else, how do you need those? How do you got your else? But I feel like there's this, you know, wild, sprawling array of really exciting tools. One picture I often have in my head is, if all you have are, you know, purple Lego bricks, right?

If you can't build that much, you just say stop. But, and then think of these tools as being a kind of Lego bricks, right? And the one tools you have is as if, you know, just a purple Lego bricks, but the red one, the black one, and the yellow one, the green one.

And as you get more different colored and shaped Lego bricks, you can very quickly assemble them into really cool things. And so I think a lot of these tools, maybe one's that's wrapped in progress, different types of Lego bricks. And when you're trying to build something, you know, sometimes you need that right squiggly weird shaped Lego brick, and some people go and they can fuck with it and just get the job done.

But if you never build evals of a certain time, then, you know, then you can actually end up spending, like a few extra months to do something that someone else that's done that before could say, "Oh, wow, I should just put a few thousand strangers and I'll just judge." And just go through that process and get it that much faster.

So, one of the unfortunate things about AI, it's not just one tool. When I'm coding, I just use a whole bunch of different stuff, right? I'm not a master. I've stuck myself up with enough tools to sell that. So, yeah, and I think having that practice with different tools, but also helps .

And one of the things, they've also changed. So, for example, just all of a sudden, have no longer, longer context. And a lot of the best practices from rag, from, you know, a year and a half ago, or whatever, are much less relevant today. And I remember, Harrison was really ready to talk about these things, and play with the early managing rag frameworks across this organization and all that.

that, as the content release got longer, now we've just done a lot of stuff into our content. It's not that rag has gone away, but the hyperpractic team has gotten way easier. There's a huge range of hyperpractors that work, you know, like, just fine. So, as algorithms keep harvesting, the instincts we hold, you know, two years ago, which may only not be relevant than you want today.

You mentioned a lot of things that I want to talk about. So, okay, what are some of the Lego bricks that are maybe underrated right now that you would recommend that people aren't talking about? Like evals, I think, you know, we had three people talk about evals, and I think that's top of people's mind.

But what are some things that most people maybe haven't thought of or haven't heard of yet that you would recommend that we're going into? Good question. I don't know. Yeah. I want to share this. So, even though people talk about evals, there's some ways that people don't do it.

Why don't they do it? I think it's because people often have, I saw a post on evals rights as well. People think of writing evals as this huge thing you have to do, right? I think of evals as something I'm going to fill together really quickly, you know, in 20 minutes.

And it's not that good, but it starts to complement my human eyeball evals. And so what often happens is our system and this one problem that I keep on getting regression. I thought I made it work, then it breaks. I'm here, I embrace it. I've done it. Then I call it a very simple eval, maybe with, you know, five different examples in some very simple eval situation to just check for this one regression, right?

This one thing, right? And then I'm not swapping out human evals for automated evals. I'm still looking open myself, but when I change something, I don't want to see evals to just, you know, take this word into something psychopathic and think about it. And then what happens is, just like the way we write English maybe, once you have some slightly helpful but clearly very broken, imperfect eval, then you start going, you know what?

I can improve my eval to make it better. I can improve to make it better. So just as when we build the public applications, we build some, you know, very quick and dirty thing that doesn't work and we could make it better. For a long of the way, I built evals.

I built really awful evals that barely helps. and then when you look at what it does, you can go, you know, it's this email's broken. I can fix it. And you can improve it for you to make it better. So that's one thing. Actually, one thing that people have talked a lot about and I think is so automated is the voice stack.

It's one of the things that I'm actually very excited about voice applications. A lot of my friends are very excited about voice applications. I see a bunch of large enterprises really excited about voice applications, very large enterprises, very large users. For some reason, while there are some developers in this community doing voice the amount of developer attention on voice stack applications, there is some, right?

It's not really important, but that's one thing that feels much smaller than the large enterprise importance I see as well as applications going along the way. And not all of this is the low-time voice in here. It's not all speech-to-speech native volume or volume or volume. because I find those models are very hard to control, but when we use more of an agenting voice stack workflow, which is great, which we find much more controllable.

Boy, ADFM work with a ton of teams on voice stack stuff that some of which hopefully be announcing the future of us seeing all the very exciting things. And then, other things I think underrated, one of the ones that maybe is not underrated but no others should do it, I think many of you have seen that developers developers using AI systems about coding is so much faster than developers and don't.

I've been, it's been interesting to see how many companies CIOs and CTOs still have policies that don't let engineers use AI systems going. to improve. I think, maybe sometimes are good reasons, but I think we have to go across that because, frankly, I don't know, my teams and I just hate about the coding on AI assistants.

So, I think, some of this is something that you have to do. I think underrated is the idea that I think everyone should learn to code. One fun fact about AI fund, everyone in AI fund, including, you know, the person that runs our front desk, receptionist, and my CFO, and my attorney, and the general counsel, everyone in AI fund actually knows how to code.

And, it's not that one that you're salvaged, is there not, but in their respective job functions, maybe as a mother or a home code, that's able to tell a computer what they want to do. And so, it's actually driving more productivity progress across all of these job functions that are not salvaged.

So, that's an exciting result. Talking about kind of like AI coding, how, what tools are you using for that, personally? So, so, we're, working on some things that are not getting announced. Exciting. So, maybe, I, I, I do use Cursor, Ritzer, um, um, um, that's some of the things.

All right. Talking about voice. If, if people here want to get into voice and they're familiar with building kind of like agents with LLMs, how, how similar is it? Are there a lot of ideas people, all they have to learn? Yeah, so, in terms of, there are a lot of applications where I think voices, you know, increase certain interactions, um, that, um, that are not much more.

It, it turns out that, uh, it turns out from an application perspective, um, it turns out that this problem is kind of intimidating, right? For other applications, people can say, tell me what you think, here's a block of text problem, read a bunch of texts from you. That's actually very intimidating for users.

And one of the problems with that is, um, people can use backspace, and so, you know, people aren't just as little learned to respond by the text. Whereas, the voice, you know, girls fall in, and she said, "If you're talking, you could change your mind." And I say, "Oh, I changed my mind, forget everything." And I find that, um, the lot of applications for the user friction to just getting them to use it is lower.

You say, "You know, tell me what you think." And then they respond. Um, so, in terms of voice, one of the biggest difference is, um, in terms of, um, if you can, if someone says something, you know, they want to respond in, you know, some point a second, right?

Less than 50 milliseconds is great, but I agree, I'm using some point a second. And with a lot of, um, agents, they weren't closed, they were run for, many seconds. So, when 2009 wanted to grow Avatar to build an Avatar, if you want, um, um, our initial version had kind of five to nine seconds of literacy.

And it's, and it's just a bad user experience. You say something, you know, nine seconds aside, the Avatar response, but so we wanted to build things like, um, uh, how to create response. So just as, you know, if you ask me a question, I might go, huh, that's interesting.

So, we wanted to basically do that to hide the latency, um, and it actually seems to work great. And there are all these other little tricks as well. Turns out, we're building a voice, um, customer service chatbot. it turns out that when you play background voice, at a customer contact center, it's a dense silence, people are much more sensitive on that, of that, you know, latency.

So I find that there are a lot of these things that, um, um, that are different than a pure text based element. but in applications, for a voice based modality, there's some reason to be comfortable and just not talking. Um, I think it sometimes reduces user friction. So, you know, getting some information out of them.

Save, I think when we talk, we don't feel like we need to deliver perfection as much as we operate. Um, so it's somehow easier to really just start blurting on your ideas and change your mind and go back and fail. And that lets us get the information from them that we need to help the user too.

that's interesting. Ah, wow. One of the, one of the new things that's out there, you mentioned briefly, is MCP. How are you seeing that transform how people are building apps? What types of apps are building and what's happening in any business? Yeah, so I think it's really exciting. Um, just this morning, we released with Anthropi and my show was on MCP.

Um, uh, I actually saw a lot of, uh, stuff, you know, on MCP that I thought was quite confusing. So, when we got through Anthropi we said, you know, let's get really good shortfalls on MCP that explains it clearly. I think MCP is fantastic. I think it's a very clear market that, you know, that Okunai adopted it.

Also, I think speaks the importance of this. Um, I think the MCP standard will continue to evolve, right? So, for example, I think one of you, many of you know what MCP is, right? Makes it much easier engines primarily, but frankly, other types of software to plug into different types of data.

When I'm using Elvis myself or when I'm building applications, frankly, for a lot of us, we spend so much time on plumbing, right? So I think, for those of you from Bosch enterprises as well, the AI, especially, you know, are like pretty darn intelligent to do a lot of stuff when given the right context.

But so, I found that, I spent, my team spent a lot of time working and plumbing on the data integration so you get the context to make it feel to do something that often is pretty sensible when it's perfect in the context. So, MCP, I think is a fantastic way to try to standardize the interface.

So, tools, API calls, as well as the responses. It feels like, it feels a little bit like, wow, that's, you know, a lot of MCP services you find the internet do not work, right? and the authentication systems are kind of, you know, even for the very large companies, you know, MCP services, the company is not clear and you know, authentication token, token works, it's biased, a lot of that going on.

I think the MCP protocol itself is also early. Right now, MCP gives a long list of the resources that will, you know, eventually, I think, multiple, you know, I don't know, you know, I don't know, I don't know, the MCP, it is, the land graph, the land graph has so many API calls, you just, you can't have like, a long list of everything under the sun, for an agent to sort out, and so, I think MCP is a really fantastic first step, definitely encourage you to learn about it, that will make your life easier, probably, if you find a good MCP server implementation, so how it's on the data integrations.

And I think, I think what we're important, is this, this idea of, when you have, you know, N models, or N agents, and N data sources, it should not be N times N method, to do all the integrations, N plus N, and I think MCP is a, is a fantastic first step, it will need to evolve, like a fantastic first step, to one that type of data integration.

Another type of protocol, that seemed less fun, than MCP, is some of the, agent to agent stuff, and I remember, when we, when we were at a conference, a year or so ago, I think you were talking about, multi-agent systems, which this was kind of enabled, so, how do you see some of the, multi-agent, or agent to agent stuff, of all of that?

So, I think, you know, the agent to AI, is still similar to me, both of us, right, we, we struggle, giving our code work, and so, making my code, my agent work, with someone else's agent, it feels like, a two miracle, you know, . So, I see that, when one team is building, a multi-agent system, that often works, because we build, a bunch of agents, they could meet themselves, that works, but, right now, at least at this moment in time, maybe I'm off, the number of examples I've seen, of when, you know, one team's agent, and correctional agents, successfully engaged, totally different teams, agent and correctional agents, I think we're a little bit, early to that, I'm sure we'll get there, but I'm not, personally seeing, you know, real success, huge success stories, of that, yet, I'm not sure, no, I agree, it's, I think it's super early, I think, if MCP is early, I think, the new page and stuff, is just earlier.

Another thing, that's kind of like, top of, what's mine right now, is it's kind of, vibe coding, and all that, and you touched on it, a little bit earlier, with how people are using, these AI coding assistants, but, how do you think about, vibe coding, is that a different skill, than before, or what kind of purpose, does that serve, in the world?

code, you know, barely looking at the code, right, I think it's a fantastic thing, I think it's unfortunate, that that called, fine coding, because it's, just leaving a lot of people, and I'm thinking, just go to the vibes, you know, and frankly, when I'm coding, for a day, you know, with fine coding, or whatever, with AI coding assistants, I'm frankly exhausted, by the end of the day, from TV, intellectual exercise, and so, I think the name is unfortunate, but the phenomenon, in this world, is great, so, I, I, over the last year, a few people, have been advising, others, to not learn to code, on the basis, that they have more, to be in coding, I think, I look back, at some of the worst, career advice, ever, because, over the last, many decades, is coding, easier, more people, started to code, so, it turns out, you know, when we went, from punch cards, to keyboards, that turned them off, very well, but, it turns out, one of the most important, the most important schools, in the future, for developers, and not developers, is the ability, to tell them about, exactly what you want, so, to improve, and, I think, understanding, at some level, how computer works, lets you, probably, to improve, and, I think, understanding, at some level, how computer works, lets you, probably, to improve, and, to improve, and, I think, understanding, at some level, how computer works, lets you, probably, to improve, and, to improve, and, which is why, I still, try to advise everyone, to learn, one part of the migration, and Python, or something, and then, I think, I personally, about, a much stronger, Python developer, than, say, JavaScript, right?

But, with, AI system coding, I now write, a lot more JavaScript, types of code, than I ever used to, but, even with debugging, JavaScript code, than something else, building, that's, really, understanding, you know, what an error case is, what does this mean, that, that's, really, . If you, if you don't like the, name, do you have a better name, than mine?

That's a good question. I shouldn't think about that. We'll, we'll get back to you on that. Good question. One, of the things that, you announced recently, is a new fund, so, congrats on that. Were people, in the audience, who are going to be thinking, of starting a startup, or, looking into that, what advice, for them?

So, I think, it was an interest to do, investment companies, that we co-founded. So, I think, in terms of, looking back on, lessons learned, the number one, I would say, the number one, predictor, of a startup success, is, speed. I know, it's looking at value, but I see a lot of people, that I've never seen yet, the speed, with which a skilled team, can execute.

And, if you've never seen it before, I know many of you have seen it, it's just so much faster, than, you know, anything that, slow businesses, you know, how they do, and I think the number two, predictor, also very important, is, technical knowledge. It turns out, we look at the skills, to some things, like, how you market, how you sell, how you price, you know, all that is important, but that knowledge, has become a little bit more, widespread.

The knowledge that's really rare, is, how does technology actually work, so quickly? So, I have deep respect, of good luck, and people, when pricing is hard, you know, marketing is hard, position is hard, but that knowledge, is more diffuse, and the most rare resource, is someone, that really understands, how the technology, grows.

So, when I find, technical people, that have good instincts, or understands, do this, don't do that, do this, go twice as fast. And then, I think, a lot of the business stuff, you know, back to the college, it's very important, it's usually easier, to figure out something. Alright, that's great advice, for sharing something.

Um, we are gonna wrap this up. We're gonna go to a break now, but before we do, please join me, giving Andrew a big hand. a big hand. Thank you. Our next session, we'll begin in 15 minutes, of course, we'll have an opportunity, to hear our fellow attendees, if you're about to, if you're about to, you're about to, we're going to be to this, at NOVA.

If you'd like to meet, the line chain, at the, or, our, our, our, our, who's also in NOVA, come say hello, we'll be passing out, special swag. Please return to the main stage, in 15 minutes, for our next session, for our next session, for our next session. our next session.

is the first session. the first session. the first session. the first session. the first session. the first session. So, let's go. So, let's go. So, let's go. So, let's go. So, let's go. So, let's go. So, let's go.