awesome thanks so much for coming I wanted to give a bit of a talk on implementing MCP clients and talking about MCP at scale within a large organization like Anthropic I wanted to give first a little introduction for me my name is John I've spent 20 years building large-scale systems and dealing with problems that that causes and so I've made a lot of mistakes and I'm excited to give maybe some thoughts on avoiding some of those mistakes I'm currently a member of technical staff here at Anthropic and I've spent the past few months focusing on tool calling and integration and implementing MCP support for all of our internal like external integrations within the org and so looking at tool integration with models we've kind of hit this timeline where models only really got good at calling tools like kind of late mid last year and suddenly everyone got very excited because like your model could go and call your Google Drive and then it could call your maps and then it could send a text message to people and so there's a huge explosion with like very little effort you can make very cool things and so teams are all trying to move fast everyone's moving very fast in AI custom endpoints and then start proliferating for every use case there's a lot of like services popping up with like slash call tool and slash like get context and then people start to realize the digital needs of some authentication there's a bunch of stuff there and this kind of led to some integration chaos where you're duplicating a bunch of functionality around your org nothing really works the same you have an integration that works really well in service a but then you want to use it in service b but it you can't because it's going to take you three weeks to rewrite it to talk to the new interface and so we're in this kind of spot and the place that we came to at anthropic is realizing that over time all of these endpoints started to look a lot like mcp you you end up with some get tools some get resources some elicitation of details and even if you're not using the entire feature space of mcp as a whole immediately like like you're probably going to go extend into something that kind of looks like it over time and when i'm talking about mcp here there's kind of two sides to mcp that in my mind feel a bit unrelated there's this json rpc specification which is really valuable as engineers it's like a standard way of sending messages and communicating back and forth between providers of context for your models and the code that's interacting with the models and uh getting those messages right and getting those messages right is the topic of huge debate on like the mcp mcp uh repos if you're involved with any standardization process ever you know how those conversations end up going and then on the other side there's this global transport standard which is the stuff around streamable hdp oh 2.1 session management and global transport standard is hard because you're trying to get everyone to speak the same language and so it's really nitty but there's not a lot of like most of the juice of mcp is in this the message specification in the way that the servers are interacting um and so we started asking ourselves like can we just mcp for everything and we said yes with the caveat that yes is for everything involved in presenting model context to models um we have this format where your client is sending these messages something's responding with these messages um where that stream is going it really doesn't matter it can be on the same process it can be another data center it can be through a giant pile of uh enterprise networking stuff um it doesn't really care at the point that your code is interacting with it you're just calling a connect to mcp and you have a a set of a set of tools and methods that you can call so uh standardizing on that seemed useful um standard why standardize anything internally um being boring on stuff like this is good it's not a competitive advantage to be really good at making google drive talk to your app it's just a thing that you need to do it's not your differentiator uh having a single approach to learn as engineers makes things faster you can spend your cycles working on interesting problems instead of trying to figure out how to plumb uh integration and uh if you're using the same thing everywhere then like each new integration might clean up the field a bit for the next person who comes along uh it's it's overall a good thing in cases like this where we're we're not really doing interesting we're plumbing context between integrations and things that are consuming the integrations uh why standardize on mcp internally um i and this is where i might make an argument to everyone that there's already ecosystem demand you have to implement mcp because everyone's implementing mcp so why do two things um it's becoming an industry standard there's a large coalition of engineers and organizations that are all involved in building out the standard uh all of the major ai labs are represented in that so you you know that as new model capabilities start to be developed those patterns will be added to the protocol because all the labs want you to use their features so i think the standardizing on mcp internally for this type of context is a it's a good bet and one of the things you get with mcp is that it solves problems that you haven't actually run into yet like there's a bunch of stuff in the protocol that exists because there's a problem and a need and having those solutions at hand when you run into them is really important so sampling an example of where this might be valuable in your company you might have four products that have four different billing models uh for reasons because you're building fast um you might have a bunch of different token limits you might have different ways of tracking usage this is really painful because you want to write one integration service to connect your slides and how do you go and like hook the billing and the tokens up correctly and mcp has uh already has sampling primitives so you can build your integration you can just be like okay your integration sends a sampling request over the stream uh the other end of the pipe fulfills that request you can go and hook it in everything works great and so this is a thing that uh a shape problem that might take you a bunch of effort internally without this but you already have the answer kind of gift wrap for you in the protocol and so at anthropic we're running into some requirements converging we're starting to see external remote mcp services popping up like mcpu.asana.com which is really cool we wanted to be able to talk to those talking to those is complex because you need external network connectivity you need authentication uh there's a proliferation of internal agents people have started building pr review bots and like slack management things and just lots of people have lots of ideas no one's really sure what's going to hit so we're having a huge explosion of llm backed services internally uh with that explosion there's a bunch of security concerns where uh you don't really want all of those services to be going and accessing user credentials uh because that ends up being and being kind of a nightmare you don't want uh outbound external network connectivity to be everywhere um auditing becomes really complex uh and so we are looking at this problem we wanted to be able to build our integrations once and use them anywhere and so a model i was introduced to by a mentor of mine and a while ago is the pit of success which is the idea that um if you make the right thing to do the easiest thing to do then everyone in your org kind of falls into it and so uh we designed a service which is just a piece of shared infrastructure called the mcp gateway that provides a single point of entry and provided engineers just with a connect to mcp call that returns a mcp sdk client session on the end and we're trying to make that as simple as possible uh because that way people will use it if it's the easiest thing to do um we use url based routing to route to external servers internal servers it doesn't matter it's all the same call uh we handle all the credential management automatically because you don't want to be implementing oauth five times in your company uh it gives you a centralized place for rate limiting and observability uh i have an obligatory diagram here of a bunch of lines going in and out but uh uh here's a gateway in the middle this is kind of the thing just one more box will solve all our problems uh can i go next uh where is my yeah uh so the uh the code that we have here we just made some client libraries we just mcp gateway connect to mcp uh we pass in a url an org id account id this is like a bit simplified we actually pass a signed token to authenticate because it's accessing credentials but this is the basic idea and then importantly this call returns an mcp sdk object which means that when you feature get added to the protocol you just update your mcp packages internally you get those features across the board everything works great the same code seamlessly connects to internal external integrations when it comes to transports uh and this is a bit high level on hand wave because everyone's setup is different um internally within your network it really doesn't matter you can do anything you want we've got the standardized transport for connecting to external mcp servers um but really just picking the best thing for your org so we went and picked websockets for our internal transport and here's just a quick code example it's nothing special we just have a web socket uh that's being opened we are sending these json rpc blobs back and forth over the web socket and then if i can make this scroll down at the end we just pipe those uh read streams and write streams into an mcp sdk client session and we're good to go we've got mcp going um you might want to do this with grpc because you want to wrap these in some multiplexed transport so you don't have to open one rub socket per connection that's pretty simple also uh we have read stream right stream at the end uh starting to see a pattern here you can do like unix socket transport if you want you can just have uh messages be passed that way read stream right stream at the end mcp works great um i threw in an enterprise grade email transport implementation over imap um which is pretty much the same thing you just go through here is our server we're sending emails back and forth uh dear server i hope this finds you well uh mcp request start and then we pipe those into a client session at the end and so it truly doesn't matter like whatever it takes inside your organization is great we set up this unified authentication model where we're handling oauth at the gateway which means that consumers don't have to worry about all that uh all that complexity in their apps we added a get oauth authorization url function and a complete oauth flow because you might have different endpoints at anthropic we have api.anthropic.com and we have claw.ai and we might want those redirects to go back to different places but uh this is tied on the gateway it's really easy to start a new authentication a real advantage of having this put on your gateway is that the credentials are portable if you have a batch job that you're kicking off um your users don't have to re-authenticate to that you're just calling the same mcp with your internal user id and they get everything added correctly you're also internal servers don't have to worry about your tokens um so your request comes in internally for us we're hitting a websocket connection to mcp gateway uh with the auth token provided as headers to that uh that gateway receives your stored credentials you create an authenticated sdk client you just pass in the bearer token to the auth header uh and then you're good to go the mcp client receives a read stream and a write stream and so you just plumb those reach from the right stream into your internal transport and you're and you're you're good to go uh one of the things that this gives for your org that's not immediately obvious but is really valuable is a central place for all of your context that your models are asking for and all the context is flowing into your models for your org there's some papers written on mcp prompt injection attacks there is a general risk of uh models going and having access to google drive and deleting everything in google drive there's some need of uh enforcing policy you might want to be able to like ban malicious servers um do some content classification on the request see what's coming in kind of given audit and the really nice thing about this is that because it's mcp all of your messages are in a standardized format so it's really easy to hook into that stream and be like okay here is my tool execution message processor here is my tool definition thing or here's my resource management and so the payoff that you get from this is um adding mcp support to new services is as simple as possible you just go and import a package it doesn't matter what language you're in we've got multiple languages internally they all have their own kind of packages engineers can focus on building features and not plumbing uh you have the operational simplicity of having a single point of ingress egress and standardized message formats and you get future features for free as the protocol involves you get all of that work uh naturally and so i just wanted to go through some takeaways from this that i want to put to put to you is that mcp is really just json streams and how you pipe those streams around your infrastructure is a small implementation detail it's a couple lines of code hook the stream into the client sdk that makes the messages uh the you should standardize on something anything i think mcp is a good idea if you don't think it's a good idea like just pick something um your future self will thank you build some pits of success you really want to make the right way to do a thing the easiest way to do a thing and then everyone just falls into doing the right thing naturally and also centralizing at the correct layer so solving some shared problems off and external connectivity once allows you to spend your time working on uh more interesting problems that are more valuable to you and your your business your business um thanks that's all i got for you uh thank you so much for coming out