10x Development: LLMs For the working Programmer

Hello, are you all awake and you just have your coffee, that's great, welcome to my workshop about LLMs for programming or like how to become a 10x engineer, which I'm like actually pretty serious about. I put some handouts out, so I didn't print like 200 of them, but if you can sit next to someone who has some, there's like a couple lying around in this part of the room, and in general I encourage you to have a partner, two partners, because it's a workshop and it's like nice to share ideas.

There's a URL that's to a GitHub repo which has nothing in it, but it has the handouts as PDF and it has like a list of links or something like that. I encourage you to join the Slack channel, because of the size I won't be able to really interact with individuals, but I will be able to through Slack.

The only prerequisites you need are an LLM, so I use OpenAI just like the chat interface or like CloudSonet is great, but you can use like if you want to have fun and use like a small one on your laptop, you're welcome to. Find.com is also great, like anything that's all you need.

No code, no Git fork, no nothing. And with that, I'm manual, I'm like a software engineer by heart, I knew I wanted to be a programmer when I was six, and when GPT-3 came out, I like co-pilot, I was like what the, and started just like doing everything I do usually after 25 years of being a professional software developer with LLMs and just like keep finding new techniques until today.

I kind of am not on social media, I don't really follow what other people are doing, so this is kind of like all homegrown kind of stuff, but always with the focus of like actually doing like e-commerce work, PHP programming, just like down to earth stuff. And so I'm going to show first just like a few general concrete techniques.

Like I think a lot of programmers don't really vibe very well with the technology especially because the hype around it is like it's going to replace you. Us. Which is kind of true. But so I'll give like a bunch of techniques just like to ease and getting into it.

So who of you are programmers? Cool. Is there someone who's not a programmer because you can still stay because it's everything will still apply actually. Who's using LLM's co-pilot, well I'll put co-pilot a little bit apart, but like LLM's code generation for their work? All right, cool. I guess that's why you're coming to the fair, huh?

So after a bunch of general concrete techniques that I use which I feel are pretty important I'll go into the main concept of how I approach programming with LLMs which is like treating them as as translation engine. Not like as something that reasons or something that like writes code and can figure things out with agents.

It's just like oh they're good at translating one type of language to another. Which is honestly what we do as programmers as well, right? And we take GitHub tickets or whatever and we convert them to code. Or the other way around when the code doesn't work we create GitHub tickets.

Then a pretty important, this is like a new concept I haven't really played with but it's so massive is treating LLMs as like word simulators and what we can do with that in the context of just like pragmatic programming. And finally a bunch of like what I try to distill is like what are the skills that we need in this new age of software developers to become really effective because they're very, very different from from what they were in the past.

But some of them stay the same which is also good. So what I'm going to do is like I'm going to go through these pretty quickly. I wrote up, I wrote them up on the handout and the handout you'll see these gray blocks. They're concrete examples of just prompts.

Like I don't want to paste 20 pages of GPT transcript that no one cares about. But what you can do and try to maybe it will work just like paste these prompts. into chat GPT and say what happens and like there's some sequences which might or might not work depending on what the model outputs but hopefully you can follow your way around.

And there's links which obviously don't work on paper to some of the transcripts I just did yesterday. But really it's about you trying them out. Like don't try to, these are not examples you should follow. this is not the way to do it. It's just like an example of applying what I really want to get at.

And I'll give you about 10 minutes after showing you these things to basically communicate with your neighbors, try to solve some problems that you have, come up with some or just use some of the examples I show to get a sense of how these things work. So first technique which I see a lot of programmers not doing is not regenerating.

You'll come in, you'll put some prompt in it and I'll be like either it outputs something great and you're like holy crap or it outputs something that's buggy and you're like ah it doesn't work. But if you regenerate 10 times you'll see like oh it maybe works three times and five times it outputs something that's like almost working and then two times it's outputting something that's completely random.

The API works better for that because you can up the temperature like I realized chat open AI used to be much more wide in its outputs. So if you regenerate it often comes out with the same stuff. Second one is like instead of trying to correct the models and like no this is wrong this is not the right method and stuff like that just go back and edit either your prompt or if your UI allows it like LM studio or Libra chat like other frontends allow you to edit the LLM's response.

So if the LLM outputs bad code just go back fix the code and then the LLM thinks it output the right code. So it's really useful. Then a next technique is to often clear the context like I don't think I do more than three four prompts in a conversation and then I just take the output at the end just start a new conversation that tells me if you know my technique is robust.

Because I can't you know if I have a 20 page transcript I don't really know what's going on. I don't know what the context has been summarized to and usually by the end you're just like in LLL and it works much better with the bigger models now. So you know maybe this doesn't apply anymore but it's book good practice anyway experiment with different models because they all kind of have different vibes and you get to feel these different vibes and how they change over time or like how smaller models might just need more coaxing.

But can actually do the same stuff than the bigger models and this is all through practice. So like practice practice practice practice practice. I have the trouble that I just get rabbit hole when I have a task to solve I try to write the prompt that solves a task and then I try to write the prompt that generates the prompt that solves the task.

But you know practice. And then this is some very concrete stuff. Just make like custom system prompts that you can put into like chat open air whatever don't worry about changing them like if you have a session today where you're going to write PHP just change the system prompt to be PHP oriented like which libraries you use maybe what code formatting you use.

Just because it saves like a ton of time that's a very concrete technique there's a prompt on how to do that. And I want once I get to this slide maybe we can try it out. Generate helpers like every time you have a problem just ask the model to write a script to solve it right like often I put stuff in my clipboard and I have to like edit it to put it back into my code.

Maybe the imports I don't want to always like have to remove the imports when I paste into my ID so I can ask like a GPT write a shell script that just opens the clipboard and an editor and or open the shell script and just open the clipboard and remove the imports at the beginning.

It's like a simple sad three liner just generate it and at the end of the day you can throw it away or save it. And this might be the most important technique summarizing your transcript and I want you to do that today to exchange with your neighbor when you try something at the end just like summarize what I did as a read me as a how to as a wiki entry as a RFC and then share it with your neighbor so that they only have to read like one clean document and not like 10 pages of whatever.

So I really want you to do that today if you're like working with your neighbor which I really encourage at the end summarize the transcript and just send them that even if you do like rabbit holes I'm fine trying to solve a problem right like oh what's this type script bug that I have at the end you can summarize your search and just send it over and say like look this is this is how I solve that here's all the links if you want to look at it.

So that's something you would never do before when you were readings like overflow and like Google searching you would never send someone your Google search history. But now you can and a final technique is like when I ask it to when I try to figure out how to do something I use ridiculous domain examples like instead of working on my app which is often going to be kind of a meta app right like if I do infrastructure.

That's going to be like programming jargon and the model might get confused because I want to you know create a script that create scripts or manage the scripts. So instead I always ask it to do like you know manage dinosaurs or like zoo animals or like write an operating system about Tolstoy and then you see in the output.

What is Tolstoy relevant is like what it fills into the pattern and then when it says like Colonel driver you know okay that's Colonel driver but if it says like Colonel driver system function interface you're like you're not sure if it's like a driver about functions if it's the function of the driver like.

So I have an example here like write an Arduino Arduino Arduino for remote controlled T-Rex and I'll show that on screen and while you know for the next 10 minutes I want you to try out some of these things like you know write some helpers like create some system prompts or summarize some transcripts that you have research how to how bourbon bourbon is made and summarize you know your research to share with your neighbors like just.

Try different things and while you're doing that I'm going to watch the slack if you have cool examples you can also open PR's on the GitHub on the on the GitHub repo so that everybody can see them later on and I'll answer questions and if no one has questions I'll just do my own stuff in the browser they can look at for inspiration so before we get into that like maybe I'll just show you one of these of these prompts.

And you know the code doesn't need to work but it like gives me an idea of like different concepts. Can you read this is this big enough. Real time preemptive. Prey hunting. I don't know just mixing up words. I don't know just mixing up words. This is amazing. The artifact stuff is just amazing.

And here you can see right like the tasks are prey detection. I know okay like it understood that the task name is something different than the tasks thing itself. But so here I can tell that it used free artists and it's like no implement and artists from scratch and then you can just like rabbit hole right so going to implement it from scratch and I'm going to summarize the transcript and then you can see how that works and how useful it is.

So this doesn't necessarily need to work right but it gives me a bunch of stuff to look at and try to Google maybe. You know I can just try it out and run it. But then I'm like okay well where did it kind of fetch this from. You can probably Google it because it's in the training corpus.

And that gives you like a good start of either a lot of boilerplate that you want to change right. And then maybe you're like okay well I know how to do a timer interrupt I will replace that with my own. You can edit the LLM's response to put in your timer interrupt and regenerate and it will be like oh this is how you write interrupts.

I'm going to write interrupts that way as well. And you have a really efficient thing because it's still two interactions with the LLM right like fast inference and stuff like that. Summarize this as a read me for my colleagues. And then this is like high value when you work on a team and you can just be like look this is how my artist works.

I would never have written it by hand. Documentation of that quality right. You can say like add examples list of caveats only output new text. I will make many typos the model doesn't care. All right. Your turn. I'm watching the slack. If you want to share ideas of stuff to build like feel free to do it so that people who are you know early morning.

Yeah. But can you later on asking the slack that way I don't have more people can do it. But go ahead. I just wanted to say that it works nice for mermint the charts. You can create the charts of your . Yeah. Oh nice. One more generic technique. When you talk about doing something just type that into the LLM and then we'll do it.

It's like programming at the speed of mouth. All right. All right. And then. All right. All right. And then. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Could you spend time to join the Slack channel? Oh, yeah, sorry. So the Slack channel, to join it, the Slack channel is called LLM working programmer, workshop LLM working programmer.

Sorry about that, maybe I'll just leave this light on you have the handout, maybe, or, and I'll just mess around in a different tab. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.

Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thanks. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.

Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. I'm going to take a minute showing what I did.

Thank you. Thank you. In Slack, so that everybody can see them or make a pull request to the repo. And for the sake of time, because we have 50 minutes left and I have a lot to show, let's just move on. Is there any blocker? You all good? All right, so this is the main thing.

Because no one knows how LLMs work, no one knows how LLMs work and we never know how LLMs work, because LLMs are basically these weird computational artifacts that have been trained on whatever gigantic corpus we've trained them on, which no one has the time to read, let alone understand, right?

So no single human will ever understand what is the model, you can just kind of poke at it and connect it to what we know, and so in a way it's not like a computer technology or a math technology or whatever we want to call it, it's like a cultural technology.

It's like based on human culture and if you know your way around human culture, you'll be able to squeeze out more of the model, not just programming, right? So it's all about language, the model really doesn't care if it's programming language, if like a formal language, or if it's like German or symbols or whatever, because the only thing it does is transform tokens into tokens and it looks back at its own tokens doing so.

So even if we say like, oh please output JSON, it's like, I'm going to output numbers, probabilities of all the numbers that exist, but so if you make the switch away, right, like you're writing code, but like there's no reason that what is in the code isn't actually also like language, suddenly it makes a lot of sense of calling a variable like year over year revenue and not call it A, even though as programmers we think like, ah, that doesn't really matter, right, like, but actually the model, it matters a lot.

Because one is like, the token A, the other one is the token year over year revenue, and then suddenly it's able to access all the like, ah, finance literature to do, to write your code, even though you think that's like just, you know, um, a stupid dashboard, so understanding that means that suddenly things like theater, poetry, marketing literature, like textbooks, mathematical formulas, all of that is like fair game to write code.

So for example, theater, I bring that up as the first one, the models are trained to be nice, right, like they're, they're re-instructed to be like, yeah, I, I, I understand, like please, ah, please tell me what to do, I'm, I'm happy, and you lose a lot of conflict, which is useful, like when you do a code review, I don't want someone to be nice in the code review and give me platitudes, or like, you know, like, shit sandwiches.

I want something, I want something that just like challenges every line I give it, and so if you say like make a theater play, code reviewing my code with like six different people just like arguing about it, then suddenly in one inference you have like a really good code review instead of someone telling me to handle my errors.

So that's kind of the main thing, right, it's like thinking of them as culture, not as technology, ah, or as cultural technology, ah, and so the way to approach is, I think of them as translation engines, right, like there's many, many different kinds of languages, like when I talk to my mom, I have a different language than when I give a talk, I have a different language than when I write Python, I have a different language than when I write Python, I have a different language than when I try to write like a math paper, um, and so some of them are like from less formal to more formal to more formal, Once you step into the formal language world, you can use a computer to interpret it or do stuff with it more easily, um, but you can also, you know, transfer from one domain to another, you can go from the T-Rex domain to the real-time operating system domain or mix both together, um, it's just tokens, right, um, but so what that means is writing software now is about decomposing the problem that you want to solve into language translation steps and then let the model do these language translation steps, understanding which of these steps are too much for certain model, or how to do them, like how to prompt it to say, transform, you know, when it outputs JSON, it doesn't output the right JSON, you might put in like, this is important for your career, that's like kind of a cultural technique to get it to output JSON, which, which is a little bit weird as a programmer, um, so here's a bunch of examples, and like, I'm not gonna go through every graph, but like, feel free to look at them in the handout, um, this is, for example, an example of how to transform, for meeting transcripts, or like on Zoom, or you have like your software planning meeting, everybody interrupts each other, it's like a big chaos, no one's gonna go back at that transcript and try to understand what was discussed, uh, but now you can, right, and so the steps to do that, because if you just ask, like, tell me what we discussed last week, they're getting good at that, like, like, it's a little bit scary, but also, it used to really work not very well, 'cause you, you have like, 28 pages of transcripts, with a lot of, like, just people interrupting each other, and using colloquial language, so the different steps to do it, that I like doing, is like, first I wanna get a summary, because Claude has gotten really good at, like, figuring out, actually most of the information that's in there is a bullet point list, it's not good at details, I just want, like, which topics did we discuss?

Bam, five bullet points, then I take this translation step, right, from transcript to five topics we discussed, and then for each topic I go back and I select, for each of these topics, tell me exactly what we did, and now it has, like, a direction to know what to grep for in this huge transcript, that's only relevant to it, so you get this, like, pretty good result.

People call that, like, there's plenty of papers about it, I think, right, like, with complicated names or whatever, but technique is really just translation steps, it's like, from a topic to get all the details to that topic, no need to have, like, really fancy names about it, no need for fancy prompts, you just go back, paste the summary, and then for each point in the summary, like, give me the details.

Then the next step is, once you have these technical details, and you take the transcript again, you say, like, oh, give me action points, like, who's going to do what, which we discussed in the planning meeting, and just give me a list of, like, oh, Enrique's going to do that, and Sarah said that maybe we should do this, you know, and you get, like, a clean list of action points for each of these, like, fairly detailed technical summaries of each transcript point.

If you do that one shot, there's no way that that will work. But if you decompose in these, like, little translation steps, then you get it, and at the end, you can just take the action points and the technical details we discussed, maybe paste some of your code base in there, and create the GitHub issues, which already have, like, a plan laid out, they're assigned to the right person, you can ask to add tags to it, the way you do tags, like, name them correctly with whatever ticket naming convention you use.

So that would be an example of, like, a workflow for transforming transcripts into GitHub issues. No need to really automate that, like, it's just a list of things you can paste, and then in the conversation, if you're seeing it veering off a little bit, you say, like, oh, no, please tell me what Enrique said about XYZ, I remember it.

I don't want to script and, like, kind of formalize that, just put in transcript tomorrow, I'll forget about it. So, I have a couple of scripts to help me do prompts, and I'll get back to those, but I zero-shot everything, like, haven't done agents, whatever people call agents. So that was an example.

You can do other techniques, right, like, take this meeting transcript and the existing docs and code, and just say, like, oh, we talked about refactoring this thing, and now give me a concrete plan. Like, tell me which API should be renamed to what based on what we discussed. I don't want to do that by hand.

It's like I need to look for the functions or whatever. Just paste all the names, say, like, how do you rename them. Write the script to rename them with said, you know. You can add in, like, the technical details we worked out, right, from the summary. We had all these, like, oh, we discussed this way.

You can maybe put in, like, your RFC guidelines. You know, this is how we write RFCs, and then you get an RFC out of the refactoring planning that we discussed in the meeting. But with the right function names, maybe with a script or an actual refactoring that you can put into a test branch, for example.

And then you have an RFC that people can discuss, right, and then you can go back. RFC comments, just put them back with the RFC and the technical details in the summary and just, like, update the RFC. So this one's a little bit more meta. And I have to see how much time we have.

This one is, like, once we have the GitHub issues, right, from this transcript, I can formalize them, because just, like, output them as YAML. Like, make me title, tags, assignee, ticket number, link to your whatever. And then you have, now you have a formal representation that a computer that we can write programs for, right?

So then I ask it, like, oh, make a shell script to just take this YAML and create the issues for real, like just use the GitHub API. Models are usually pretty good at zero-shotting that. If they mess up the GitHub API like they used to do, just paste the API in there.

Then it will usually work. And then you can take this script, which is pretty simple, but iterate on it, right? Like, add a help, add a readme, add more flags, maybe confirmation, add, like, colors and little emojis, make, like, an NPM package for it that everybody can install. That's all, like, little translation steps going from a JavaScript to the package JSON.

It's, like, not really reasoning. A model can do it great. But when I do it, like, it's really annoying and tiring. And then I'll, like, forget that you have to write, like, ID uppercase and ID lowercase, and it costs me the afternoon. So, what you can do in three minutes, right?

Like, because I was talking about it, but if the model, if I just pasted it into the model, I would be there. I would have the script. It would probably work. So, now, you can actually talk, okay, now that we have the script, how do we change our workflow within the team, within the company, within the open source project to just automate it, right?

Like, we have a meeting transcript where it's, like, create GitHub issues, boom, it's all kind of automated. You maybe want to look at the YAML before you press fire, right? But there's a decent chance that at least you get a lot of boilerplate and you can replace, like, a couple of LLM fringe-speak.

What you can do now is just, like, take the script and the Slackbot API and say, like, make a Slackbot for this. Notify the people that have been assigned the tickets. You take your GitHub transcript, paste it to the bot, and then everybody gets notified of their GitHub tasks that come out of it.

And then I did some meta-meta stuff, which I'm going to skip. If we go back to this workflow now, right, which probably took us, like, maybe an hour to build all of this. A good chance that, you know, if it's annoying to get, like, the Slack access API token, or if you like some OAuth stuff, you just, like, write me a script to get a Slack API token.

I really don't want to look up how to do it. Just give me the URL. And, but what you can do now is that as part of these prompts that we give it, every individual developer that you have on your team, they can, like, put in their preferences, right?

They could say, like, well, I don't know about this library, so explain it a little bit more when it goes about this thing. I like bullet points. You know, my background is this and this and this, but currently I'm busy with this other part, so only tell me about parts that overlap.

You can, that's just, like, a TXT you can have on your GitHub, right, or somewhere, but every time the Slack bot, for example, is going to create the GitHub issues assigned to you, it's going to put this thing into the context, and then you will have, like, tailored documents that explain how to do something.

So, you know, if you have the intern, and you know the intern doesn't really know much about APIs or, like, HTTP, you can just put in their prompt, like, explain HTTP, as it relates to this ticket, or explain our internal infrastructure deployment script. Onboarding, super easy, just have, like, an onboarding TXT that's ready to go, explaining, like, the architecture of your code base, just put that into the new developers' GitHub issues, and make them match, right.

Easy, it's just language. So, for the sake of time, you can go through some of these. Maybe I'll show, one that's really funny is, like, do you want to write API clients? I just don't even look at the docs anymore. It's like, I record, say, I'm in a browser, and I click on things, and I want to write, like, an API for the cloud UI, which they don't want me to do, but they have.

I just record the entire session, clean it up a little bit, and then just say, like, well, write the documentation for this mass of HTTP requests that I have. And once I have the API documentation, I'm just like, well, write me, like, a proxy. And then I have a proxy for cloud AI's newest thing.

And let me see. This one's great for learning. You have a topic you want to learn. Get the search result. Existing knowledge that you have. Maybe some code base or preferences that you want. Create a wiki entry for your personal knowledge base. So you want to learn about advanced TypeScript, Lambda things.

Just look for it. Take your existing knowledge, saying, like, my background's in Ruby. Give me some exercises and project ideas to build. Maybe use the guidelines for my company to format these exercises so I learn two things at once. Create a prototype. And then maybe once you have a prototype, you just, like, paste your actual API and select, well, just make it real, right?

And you get the actual implementation. But those are all, if I just said, like, you know, solve the problem and teach me how to do this, it's not going to work. But if you decompose it to these little steps at each point, not only do you get value of doing it because you see the output, but you also get, like, artifacts you can share, for example, right?

Like, once you explain how Lambda TypeScript stuff works and your colleague also wants to learn it, you just give them the exercise of the transcript. You can help them. Maybe you put their personal prompt into it and their background's, like, in Lisp and then it will just update it.

All right. So you can take some of these graphs in the transcript. I'm going to give you, like, ten minutes really sharp so that I get to the mind-melting stuff. There is a section about DSLs. If you know about DSLs, if you don't, you really should look it up.

LMs are great at creating a language, so you can actually create the target step itself. You can say, like, well, I want to write reports, just, like, create a language to write reports. And then you can target that and, you know, implement the interpreter for that language. So I really -- there is an example set of prompts in here about DSLs where I create, like, a text adventure game.

I did that, you know, this morning at breakfast. Try those out. It's, like, on page -- page no number. But you'll find it. Creating adventure games, three lines of prompt, and I create, like, an alien-inspired adventure game with, like, a full source code and stuff. So, yeah, try stuff like that.

A good target is also self-contained HTML plus JS plus whatever your favorite library is. There's a couple over here, and then if you have neighbors, I recommend just working with your neighbor as well. Like, sharing the handout and communicating. It's, like, yeah, I think that was the last handout.

It's also as a PDF on the repo. If you want to see the repo again, I guess it's in the Slack channel. So, the Slack channel is LLM workshop working programmer. It's a great way to see the chat. It's a great way to see the chat. It's a great way to see the chat.

It's a great way to see the chat. It's a great way to see the chat. It's a great way to see the chat. It's a great way to see the chat. It's a great way to see the chat. It's a great way to see the chat. It's a great way to see the chat.

It's a great way to see the chat. I'm going to accelerate a little bit, like four minutes. You should be able to write like five programs in that time. I'm serious, right? Like try out my prompts and just replace create a YAML DSL for the program you want to write.

you want to write. Thank you. It's called workshop LLM working programmer. If you're in Slack, are you in the AI engineering Slack? Yeah, just look for workshop and then you should find it. Yeah, it's a channel. Yeah, actually, let me put that back. Here's the Slack. Here's the GitHub repo.

Hello? Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you.

Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. Thank you. I'm moving on because I've got some really important stuff to show you. And one is, so Josh, no, Brenton just asked, like, have you found LMs able to work with PCAP files?

I'm like, not really, but like create a program that outputs the PCAP format in a format that, you know, in a language type that the model can understand. So I said, write a YAML DSL to represent interesting DNS traffic out of the PCAP file. That looks pretty useful, right?

Like, this is language that the LLM can do well with because it's called DNS traffic, destination IP. It's not called like byte 0x5 or like DNS underscore, I don't know, whatever. And then I'm just like, write a Python file that takes a PCAP file and outputs that YAML. And now I have a script that I can give every PCAP file, outputs that into something that DLLM can use, and I can put it into my LLM agent loop or whatever.

That's what I mean with 10X engineering, right? Like, in the three minutes here, I wrote another one because, so, what Josh was asking, right, is like, oh, when I put in code base in my thing, like, what does that mean? It's like huge. But usually when you interact with a code base as a human, you either interact with a small, certain part, you read some docs, you maybe look at the function APIs.

So just create a YAML DSL to, like, grab your code base for interesting stuff, right? So maybe I want to get all the functions in Python files as well as the classes. So I said, like, invent a YAML DSL to create a program that finds functions and classes in my code base.

As well as the doc string. So it outputs this. I was like, yeah, that's a cool idea, right? If I don't like the idea, just regenerate until I have a language that matches what I want. Here it added, like, recursion, which I like. And I was like, well, I also want, like, the markdown files.

So I'd, like, made up, you know, two things for the markdown. And then my third prompt is just, like, implement it, please. And then now I have a Python file with a YAML that I can tell how to look for stuff in my Python code base. And then, well, now I can run it on my massive code base.

And I just have the function titles, the doc strings, the markdown document titles. And that's a good context to give to my DSL, right? Like, took me 30 seconds. Might maybe not work too much, right? Like, in real life, I would say, like, this is an afternoon project, maybe, to make it really nice with the readme, with unit test, with, like, examples, and install package, whatever.

There is a graph on how to do this. Like, this is maybe one of my main techniques, is, like, writing these little fragments that I know are good context to do a certain task. So some of them could be, like, oh, write a new widget to do something. And then I'll actually add, like, grep for these widget names, maybe add the documentation.

Out of the Git log of last week, I'm going to create a how-to, and then that's going to be the how-to that I'm used for prompting. And now I have, like, a little script that I can, every time I want to write a new widget, now I just paste the result of that script in, so that's the life status of the codebase.

And then, usually it does pretty well, right? Like, if you give it, like, first write the props, write this and write this. So this thing, and I do this for third-party libraries, right, which often don't have, like, good documentation. I'll go over the codebase and just generate good documentation that I just need for, you know, for the task at hand, and then it takes me, like, a minute, usually.

Sometimes it'll take me, like, the whole day. I'm like, what am I doing? But, um, pretty good technique, there's a graph on how I do it. Um, and now I'm gonna quickly switch to the wild stuff. Um, so I went over DSLs. So, word simulation, right, like, everything you tell the model is what it thinks reality will be, or whatever it uses as patterns to grab its training corpus.

If I tell it you are a wizard, it will pretend to be a wizard. If I tell it to write poetry, it will write poetry. If I tell it, like, time travel exists, it will say, like, sure, time travel exists. So anything that's kind of in the training corpus, so things that humans can think up so that you can think up, the model will, like, usually know what to do with, right?

That also applies to, like, formal languages. If you say, like, your world is, like, def function paren, whatever, and it's like, yeah, cool, def function paren, that's my world. Um, so, one thing you can do, for example, for code reviews, right, is, like, I tell it you're a reality TV show, code survivor, and, like, each participant in this reality TV show is one of my variables or functions in this piece of code, now have them, like, fight each other and figure out who's the bad one, right, and then vote them out, like, who's got the most bugs, and, um...

So this is based on the concept, like, whatever you tell it it is, I'm gonna use what's in the training corpus, like, reality TV drama, um, so I'm gonna write some bad PHP code, 'cause all the code I write is great, so, um... And I have a GPT for that, that I shared, um, but really, you can, like, write is the idea, it's not the prompt, uh, I don't really do any prompt refinement engineering, except when I want to repeat something in a, in a loop with a small model, um, I make typos and all that stuff, so now I have, I have this, uh, I have this bad PHP, right, and I'll just paste it into my code survivor thing, and this has multiple, right, so the contestants are, are, like, different variables, and I told in the format and the prompt, you know, just, like, each one gets to say one thing and just, like, attack someone else, identify the conflict, and then, there's some kind of format in there, and, and one of the advantages is, like, A, I get conflict, right, even though the model has been trained to be nice, this is, like, standard red team tactics to just exploit LLMs, this works really well to do this stuff, just have it pretend to be a different world.

It allows me, back when ChatGPT was, like, really annoying and said, like, please fill in the rest, dot, dot, dot, um, which it doesn't do anymore, this would allow me to say, like, make six rounds, and it would do six rounds, right, like, it was so strong in the prompt, that I could, in one inference, right, like, compare that to code review my PHP, this has already a lot of concrete information in it, and I can, if I don't like it, like, write three more things, and there is the comeback participant, and it doesn't need to be a variable, right, it's like, uh, uh, SQL injection, it can be a concept, like, concept can be a character, it can give them, like, a personality, um, and then SQL injection will start to be, like, dissing other people, which is pretty funny, um, but it's useful, right, like, this is proper engineering, it's not, like, it's not fun, it will actually tell me, like, you should use MySQLiClose when you finish this, and you forgot, like, I'll vote you out, um, and so once you have all this garbage, which I don't even really need, it's, like, funny to look as it scrolls by, because they're like, hey, we're just, sanitation is not our job, whatever, um, at the, oh my god, this just goes on for pages and pages, which was really hard for GPT to do back then, like, it would be like, oh yeah, I think this variable's bad, like, please handle errors, and I'd be like, what?

Then you can do, like, write a sober code review report at the end, um, just based on everything that's in the context, like, now strip the reality TV part out of your language and just keep, just keep that stuff, right, um, and actually suggest fix, and I could then turn that into GitHub issues or whatever, uh, you get the idea, um, but so this is the world simulation thing, is that you can really make everything up, like, one technique that I have is time travel, because when you correct the model, it's, like, useful information, right, it tells the model what not to do, but if you have 25 messages telling you not what to do, it will confuse it a little bit, so I'll do my 25 messages, I'll get to the right code at the end, I'll select summarize what you did, and then, ah, thank you.

It will be, like, um, summarize what you did, then I'll rewind the 25 things, and I'll paste in, like, this is you from the future, this is what you tell your past self, and now I have, like, this pretty good, and it knows what to do with time travel, people coming back from the future telling, like, your young version what to do, and it, like, it's able to generate better code out of that, so, weird engineering concepts, um, so try it out, like, it, this is not, like, this is not fun, this, this actually works, right, um, presidential debate is great, uh, Greek drama is pretty funny, um, but where it gets really wild are prompts, like, you are the application I want to build, please start, right, and now suddenly, like, going to pretend to be what you want to build, and so you can tell, like, like, please output your UI as, like, a concise DSL, and it's like, well, I've got, like, a main widget, I've got, like, a text area, and I've got this, and I've got this, that's formal language that you can already, like, visualize, um, but also you can be like, well, no, I want a sidebar, and it's like, yeah, sure, you have a sidebar, uh, so, like, click the third element of the sidebar, and we'll pretend that I clicked the third element of the sidebar, right, um, so just to show you, uh, you are an application to plan flight, travel, and sightseeing for me and my kids, output your UI as a concise DSL YAML, start, so, you know, regenerate if you don't like it, if you want it to be, like, react widgets instead of, like, this high level description of what it is, uh, feel free to do that, like, you can steer it, right, can be like, no, I don't want, like, my data, I want, like, actually the UI here, I don't want to rewind it, I'm just going to correct it, and then once I have the right thing, I can, like, gaslight it and say, like, this is what you generated, like, uh, no, I want some HTML markup kind of YAML DSL with the actual UI, um, zero prompt engineering, like, I don't know what I'm doing, but, you know, this is much more closely to what I had in mind, it's like, oh, I want the actual widgets, um, and if I were to, you know, write a JavaScript that parses this, and then as callback handles, just, like, output press, I press the button X, right, it's like, uh, if I want to do, like, open the tab for day two, I don't think there's even tabs for it, but it'll just be like, oh, yeah, I've got tabs, like, of course.

Um, and so you can interact with it, and it's kind of like, is this code, is it not code, but at the end, you can say, like, oh, write a spec, right, it's going to write, like, a concise readme kind of spec, like, and it can be like, implement yourself.

Um, which, this is new, I haven't really played with it too much, I came up with this two weeks ago, but it's kind of wild, because I'm into film photography, I wanted an app with, like, three timers telling me what chemicals to mix and what, and I built it in five minutes, and I was like, this is wild, like, I just said, like, I want an app to help me develop film with this process, and, like, filled it in with the right chemicals, and all, and I was like, no, I want, like, two parallel timers that, you know, loop, and I was like, yeah, sure, and then, at the end, I was like, write a concise, had him help, JS, self-contained prototype, and then I can pass it on to the graphic designer, or maybe the graphic designer does that all day, right, and he's like, ah, no, I want a sidebar, wouldn't that be nice, and then tomorrow, he'll just send me the transcript, and I'll be like, okay, I'll make this work for real, so there we go, this is, you are an application, I showed you the code survivor, this is your turn now, five, six minutes, maybe, just, like, invent an application, play with it, like, you're an application, use a concise HTML markup, YAML, DSL, start, and then see how far you get.

TARGET, TARGET, TARGET, TARGET, TARGET, TARGET. TARGET, TARGET, TARGET, TARGET. TARGET, TARGET, TARGET. TARGET, TARGET, TARGET. TARGET, TARGET. TARGET, TARGET. TARGET, TARGET. TARGET, TARGET. TARGET, TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. TARGET. I'll actually interrupt this, I think afterwards you're going to have fun with this, but I had a really interesting question come up, it's like how do I manage all of this, right, because I don't know why every LLM company doesn't manage to make a decent history browser, it's really not that hard, I could generate one in two days probably that has tags and allows me to search, but no one manages to do that, I don't know why.

So what I use, and I would recommend not using my tool, don't take my tool as the thing, I break it off and I tweak it to my things, but just build your own, the tool to manage fragments is like you're managing text files and how to paste text files together, or maybe manage shell scripts, so my most used tool is called Prompto, I literally generate it once, it's a 100 line script, it will go over all my repos, I have config files where I say look into all these directories, if it finds a text file, or a script, or a YAML template, it will just list these, it's horrible, it has no pagination, it's whatever, but so once I have like a context, right, for like my library, when I want to do something with parameters, I have like a little script that gives me the current parameter API, and I can call it, right, it's like if I do Prompto get glaze parameters, it will find, it will look into all these repositories, like super inefficient, no indexing, no caching, it will find this parameters MD file, and it will just paste it, or actually it grabs the Go thing and just outputs, it's super ugly, right, there's no structure in this except dash dash dash, but so now what I can do, I have another tool that just, it's really just like a command line LLM prompter, it's like HTTP request with no format, and I can pass it, this as a context, right, I've got a couple of like pseudo things, which I wrote two years ago, a year ago, they're not great, I can print a prompt that it's going to output, and so like, oh, create a parameter for URLs, and so it will just paste everything together in like a big blob, and at the beginning it says like, you're a great, you're a great Go developer, actually I messed it up, it actually says PHP, so whatever, and then it just like literally says like create a parameter for URLs, right, in Go, and then if I actually run this, it will have, it will use my API, it will do a good job, like this is how I just add stuff to my code, it will have the right packages, I can literally just paste this, and it will probably run, and once I have this prototype, then I have other scripts to like integrate this into the framework, like add the documentation style I want, and all of that stuff, but so this tool, pasting strings together, and just like finding scripts in a directory, they're like nothing scripts right there, so that's how I manage all my fragments, and we have eight minutes left, so I'm going to skip to the final part, which is more experimental, it's like what are these, like you realize now that I'm like kind of not joking with this 10x, because during this workshop, I wrote like six, seven decent programs, right, they maybe don't work too well, but like you can guess that they're not far from something that's of decent quality, and then if I want to add unit tests and whatever, I can, you know, take an hour at lunch while watching Netflix to actually do it, sometimes it doesn't work, so this is like a big part of it, is like no one to step away, don't, right, if it doesn't work like within 10 minutes, don't force it, either do it by hand like you used to, it's fun, like we all love programming, maybe go for a walk and come back when you realize which language step it didn't understand, that's a really important one, I still get into full rabbit holes, where I just like don't get the task done that I actually want to do, I just like generate programs that don't work to solve the task, and it's, so knowing when to step away, because it's like breathtaking, right, like I generated seven programs, like what am I going to do with that, like I can't mentally manage seven programs within a time span, like I can't do it, and I really have this problem, it's like I have like a, an experimental repo where I put in all my scripts to kind of, I don't even, I don't even bother putting them in, right, but I have like so many, and some of them are like 5,000 line code things with like crazy features, and then I forget that I wrote them, and I write them again, and I go back, and it's like oh, you already did it, it's like shopper approved customer management things with like parallel worker queues, and I, I wrote it three times, oops, sorry, so, what however doesn't go away I think is like fundamental and practical knowledge, like I know how a real-time operating system works, so it makes it really easy to see if the model doesn't get it, right, like I'll be like no, let's, don't do it in assembly, like do it, use this pattern, or use this framework, if I didn't do it by hand or like learn about it, I wouldn't be able to do it, it would go into a big rabbit hole, I wouldn't get an RTOS, but that's the difference between someone who knows RTOS and generates a new one with an LLM, and can maybe do it in an hour, and someone will just like after a week just get nowhere, nothing works, nothing compiles, the assembly's broken, so fundamentals, super useful, what you can't totally forget is APIs, right, like I really don't care about Amazon's cloud formation API to do XYZ, like I really couldn't care less, it's not knowledge I want to take with me into my retirement, and that's gone.

But however, the knowledge of how to deploy, you know, to do Coda's infrastructure for functions as a service, that's like really useful pattern knowledge, and I can take that knowledge and apply it to like DigitalOcean and to like Amazon and Google, what I don't need to do is like figure out after 10 minute deploys that I forget to put like an IAM rule somewhere, like I really, I really don't care.

Some people do, and those are going to have a hard time I think in the next few years. While practice, I already went over it, but as soon as you think about humans in the loop, right, like, because language is always going to be useful when interpreted by a human, I don't care what the LLM, it could generate like 10,000 programs that don't work, and like what are you going to do with that?

So everything you generate with the LLM, either it's like to kind of help the LLM along to finally output something that's useful to humans, or it's directly useful to humans, right, like a readme is useful for humans, it so happens that because LLM is trained on human language, it's also useful for an LLM, but if you make it nice for humans, it will work well with LLMs as well.

Like if I have like, if I go on for three paragraphs about how error handling is useful, like that doesn't matter very much for my senior software colleagues. So I can strip it out and now have a smaller prompt and it's like going to be more focused. So if you always think in this like language decomposition things of like, how would it work for humans, how would it work for the user, what do I actually want to do, that's like a good engineering skill.

It was already before, right, but now it's actually in the small is also pretty good. Divergent thinking, I don't know how to teach creativity or whatever, smoke weed. But like thinking about, like these weird things of like, well I'm going to make like a reality TV show about my code review.

I don't know necessarily how I came up with it, but it's the best code review prompt I ever found, right. It's, it's, I don't know, I don't know how to come up with that stuff. And what's useful there is to look at people who are not programmers because they have amazing prompts usually.

And programmers are like, we're like very focused on like, oh we know how to do it and it's like it should write code. And then someone comes along and it's like, you're an alien that writes code and it actually works. That's why people who have no programming knowledge are able to build like full apps while a programmer comes in.

So it's like, well I forgot to do the for loop check and this is worthless. And in the meantime the non-programmer is like, well I wrote an Android app that like allows me to fill my daughter. And abstract thinking, I think a lot of it is like, once you build something, knowing how to build the thing that builds the thing.

Or find like the deeper abstraction of it and then focus that abstraction, I think is pretty useful. I, my background is in Common Lisp a lot. So there's a bunch of languages like Common Lisp, Ruby, Haskell that are very focused on like creating abstractions. Manipulating the language that actually solves the thing.

I think that's like a pretty interesting thing. Not that you should write Common Lisp or Ruby, right. But it's like getting familiar with the concepts that make up the community around that language of creating compilers. or like creating like dynamic dispatch DSLs. Or finding like mathematical abstraction of how to do control flow or so.

That, that's really useful and that kind of goes into language design. It's not just like programming language design. It's like any kind of language design. It's like our head, like my favorite format is like title, one sentence, code example. That's like a language design thing. It's a, it's how I like to have my docs.

So that's it. We have two minutes. Maybe there's some questions. I can, I guess I can take live questions and then repeat them maybe. Oh, there's a microphone. Great. Thanks so much for the great session. It's amazing to be here in a room with like-minded people who will be probably geeking out on all these things for so long.

And thank you for sharing your, your findings. Couple questions real quick. One of them is, are you finding yourself working with other people in your team? Kind of spearheading some of the incubation portions of work following these techniques. And then the cover, the cavalry comes behind you to build the actual thing.

Have you found yourself that kind of role? And then the second question is, I see you can push yourself going really far with all these techniques. But how much are, have you tried using some of the auto GPT kind of metaphors as well? I'm curious. Thank you. Um, so I just recently joined a team.

I was like kind of solo for a long time and it's hard. I don't know. Uh, it's, I, I don't know how to bring people on board with like how I do things where I'm like, I write 5,000 lines of code an hour and then I throw them away.

Um, I don't really know how to bring that into the team currently. And then for the second question, I, I guess that's going to be the last one. Cause I, I think the, the time is pretty strictly enforced. Um, what was the second question? How much do you rely on auto GPT or those kind of things?

Oh, zero. Um, I just like zero shot stuff and then I'm in the chat API. So I don't, I don't use, I, I use copilot for, you know, like just as, as copilot is, it's a very different thing. Um, and then that's it. That's a wrap. Cool. Thank you.

10x Development: LLMs For the working Programmer - Manuel Odendahl

Transcript