in recent weeks there has been a lot of turmoil surrounding ai technology it increasingly seems now like those grand promises of super intelligence or ai systems automating most of the economy these grand promises are probably not going to come true but what about the more practical promise the one that ai tools are going to make knowledge workers more productive especially if you do something like computer programming for which ai is well suited that's still true right well the answer turns out to be complicated a recent study about ai productivity yield a completely unexpected result it created a glitch in the matrix so to speak that leads to some deeper truths about this technology and its potential role in our work today i want to explore all of this if you're at all interested in how ai might impact your job in the near future this is an episode you won't want to miss as always i'm cal newport and this is deep questions today's episode a glitch in the ai matrix all right so i previewed there was a study that caught a lot of people off guard i want to load this on the screen here for those who are watching instead of just listening this study came out in july of 2025 it was produced by a non-profit called metr that's often pronounced meter it's a non-profit that does evaluation of ai and its capabilities they do high quality studies they don't have any particular bias ai companies often cite their results when they find them to be useful or impressive so this is sort of a neutral body that does these reports here's the title of their july report that caused a bit of a surprise measuring the impact of early 2025 ai on experienced open source developer productivity so what did they do i'm actually going to read from the methodology section because i think it's important to understand like exactly what they were testing here so let's read here together this is from the paper to directly measure the real world impact of ai tools on software development we recruited 16 experienced developers from large open source repositories that they've contributed to for multiple years developers provide lists of real issues that would be valuable to the repository bug fixes features and refactors that would normally be part of their regular work then we randomly assign each issue to either allow or disallow use of ai while working on that particular issue when ai is allowed developers can use any tool they choose primarily cursor pro with clod 3.5 or 3.7 sonnet which were the frontier models at the time of the study when disallowed they work without generative ai assistance developers complete these tasks which average about two hours each while recording their screens then self-report total implementation time they needed all right so that's the setup it's it's a very elegant experiment here's developers working on the normal stuff they develop and issue by issue hey for this issue i want you to use ai for this issue you don't for this issue you don't use ai you do right so you get this nice randomized control of all the developers are both using ai and not using ai in a randomly selected manner the simple thing they did want to figure out is how long did it take to do these tasks when you were using ai versus not now here's where the conventional wisdom says computer programming is something that language models do well so of course it makes productivity go up i'm going to load up on the screen here the core chart of this whole paper so if you're looking at this what you're seeing right now is a bunch of green dots so what these are showing the the things you can see the data points you can see on my screen right now is predictions from various people about the increases of productivity that they thought ai would give them so if you'll notice there's a line in the middle here that's neutral so a data point on that line means ai makes no difference if a data point is below that line it means ai slows you down if it's above it it means ai speeds you up so the the first point is from economic experts they ask economic experts how much more productive will this make these programmers in this experiment and on average they said uh around the 40 speed up they ask machine learning experts so people who knew the technology they were similar yeah it should be about a 40 speed up they asked the developers themselves during the study like hey how much more productive do you think this is making you and they were like yeah like between 20 and 30 more productive and when they asked them after the study was over they were you know around the same place like a little bit over 20.

what was actually measured i'll scroll up to show you that result the observed result is on average they were about 20 percent slower than the people not using ai so the ai tasks were slower than the tasks in which the programmers did not use ai this was an unexpected result they thought they would just be measuring how much more productive ai made these programmers they were unexpected to see uh that it makes them slower so this is a bit of a mystery and it's one we're going to look into because it gets to something core about ai and doing knowledge work why did programmers despite their predictions become less productive when they used ai all right so if we want to solve this particular productivity paradox i think what we need to do next is understand what type of work are these computer programmers doing we got to get specific here now i'm going to use a term that most of my audience is familiar with computer programming handling these tasks which are a combination i looked into more deeply but it's a combination of creating original code or fixing code they require what i call deep work deep work as you know is a task that require you to focus without distraction on something that's cognitively demanding so deep work is when you give your full attention to something that is demanding and you try to keep your attention on it as intensely as possible i sort of wrote the book and it's sort of i wrote the book on it i coined the term it's been 10 years jesse not hard to believe yeah in january it's going to be the 10 year anniversary of deep work and i guess i'm going to follow through with the promise i made my readers that if this book is still selling after 10 years a face tattoo boom like mike tyson like mike tyson deep work right across the forehead ryan holiday put his book title on his forearm he's going to one-up him right across the face that's what a real committed author does uh so anyways here's the type of work we're talking about when we talk about computer programs deep work right it's something that requires uh focus you're creating something complicated from scratch using your brain is kind of demanding and it benefits from being able to give it undistracted focus in the modern knowledge economy the core argument i make in my book on deep work is that this is the effort that creates most of the stuff that's actually valuable to the marketplace all the other things we do the emails the meetings the filling out the forms the submitting the things the tickets to it the making the powerpoint presentations that's basically all efforts that support deep work the stuff we have to do to keep the lights on and tell people about what we did but you can't run a profitable company just off of email meetings and powerpoints ultimately someone has to actually create something valuable using their brain and that requires deep work so in this study we had programmers doing deep work who used ai to help them and they thought that would make their deep work more productive but in reality it made things worse great so now we're narrowing in on the answer to this paradox so we know what these programmers are doing deep work and we know that ai wasn't helping them so let's narrow in this question even farther how specifically were these programmers integrating the ai tools they were using into their deep work efforts according to the study authors the programmers were working interactively with these tools so what they were doing was a lot of can you produce me this code i'll look at this code hey can you fix this about it or this is not working so kind of back and forth like that or here's some code you see any mistakes great then the the programmer would run it and be like it's still not quite working right can you try to fix it here so it's a very interactive loop they were going back and forth having code produced checking the code having the ai check the code looking at what it sent back so think of it as like a a back and forth interaction um here is how let me see here here is how the uh paper itself describes this let's see here okay here we go here's text from the actual paper about this collaboration when allowed to use ai developers spend a smaller proportion of their time actively coding and reading searching for information instead they spend time reviewing ai outputs prompting ai systems and waiting for ai generations interestingly they also spend a somewhat higher proportion of their time idle where their screen recording doesn't show any activity so the non-ai group when they're working on a task without a without ai is more time makes sense like actually writing code and the ai group is spending more time prompting asking for code asking for it to check code so it's more of this like interactive loop i'm going to give a name to this let's call this cybernetic collaboration because these programmers are collaborating on their deep work with a computer so it's cybernetic and they're collaborating let's call it cybernetic collaboration they're basically trying to split the cognitive effort of producing this code or fixing this code between them and this digital mind now intuitively cybernet collaboration should make you more productive why not right like you're you don't have to do as much deep work anymore because you're offloading some of it to a machine and machines are fast and machines are precise and hey this seems like you've just made things more productive but of course that didn't happen so now we've really narrowed in this paradox they're doing deep work they're collaborating interactively doing cybernetic collaboration with the ai while they're doing deep work and that is not returning more productivity it's still taking them longer to produce stuff than when they weren't using ai at all so let's narrow in even farther and let's step back now and ask the question what role does collaboration play in deep work is that something that you can do with multiple minds now this is actually an important question because a lot of people who had a glancing encounter with my book on deep work assumed that it had to be solitary but it doesn't actually i know a lot about collaborative deep work because this was one of the core skills i had mastered as a theoretical computer scientist like this is if anyone knows how to do collaborative deep work it's scientists that do math theory like me i learned how to do collaborative deep work first at mit when i was doing my doctoral work and then refined these skills as uh faculty computer science science faculty at georgetown i write in my book deep work about how to do it successfully collaboratively so yes deep work can be done collaboratively but how is it done successfully if you want to do deep work with someone else you want to collaborate in practice if you're hanging out with mathematicians how do we do collaborative deep work uh successfully well here's the thing you talk to any professional thinker who does this they'll say the same the reason why collaboration i guess i should say the way you make collaboration help with deep work is that you use the presence of other people to increase the intensity and duration of your focus so the underlying formula focus produces results deeper focus produces better results that formula reigns supreme so when i would sit down with my frequent collaborators to work on some sort of theorem or some sort of paper we're working on the whole game was trying to get even more intense focus here's how this works if i'm sitting at a whiteboard with two other mathematicians a couple things happen one i'm going to maintain my focus on this problem much longer than if i'm just alone looking at a notebook why because if i'm alone there's very little penalty for me allowing my attention to wander ah this is hard let me just like let my attention wander let me let some of the steam out of the metaphorical steam engine here let me go check something else nothing bad happens if i do it's just me trying to contain myself if i'm at a whiteboard with two other mathematicians however and i let my attention wander i lose the threat of thought and then what do i have to do a couple minutes later something that's kind of embarrassing i have to like hold on hold on go back i missed what you guys were just saying here because i let my mind wander everyone has to stop everyone has to go back so the social pressure of keeping up means that you keep your intensity longer the other thing that happens when you're working with other people is it pushes your intensity of concentration deeper right because you're sitting there trying to understand this someone has a breakthrough like okay hear me out and they go up to the whiteboard and they start working on some sort of like complicated simplification of an equation or some sort of graph construction and you're like man i gotta really lock in to follow this right someone is trying to download something complicated they just had an insight about to my brain and the only way to get there is to like really lock in and focus so it's a focus accelerator that is what you get when you're doing deep work i think it's off there now but jesse that whiteboard we had here in the hq for i used to bring like during the pandemic starting the pandemic used to bring my collaborators from hopkins and georgetown they'd come here and we'd work on the whiteboard yeah i don't know if it's still up there or not but like the last thing that there was so we would sit at that literal whiteboard and the last thing that was up there for a couple years we actually won an award for that paper so maybe i should have kept that it's a good paper but yeah that's what we do so uh there's other advantages of working with other people but this is like one of the big things you get out of deep work if you collaborate right is that it makes you focus harder and focus longer so again there's this underlying equation intensity of focus times time is how much you produce when you're doing cognitively demanding work that still reigns so collaboration all you're doing if you're doing it right on hard things is getting more focus you're squeezing more focus out of it i call that the whiteboard effect in my book go back to cybernetic collaboration that's not what these programmers are doing they are using collaboration with ai to reduce the amount of intensity of focus they have to experience to get those breaks you produce it i'll look at it and it's easier for me to try to debug what you did that was broken than it is for me to have to produce it from scratch it's easier for me to have these nice breaks while i'm waiting for the ai model to generate the code i'm going to look at which can take a little while we know this jesse because we get this question all the time now what should i do while waiting for the ai model to produce my code like we only put it on the show like once but we get this question a lot so that gives you a break so it makes the experience more pleasant there's actually the the authors of this say this somewhere in the study that it was a more pleasant experience for the programmers because you get all these breaks you don't have to be locked in when you're just looking at the bank blank coding page no word no piece of code is going to get there until you come up with it and write it it's really hard there's no real break until you tell yourself to take a break because you're trying to get all this code together compiling for small programs is fast too you don't get much of a break from it and so it's much more pleasant cybernet collaboration means much less uh much less intensity of focus much less duration of focus it takes less energy it feels nicer but that's why they're slower because intensity of focus is what tells you how fast you're going to go intensity of focus tells you how good the stuff you're going to produce is when it comes to deep work so the whiteboard effect says yeah come work with me so i can focus harder good stuff gets produced cybernet collaboration by contrast says i want a computer to offload some of the work so i get a lot more breaks but that means my brain is producing a lot less slower less quality stuff and that is why the time required to get the work done begins to go down now like look if the machine was actually able to take over all of the deep work that would be different if you could literally just say vibe code this program and then commit it that'd be great because now you're like i don't do any deep work at all but of course the machine the machine can't do that in cybernet collaboration they can't do that yet so now you're doing this back and forth dance where you ultimately still have to come up with the questions and edit the code and get everything to work but you're cutting down your intensity of focus so the whole operation inside your brain is going slower i want to bring up another quote here this is from an article from the atlantic that was written by roger karma and it talked about this study among other things the title of this article is just how bad would an ai bubble be and it really is about ai and productivity i'm quoted towards the end of this article it's a good article but here's what he heard from the meter developers about what was actually happening as they were doing this sort of cybernetic collaboration so let me quote here even the most advanced systems make small mistakes or slightly misunderstand directions requiring a human to carefully review their work and make changes where needed this appears to be what happened during the meter study developers ended up spending a lot of time checking and redoing the code that ai systems had produced often times often more time than it would have taken simply to write it themselves right so they were just in this loop of you riot i'll look at it you debug let me look at what you're doing but because they were avoiding the full intensity of focus that they're capable of because they avoided maintaining their focus without having context shifts or other distraction the work they were doing was just like not at their peak so now you you're doing all this work of cleaning stuff up and you're not operating at your peak this is the potential danger of cybernetic collaboration that when you downshift your mind let me downshift my focus intensity it just doesn't work as well it might feel nice but deep work doesn't really have a lot to do with nice all right so that's what seems to be going on there so some combination of this rhythm of going back and forth creates like a lot of work you might not have had to do before and more importantly it reduces the gear which your brain is operating so these programmers were just it was pleasant but they were producing you know stuff slower it was taking them longer to figure things out that's my one of my understandings from what is going on there uh but let's summarize this all the end of our deep dives we like to do some takeaways we've been experimenting with music our last takeaway theme music which i thought sounded like the beginning of a cable show about bass fishing a little aggressive so we're going to go a different way today for our takeaway theme music we're going to do something a little more you say jesse intellectual yeah calming all right so let's let's do our takeaways with a little bit of calming background music all right so what are our takeaways deep work rewards intensity of focus and if you add anything into your workflow that's going to reduce this intensity you'll probably get less productive this seems to be the trap that a lot of knowledge workers experimenting with ai right now are falling into focus is hard it doesn't feel pleasant it's tempting to try to make it go away but that doesn't make your work better now i'm absolutely convinced that there will be upcoming applications of ai that will help our productivity i think they will focus more on automating shallow tasks or speeding up things that don't require you to focus but cybernetic collaboration that is not the key to the ai future focus remains absolutely essential to doing deep work and for now it remains something that has to be hard to do i feel like we should slowly high five or something we did it so there we go there's a lot of other things going on in that study but i mean i think that's a lot of what's going on to it it's like look deep work is hard it's tempting to make deep work easier ai can help you do it but doesn't mean you're going to produce more work i love the takeaway portion of the deep dive segments now i'm working on it i think we're still working out the kinks but i like because while you're going through it i it's just so the audience knows i don't necessarily know what you're going to say and i was like well what do i do yeah so i mean i think it's good like sometimes it's advice and sometimes like here is the core message like here's the here's the thing i'm taking away from it so anyways that's a cool study man there's so much controversy around it like a lot of programmers are like well they don't know how to use the ai tools not true actually these are tools they got to choose which tools they use and these were developers that largely were already using these tools so you know there's always that argument of like well they're just using it wrong another argument it's a small study yeah it is small but it's a good signal it was a pretty clear signal it's pretty well designed another argument which i think is more important is some people will say yes that use of ai or you have it generating code from scratch is slower other uses where you're just automating like looking up information making that faster i believe that would make a programmer more productive so it does kind of uh that's not trying to get in the way of the deep work it's trying to speed up the things that's not deep work so you can spend more time doing deep work but basically i'm i'm sort of like a human a humanist intellectual chauvinist here the brain focusing hard is an incredibly powerful tool be wary of things that gets in the way of that i mean unless you really can just like outsource to work completely it might it might be some fool's gold all right we got good questions coming up got a good final segment this one will be interesting to jesse i did a little bit of original data journalism there's a claim out there that i'm looking into and the scene like is this claim true i actually looked up some data from something something it'll be kind of fun just investigate journalism but first before we get to the questions we have to do the thing that everyone is most interested in which is hearing from a sponsor jesse we have to talk about cozy earth as listeners know i'm a huge fan of their sheets they're the most comfortable sheets we own i think we have something like three different sets to rotate through we travel with them when we're away in our house rentals in the summer i will say this jesse if i was invited to stay in the lincoln bedroom at the white house i think my first question would be what type of sheets do you have second question would be why are you inviting me but first question would be what type of sheets you have but cozy earth here's what i'm excited about they've got some a new product one that i have been waiting for right so i'm a professor i'm a writer this means i'm not wearing a suit and going to an office every day and so i often wear pretty casual clothes and as the weather cools down i tend to wear sort of like athletic type pants you know i'm often on camera from the waist up it doesn't matter uh it doesn't matter what i'm wearing i mean i'm wearing bikini briefs right now because you know it's hot you got to get people to want cozy earth has a new product called bamboo joggers so that sort of like casual pant in their super comfortable style fabric that we already know and love you can wear their irresistibly soft jogger pants just about anywhere they're made from that viscose bamboo uh fabric this joggers are the softest staple wardrobe you're gonna one of the softest staples you're gonna have in your wardrobe whether you're staying in or stepping out and what about if you're going to like a slightly more formal affair so like you know me going to the lincoln bedroom to investigate their sheets well they also have you covered with their everyday pant which i have a pair of and i also love they they look snazzy but they're also like super comfortable with that fabric so cozy earth nailed comfort with their bamboo joggers and their everywhere pant that's next level everyone i know who is trying these things out says this is really comfortable so i'm a big cozy earth booster you should be too go to cozy earth.com deep and use my code deep for 40 off the best pants joggers shirts and everything and if you get a post purchase survey tell them you heard about cozy earth right here built for real life made to keep up with yours cozy earth that's cozy earth.com deep and used to code deep this show is also sponsored by better help we talk a lot about the different types of advice that might help you be more organized or make more progress on the things that might matter but you know what advice is probably the most important of all you need to cultivate a healthy relationship with your own mind everything else rests on that foundation if you want to improve this relationship therapy might be exactly what you need a professional who can help you understand your own mind and improve that relationship better help makes therapy easier with over 30 000 therapists better help is the world's largest online therapy platform they serve more than 5 million people globally it works they have an average client rating of 4.9 out of 5 stars for a live session based on over 1.7 million client reviews it's convenient too you can join a session with a therapist at the click of a button helping you fit therapy into your busy life plus you can switch therapists at any time as the largest online therapy provider in the world better help can provide access to mental health professionals with a diverse variety of expertise talk it out with better help our listeners can get 10 percent off their first month at betterhelp.com if they go to betterhelp.com deep questions that's better help h-e-l-p dot com slash deep questions all right speaking of questions let's move on to our own all right first questions from sam i found that using chat gpt questions can act as project diaries i now keep multiple chats for projects i'm working on it enables a shutdown and a place marker for where i need to restart work can chat gpt chats act as single purpose notebooks for projects i mean sure a couple questions here what is a project note like this idea of like i need notebook to keep track of the progress on on my project was that something you're doing before i guess it's useful if you don't know where you left off and you want to have external notes on it but here's my concern my concern is that you're doing something like cybernetic collaboration with chat gpt we talked about this in the deep dive when you have this running list of notes and conversations with chat gpt i'm concerned what you're doing is transforming your work into something where you go back and forth with the chat bot you hand off more you know have them do it do some stuff and you look at what they did and you ask a question you go back and forth this is more pleasant this feels nicer it makes work seem better but just as the meter study found with programmers if it's reducing your peak intensity of focus or the sustained duration of your focus and you're doing anything that's non-trivial it also could be slowing you down and hurting the the the quality of your results the key with any type of effort that requires deep work is not reducing the difficulty of the deep work it's not reducing the intensity of focus it's setting things up so that you can reach high intensity of focus the focus mind is what produces value so that's my only issue i would be wary if you're trying to make work in this cybernet collaboration that might not be making you as productive as you think even if it feels nicer last week you talked about leaving a narrative for personal projects and originally i think this question was in relation to that but now that you talk about the cybernet collaboration it kind of makes sense but if you could touch on the narrative part too yeah so like understanding so what i meant by that is like understanding how to hit the ground running so you know if i'm writing something i will into writing sessions like okay i've gotten up to here and then i have a few notes i'm like okay here's what comes next so if i could like i'm missing this example but if i could get this key example you can we could finish this section and try to get to the end or something right so it's a little note i left for myself so that when i load up that writing project again i know what i'm doing i would do this a lot with mathematics papers as well right i would put notes right into the document where i was you know typing in my results or i would email these to my collaborators and be like okay well here's where i'm stuck but i think here are like three ways i was thinking might be useful for getting unstuck it's like a little narrative a little note so that when i get back to that work i'm like okay here here's where i was or here's where i was going to think so i do think that's useful but i usually just leave these notes right where i'm doing my work it's in the document where i'm writing it's in the in the document uh where i'm putting out my math equations and i think that's useful so you know where to start again and it's easier to get into it the thing that makes me more nervous is i this idea of this long ongoing interaction i i don't actually think that type of cybernet collaboration in most cases is a useful new form of knowledge work i think it's just escaping strain it's it's a it's a fancier version of like i'm looking at my phone a lot it sort of makes the work feel less less hard but hard is sometimes what you need all right who do we got next up is mark when using active recall to study a large volume of material how soon and how often should i revisit material i've already mastered right now i find that when i return to something a few weeks after learning it i've forgotten many of the details well i mean it depends what you're trying to do here like what are you learning and why are you trying to hold on to it if you learn something that you then use it will stay sort of active in your memory and it'll eventually will really dig in if you're memorizing some sort of information that you just basically don't come back to again you're going to need to retouch on that within a two-week window like if you let about two weeks or more go your mileage will vary depending on the information in your brain brain but if you let more than two weeks go without using information at all uh it's going to start to lose its location that is just like informal based on my experience as a as a student professor so if it's like this is arbitrary information i want to remember every week or so you want to keep going back and doing that active recall eventually if you do active recall enough it will uh lock in but again be sure that you need to if you're not using this stuff be sure you need to remember it like the best way to submit something in your memory is to use it to do it i i you know you teach the concept a couple times you really remember it you explain it to someone else because you're using it for a project you know you really remember it active recall of course simulates that by recalling the information from scratch you are firing up all the circuits you would use if you were actually applying the the technique so it's using your own brain's own memory apparatus but yeah i would find if i memorize something using active recall and just never touched it again within about two weeks i might start to forget it so i don't two weeks ish but what are you trying to remember and why and why aren't you using it so maybe it doesn't have to be all right who do we got next up is beth i've seen articles ranging from ai is terrible for the environment to studies suggesting its carbon emissions can actually be lower than traditional methods can you help put this in perspective for example by comparing the emissions from one person's chat pt gpt use to something more familiar like a plane trip and by clarifying whether ai's footprint is unique to platforms like chat gpt or similar to energy use and everyday tools like google search all right i mean i do have a hot take here um if you want like the actual answers i don't know i don't have the numbers in front of me but something like a google search is sufficiently uh significantly more energy efficient than like a chat gpt query i mean a chat gpt query you have to you have to have this frontier model loaded up in memory now this thing could have hundreds of billions if not a trillion parameters defining it right so you're not going to fit into the memory of a single gpu chip you're gonna have to shard this thing over like three four five maybe six gpus that are then going to have to be running all out just to generate tokens for your answers i compare this to a google search where they use commodity intel chips most of this stuff is cached they can be dynamic of like oh there's a little downtime on this chip hey can you like go look something up in a hyper efficient sort of cached you know search index and the the amount of computation they've got it's down you know minuscule probably the answer you're looking for is in some cdn server that's like 10 miles away so no it's a lot of computational power but here's my hot take i might i might get yelled at for this one jesse you ready for it yeah i think right now the focus on the environmental impact of ai is in part a way for people who are critical of technology or big tech to get in on like i don't like ai in a territory that they're much more comfortable with than the technology so if i'm just like i don't know like a typical like left-leaning critique of the tech industry and ai i'm super comfortable for talking environmental stuff like you don't care about the environment i do i saw the al gore documentary you don't know science the environment's important like i feel like i'm smart there when it's me talking to other people like that's where i'm smart like i know more about this than you you're you don't know science global warming's a real thing so that's like a very secure place of critique a less secure place of critique is you know the type of stuff i've been doing in my writing recently like coming in and be like here i don't think that reinforcement based learning post training refinements is giving you the sort of uh lost reductions that you would have expected on a more elongated power law curve like people getting into the technical details is very like uh it's hard and you're like i don't really know this details well but i i don't like this and i don't like the people involved and sam altman's kind of creepy and mark zuckerberg still talks like a robot whose emotion circuit board is shorted out we've covered this on the show and this is a territory that i'm much more comfortable in um just like also really leaning into like the specific ai safety concerns over the particular words that a particular chatbot will or will not say like well i'm comfortable there this is inappropriate speech but i think if we're really thinking about ai these are kind of these are kind of transient concerns because i don't think a future of these massive frontier models being queried for everything is a future that makes any economic sense anyways here's the right way to think about it if querying one of these models is bad for the environment that means the amount of computation it's using can't possibly be profitable so they can't be bad for the environment too long because it correlates also with expense and people aren't willing to pay for the massive computational expense i mean stuff it takes money to create heat because that requires electricity electricity costs money so like my thought about this is yes i would be concerned if we we had chat gpt used all the time but i just don't think it's profitable i think the future of ai is going to be it has to be systems that have much smaller models machine native models if possible meaning like this thing is running on my ipad not in the cloud somewhere on a bunch of gpus small specialized language models combined with other specialized components like policy networks for evaluating options future simulators for trying to understand what actions to take next control logic that's specific to the task at hand put together into a program that you can get your arms around that can run on existing hardware and does really good at being intelligent about a very specific thing that i'm convinced is the future of ai it doesn't have an environmental footprint that's different than you know other things that we're doing right now with our computer so there probably is it does use too much electricity but i just don't think these massive frontier models are going to it just doesn't make sense i cannot be querying a trillion parameter model that requires six h100 gpus to even produce a token for me for everything i'm doing in my life that would probably be an environmental catastrophe but we're not that's not sustainable i think frontier models like f1 cards it's a way of showing off your technology and proving that your company has the best technology but it's not the car we're buying on the ford lot you know a year from now that's gonna be a much simpler car but we'll buy it from the company who had the best f1 car because that convinced us so i don't know that's right so that's my hot take is of all the issues we have with ai the focus on environment right now it's we're not in some steady state yet i just think it makes people more comfortable because that's territory where they're familiar being the sort of higher level of the hierarchy and the argument and they're uncomfortable getting the tech because like let's be honest my people are nerds and are kind of uncomfortable to be around and we don't talk very normal and we know too much about algorithmic nonsense or whatever so i don't know that's what i think is going on so i'm not as worried about the environment because i don't think this industry can survive in a mode that is super bad for the environment it just costs too much money people aren't going to pay a thousand dollars a month to pay for all their queries for things i think smaller models can work did you know for example the model uh no one brown's model pluribus which was the first ai system to beat actual tournament players in texas hold on poker the original way they built it was with like very large neural networks and they used a supercomputer system a center at new pittsburgh to train it it was like okay and then they figured out like oh we could have a couple different networks and what matters is like the logic of how we connect them together and the logic of our ai and the system they built pluribus it can beat the professional players you can run on the laptop because it didn't have to it wasn't just like we have 100 billion parameter network that we're just like learn poker like no like we're we have a future simulator that simulates the possible mind states and cards of the other players then we have a smaller neural network train just to like understand different poker positions and say what would be good or bad and then we have some logic that we symbolic logic we hand programmed in about like okay let's calculate the value the expected value if we do this and this was the case here's what would happen let's give that a number what about this it's just like straightforward mix of neural network unsupervised learning with like symbolic old-fashioned hard-coded ai and the whole thing fits on a laptop and it can beat professional poker players so hopefully the future of ai i think is most most likely the environmental concerns are not going to be um i think substantially different than sort of the environmental footprint of the computing we have right now but we'll see all right what do we got next next up is david if you left academia would you spend your days as a writer and podcaster would you enjoy this or become antsy um yeah i would continue to write and i would continue to podcast i don't think i would become antsy so i've kind of two reactions to this no i want to become antsy uh writing and podcasting actually takes up a lot of time and it's not like i'm hurting for other things to do you know how many teams i'm coaching and my kids like schools and sports leagues right now let me guess three k six it's three i'm coaching three teams right now there's a lot of time doing though i have a lot of things to do that's fine but here's the here's the here's the bigger point though is writing and podcasting is like a lot of what i do as an academic right now so i i don't know how different it would be so you know for those who know my trajectory uh i've been a writer my entire adult life right i started writing in college i trained as a theoretical computer scientist focusing on the theory of distributed systems so i studied at nancy lynch's theory of distributed systems group at mit and did my postdoc under harry balakrishnan and his network and mobile systems group so i had a computer science specialty that was on the math behind distributed systems went to georgetown this is where like my nsf funding was all about this my papers were all about this my grad students we did a bunch of i was pretty good distributed theoretician published a bunch of papers around a bunch of steering committees somewhere around the time i was you know i got tenure then i got full professor around the time i was getting full professor georgetown my where i work they began making a real move for like we want to be one of the places grappling with technology and its impacts they call the field digital ethics like we want to be at the core of this because we're in washington dc we have a big ethics background here we were the university that like figured out bioethics in the 20th century the kennedy center for ethics we want to do the same thing for tech ethics we're in the we're here in dc we have all these policy centers we have one of the biggest tech law faculties that are law school like it makes a lot of sense and i was like this is kind of what i'm doing already with my writing i write about technology and how it impacts us i want it and so like i've really been focusing on that so i was one of the founding faculty members of georgetown center for digital ethics under inaugural director of our computer science ethics and society academic program it's the first major in the country to integrate computer science and ethics in a combined major there's places where there's majors where like you're a computer science major and you throw on a little bit of ethics first integrated one uh in the country it's like this is actually what i i'm doing largely right now as an academic is technology how it impacts us what to do about it and i do it in a lot of different forms and i think i think public outreach is important i think the podcast is very pragmatic but i reach a lot of people this way my writing for the new yorker now we're talking to a little bit more of a of like a rarefied cloud crowd this gets me in front of policy makers this gets me in front of like the senate and and on npr or whatever um but it's a way again to work on tech and its impact i do some academic papers on tech and its impact which is more for like an academic crowd and then my books fall somewhere in between so i don't really know it wouldn't my life wouldn't be that much different if i left academia to write and podcast writing podcasting is what i'm doing in academia the main difference would be there'd be less time around really smart people and less time around students both of which i like also teaching but teaching increasingly i can teach things that like helps me think about these thoughts anyway so you know no i want to be antsy and also i don't know how different it would be do you still do any math i haven't very recently i haven't yeah been a year or so so do you still teach undergrads like traditional computer science classes i do a mix of traditional computer science and stuff that's relevant for uh the computer science and ethics program yeah and i teach less than i did before too because i'm doing running these things or whatever so that's a good question though um i think we have a case study yeah before we do the case study if folks have calls then just go to the deep life dot com slash listen and submit some updated calls is that deep life or the deep life the deep life the deep life dot com slash listen there's a link yep and you can record the call right there from the browser yeah do that we have a lot of calls but they're getting kind of old right yeah you got a good shot a good pithy call on a topic i've been talking a lot recently you got a good shot now so i would go i would go record the calls uh we do have a case study however i love case studies this is kind of a long one but it's great case study on my theories of lifestyle centric planning i'm thinking so much about this because of my new books i'm happy to have this we have what theme music for this right yep do we i don't remember is it just to introduce the segment or do we play it the whole time i don't know the two episodes we did one but then last week you wanted to hear both all right play it now and see your mood at the end all right let's see what we let's hear it i just want to do a deep thoughts with jack handy all right so today's case study comes from vin i hold a bachelor's degree in interface design and in my 20s first worked as a web developer and later as a ux designer for several digital agencies i enjoyed my work but i felt stuck in big cities and kept dreaming about a life closer to nature so in my mid-20s i did something that you would surely advise against oh i threw my career into the trash and moved to norway to live and work on an organic farm i had a great time there and enjoyed every aspect of it however i didn't have a sustainable long-term plan i realized i needed and wanted to move back to my home country where my girlfriend now wife lived back home i decided that my next big move would be to find a job that allowed me to work in nature so i signed up for a nature guide program and actually managed to land a job at a as a guide at the very place where i had completed that course for the last six years i worked there running workshops and similar programs sounds like a happy ending right not quite life changed i became a father of two children and we moved closer to my parents so we could get support from the grandparents with family life this meant i was now living almost two hours away from my workplace constantly torn between family and professional life on top of that since most workshops take place on weekends i was away from my wife and our two little ones at the very time they needed me most it turned out that the dream of working in nature definitely had its flaws i'm going to stop there for a second let's take stock of where we are in the story so far we're seeing a common issue when people think about the deep life so a common issue that people have is that you fixate on a single change that you begin to believe will make everything better it's simpler to think about a single change we can sort of inhabit that change the idea of doing something radical itself makes us feel excited in anticipation and we we can sort of take the imagined feeling of like the best parts of that change and sort of expand that in our mind like my life is going to be better so savind did this with nature he's like i like nature so i'm just going to focus like a laser on if i could just be in nature all the time i would be really happy the problem as we often say when we talk about lifestyle centric planning is that your daily subjective mood is not the result of a single decision or change but on all of the relevant aspects of your life call this your lifestyle all the different things that are relevant during the day add up to give you your subjective experience of the day so you might like the part where i'm giving a workshop and it's in nature and that's nice but you have other parts of your life too like the fact that you had to drive two hours to get there and that you're kind of stressed out about what's going on with your your in your grand your parents and like the kids are you're not around enough to help and it's creating tension with your wife all of your houses maybe not it's nowhere near nature and you're actually spending most of your time in a car all of these other aspects of your lifestyle matter too and this is how you can make one radical change that you're excited about and yet end up less happy than you were before because by making this thing better you accidentally maybe made other things worse so lifestyle centric planning says you got to construct your ideal vision for an entire lifestyle all the parts of your life how would i reorient my life so that all of the parts were something that resonated and then how do i make progress towards that you're going to have you're going to enjoy your life more if you explicitly construct it so that all of the things that contribute to your enjoyment are are being pushed in ways that are better let's return this finn because he recognized that and let's see how he made some changes around this time i came across your books deep work so good they can't ignore you and slow productivity they gave me a framework to better understand my professional situation i had to take a hard look at myself and realize that for the last five to six years i hadn't built transferable career capital that could help me in a different role after some reflection i decided to revive my previously built career capital as a web developer however i hadn't been coding seriously for years and looking at job descriptions i realized how much had changed in the web development space and i had a lot of catching up to do all right so he's doing something here called evidence-based planning like so he makes this decision he's looking at his life and it's like actually i need a job that um it's going to whatever the criteria here but probably like more money more autonomy i don't have to commute so much um whatever web development is like i have career capital there let me start there but he does evidence-based planning he actually looked at real job listings like oh skill a b and c i don't have so i'm gonna i'm gonna summarize a little bit here but basically he uses like techniques from deep work and slow productivity as well as from scott young's excellent book ultra learning to begin a education process of learning the specific skills that his evidence-based planning said were important not what he wants to be true but like i need to know a b and c these employers want exactly those skills and how do i get there as effectively as possible i'll jump back to regular speed here thanks to your techniques and tools it took me about a year to get out of my difficult job situation and back into my old career the side effect i now earn more than twice as much as before my next move won't be as bold as the ones i made in the past i want to become a valuable asset for my company to get so good they can't ignore me and then use my career capital to reduce my working hours so i can have fridays off to spend in nature without having to work there so see what happened there is when he did lifestyle-centric planning he's like i like nature but by reorienting my life around my career is takes place in nature a lot of other things got worse and actually being in nature is not as great as you think when you're also like working and you realize like oh if i could take my programming expertise make it up to date make myself really valuable i can go back to programming get good enough that i can cut my hours still make more than my old salary only work three or four days a week and then spend time in nature very intentionally but now without the commute now being home with my family now things being more stable um man that's a better life so lifestyle-centric planning worked out better than just putting laying back like what's a radical thing i can do i'll move to norway i'll get a job as like a nature guide lifestyle-centric planning was not as exciting but it led them to a better place all right so we got coming up my investigative journalism beware i do a little bit of data reporting but first before we get there what you've really been waiting for to hear about another one of our sponsors so here's the thing if you run a small business you know there's nothing small about it every day there is a new decision to make and even the smallest decisions can feel massive jesse knows this i get upset whenever anyone refers to our business here as small i know how hard it is to run this just the other day for example the nice woman who runs the gift shop down the on the next block here in tacoma park asked me how life as a small business owner was going so naturally jesse and i rolled her car into a swamp that's where i can put up with that nonsense so here's what i'm trying to say it's hard to run a small business so when you can find decisions that are no-brainers you should take them and when it comes to selling thing using shopify is exactly one of those no brainers shopify's point of sale system is a unified command center for your retail business it brings together in-store and online operations across up to a thousand locations it has very impressive features like endless aisle ship to customer and buy online pickup buy online pickup in store and with shopify pos you can get personalized experiences to help shoppers keep coming back and they will come back based on report from ey businesses on shopify pos see real results like 22 better total cost of ownership and benefits equivalent to an 8.9 uplift in sales on average relevant to the market set survey so get all the big stuff for your small business right with shopify sign up for your one dollar per month trial and start selling today at shopify.com slash deep go to shopify.com slash deep shopify.com slash deep i also want to talk about our friends at my body tutor i've known adam gilbert my body tutors founder for many years and he is my go-to guy for fitness advice his company my body tutor is 100 online coaching program that solves the biggest problem in health and fitness which is lack of consistency they do this by simplifying the process into practical sustainable behaviors and then giving you daily accountability and support what you need to actually keep taking action on your plan you have an app you check in every day like hey here's what i ate here's how i did my exercise and your coach checks it hey how'd it go if you have a question like you know i'm really struggling with x like great let's fix this right away knowing there's someone checking on what you do every day and is there to help you if anything's not working that's how you get consistency and getting healthier it's not the information it's the consistency and having this online coach it is a really really good way to get healthy all right so if you're like i want to get in better shape this is the way to do it go to my body tutor.com mention you heard about it on the deep questions podcast when you sign up and they will give you 50 off your first month that's my body tutor t-u-t-o-r.com and mention deep questions all right jesse let's move on to our final segment so i want to talk about an interesting article about technology and students and i want to look a little bit deeper at it and have a bigger point to make this article actually appeared recently in the washington post it came to my attention however when tyler cowen wrote about it on his excellent blog the marginal revolution i'll pull it up here on the screen the blog post because the post the washington post is behind a paywall but marginal revolution is not all right so here is tyler's setup just summarizing this article he says until recently a nearby radio telescope meant the local school could not use wi-fi and how did that go all right so this article that he's citing here is about green green bank west virginia have you heard about this place jesse no it comes up a lot in technology circles because it's rural west virginia and the world's largest steerable radio telescope is there oh really yeah and there's a town next to it and radio interference could mess with the telescope so there's no cell phones and there's no wi-fi so it's like a town that's kind of like off the grid because of this uh this telescope that's in the middle of the town so what this washington post article that tyler cowen is citing what it was about it was actually an op-ed it was written by someone who has a book he's working on about green bank what it was arguing is the fact that there is no wi-fi in the town has been hurting the students in the school there green bank has an elementary middle school combined school about 200 something students he's arguing without wi-fi they can't use like the modern chromebook online curriculums do online testing and this is hurting the students so let me read here this is a tyler cowen citing the washington post so what i'm about to read here is from the washington post article while the rest of the country rushed to bring tech in the classrooms green bank remained stuck in 1999 without wi-fi the school's 200 students couldn't use chromebooks or digital textbooks or do research online teachers couldn't access individualized education programs online or use google docs for staff meetings even routine tasks such as state mandated standardized testing became challenging with students rotating through a small hardware computer lab where they took the exams some of the teachers say this has been a problem here's a quote here the ability to individualize learning with an ipad or a laptop that's basically impossible that was teacher darla huddle here's another teacher being quoted in the op-ed without the online component of our curriculum fully working it's really detrimental to our instruction that was sarah brown so this is the issue here they can't do this sort of modern ed tech stuff and the argument of this article is like that's held this school back and how do we know this well there isn't data on this in the article itself like specific data but um here's what the author says while these discussions dragged on so discussions about like is there some way we can get like low powered wi-fi or some other ways to get wi-fi in the school while these discussions dragged on students fell further behind in math and reading with greenbrink consistently posting the lowest test scores in the county all right this is an interesting argument here that without modern ed tech students are falling behind and that's it's like a natural experiment and look this school has the lowest scores in the county now this is like a relevant time to be arguing this because there's so much discussion about phones in schools and technology in schools and it's mainly negative like technology in schools is often negative but here's this this op-ed in the waapo that's being contrarian and tyler who also is has a real affinity for technology is like yeah come on let's be careful about it maybe some of this technology is really helping so i want to look into this uh you know i'm not an expert data journalist but like this is interesting and i want to look a little bit closer so what did i find well first is it true that green bank consistently post the lowest test scores in the county this is turns out to be called pocahontas county it's kind of interesting we're like in west virginia okay so it's pocahontas county west virginia yes the green bank elementary middle school has the lowest scores in the county but there's a problem here it's a very small county i mean it's a lot of geographical area but you know this is not montgomery county here in like washington dc the the county you know there's the green bank middle school and elementary school the county has one other middle school and a shared high school and then there's two smaller elementary schools that are in like another town over there much smaller one of which which has the county's like gifted and talented program there's only 70 kids there half of them are gifted and talented kids like we're we're not talking to big county it's a handful of schools of which like two of them of the six are uh in green bank one is shared and then there's like one other middle school and two other elementary schools they do so yes like the the elementary school in green bank is has lower scores than those two other elementary school the middle school has lower scores um than the other middle school all right but so it's it's a small county but that doesn't necessarily tell us much like maybe green bank you know for whatever reasons that part of this the county just is worse off and they just get worse scores i don't know what's going on there like do we know this is from not having wi-fi or not well what we really would need to test this is we need time series data so we need to see test performance over time right because until somewhat recently it wouldn't matter if you had wi-fi or not right so ipads were introduced in 2010 we see the rise of chromebooks and classrooms picks up in the 2010s that's really where this thing this happens so before the 2010s there would be a lot more technological parity between this particular school and the other school in the county right because no one was using technology like that in school so what we really need is like a time series that shows somewhere in the 2010s the wi-fi free school begins to uh separate that they the impact on not having wi-fi makes them worse because they might have always been worse we want to see them get worse all right so i went looking for this data it's hard to find time series data for the particular schools in pocahontas county but because pocahontas county is so small there's two middle schools one of them is with wi-fi one of them's not there's three elementary schools one with one without and the other two add up basically to the size of the one that didn't have wi-fi it's such a small county what we can do is compare pocahontas county to other counties in west virginia because presumably if like half the students in your county didn't have wi-fi and that affected them that should bring down the whole county around the time that why chromebooks etc became big in school so there is data i can get data on uh any any ap you know these are state mandated test scores i can get data on this county by county so let's see what's going on here so let me load up some charts this is from education education recovery scorecard which aggregates a lot of this data it's really interested in like what happened after covet but they have data that goes back to as far as 2009 so it's perfect for us what i'm loading on the screen here for people who are watching instead of just listening is math performance grades three to eight because again going past middle school it's a shared high school so like we don't learn much from that from 2009 to 2024.

all right so as we see here going from 2009 we see an increase there's a bit of a gap in the data but we see roughly actually math scores were getting better go pocahontas county math scores were getting better from 2009 until about 2017. in 2017 oh this is interesting the score started going down they went down went down then we get the pandemic where we don't have data but they were falling multiple years before the pandemic and then we see after the pandemic we begin to get a recovery which we've seen nationwide as things fell so low during the pandemic that there's a recovery right after so that's the chart we're seeing if we look at reading scores it's something kind of similar here the peak is around 2015 and then it begins to go down after that as well the timing here is compatible with the wi-fi hypothesis right that kind of makes sense chromebooks and all these other things that require internet begin spreading in the 2010s it might make sense that where would you first start seeing the impact of not having this technology like midway through the 2010s as these things were gaining traction that might be where you start seeing performance go down so this data from within pocahontas county itself is compatible roughly speaking is compatible with the washington post hypothesis of without wi-fi things started to get worse but if we want to do a controlled experiment we have to compare this the similar counties so what we need to do because pocahontas is half the non-wi-fi and we can't break that out so let's find other counties in west virginia that are similar in terms of size and demographics and socioeconomic status like very similar other counties nearby in west virginia where there were no radio telescopes and everyone is allowed to use wi-fi what we would expect to find if the wi-fi hypothesis was correct is that pocahontas county should have a much more notable drop in performance starting in the mid-2010s than these other counties that didn't have any wi-fi restrictions we have that data that's what's interesting so let me scroll here a little bit all right so this is we're going back to math performance here we have three stack charts so if you're if you're listening i'll explain to you what we see each of these charts has a downwards arrow and an upwards arrow the downwards arrow which is purple i'll put this up here on the screen for poca pocahontas county schools first that is the decline in math scores during that period up to 2019 where um let me get the exact dates here uh 2019 to 2022 where we had the steepest sort of losses we saw in those curves before and then the green arrow is the improvements that they've seen since the pandemic so this chart is showing here for pocahontas county schools it's just quantifying what we saw on that chart in that downhill period from 2019 through 2022 there was a uh 0.6 negative 0.6 drop in their performance compared to i think this is like state average so yeah they fell and then we see the green there's like this 0.36 uh increase so don't worry about the numbers but there was an increase uh this is the recovery after the pandemic all right so i'm just quantifying what we saw on that chart here's what's interesting though they did the same thing for all of west virginia counties that's the next one and for counties that had similar uh socioeconomic demographic and size so similar population so if you're wondering this is nicholas county hampshire county barber county tucker county and pendleton county here's what's interesting the similar west virginia districts that were the same in terms of demographics but differed mainly and they didn't have the wi-fi restrictions they saw a bigger drop from 2019 to 2022 and they saw a smaller recovery after the pandemic so the similar counties that had wi-fi never had that taken away did worse than the county in which like half the students are in this wi-fi free zone and in fact if you look at the west virginia as a whole that also was worse than what pocahontas county did so look i don't have school by school data i hear this a lot i found a lot of informal reporting online that did blame poor performance in green bank by them not being able to use internet in the school so that might be true and these teachers seem to think it's a problem but if we look at the data without having time series data from specific the the handful of schools within pocahontas county we do not see here a data story that looks at that looks like the lack of internet once chromebooks became a thing and these online curriculums became a thing began to make green bank much worse it seems like that's just a bad school those schools are bad and they've been bad and i don't know why you'd have to know about like you know it's just like what this town is i don't know what particularly is bad about that town it could be little things by the way because these numbers are so small not to get too much into the data but i was looking at proficiency on math and reading test scores broken down by the individual school so i can get those numbers for the most recent times they've measured them and they're like you know 50 better for example in hillsborough elementary versus the green bank elementary but hillsborough elementary in pocahontas county is the gifted and talented program there's only 70 kids in that school 70 kids total so all it takes is like oh we have 30 of our gifted and talented kids are there you're going to get 50 better you know number of people who are math proficient right so these are small numbers so we have to be careful um so yeah there's not a lot of schools there and the green bank one doesn't do well but we don't have evidence because that it is because they couldn't use internet connected chromebooks again the only way for this to be possible would be somehow the the lack of chromebooks and internet connected tech really caused the green bank schools to fall really hard but for unrelated reasons coincidentally the other two or three schools in pocahontas county did unusually well during this period and they sort of offset the fall that the green bank was having and that's why that county even though other counties with the same demographics as pocahontas county fell farther that there was something special happening with the non-green bank schools of pocahontas county where they offset the green bank losses maybe that's true but we would have to hear a plausible reason why that's true and actually see school-by-school data so i don't know the author of that op-ed is writing a book on green bank and he might have really good data but knowing what we know now i think this is a nice cautionary tale and it's a good reminder for myself or anyone else who talks about technology trends when there's an answer that we like it's often easy to jump to it find any point of data that seems to imply that and expand it beyond what the data says i think that seems to be what was happening here there's a nice subtle leap from the schools without wi-fi are worse to the schools without wi-fi are worse because they don't have wi-fi it's an easy leap to make but the picture gets much more murkier once you pull even a little bit on the data story so anyways i guess this is just a back to school note of we all have to be careful and take with grains of salt claims that are made that sound intuitive and there's a little bit of data to support doesn't necessarily mean it's true like it's like when it comes to cell phones in schools i have read exhaustively not just the research but the debates that researchers are having about the research and the complaints and then how the complaints are answered i i you know there's an hour-long talk i give about the evolution of the research literature on harms from phones for kids i feel like i know that data very well and it makes me confident to say these are um often a problem the benefits aren't worth it you shouldn't have phones before high school if you're a kid right that's really different than like hey that school struggles and they don't have wi-fi internet's good so anyways there we go i'm not investigative journalist data and who knows maybe this this professor has the school by school data but i think the picture in west virginia pocahontas county is more complicated than if only i could synchronize my eureka math curriculum with like online resources our kids would suddenly be much better harder reality all right speaking of hard realities that's all the time we have for today so thank you for listening we'll be back next week with another episode and until then as always stay deep hey so if you like today's discussion about ai and productivity you should see uh check out my recent episode episode 367 where i get into the whole past and future trajectory of ai it's called what if ai doesn't get much better than this that is a sort of must listen to understand where i'm coming from on ai so if you haven't heard that episode yet go check it out i'm cal newport and this is deep questions today's episode what if ai doesn't get much better than this