back to index

Knowledge Graphs in Litigation Agents — Tom Smoker, WhyHow


Whisper Transcript | Transcript Only Page

00:00:00.000 | Hello everyone. I am here to talk about GraphRAG as we're here for the track, but I'm talking about
00:00:19.760 | what to do in the legal industry and what we do in the legal industry and what does it look like
00:00:24.080 | to turn documents into graphs and use those graphs in the age of AI. I tend to have to
00:00:29.120 | qualify why I'm at places. There are various reasons why I could be talking today. You choose the one you want to,
00:00:34.480 | but generally I've been working in graphs for about a decade. I have a good relationship with the Neo4j team
00:00:39.920 | and I've been doing graphs for a long time, but primarily I am the technical founder of a company
00:00:44.000 | called whyhow.ai, and we find cases first before lawyers do and then give them to lawyers. Now,
00:00:52.560 | how we find these cases is a process that will go through, but we use a variation of graphs,
00:00:58.240 | multi-agent systems, signals, et cetera. And I'll detail through today how we do that at a high
00:01:03.600 | level and a low level, and I'm happy to answer questions at any point. This is broadly what we do.
00:01:09.600 | We work in law. This is an example. We find class action mass tort cases before other people do.
00:01:15.040 | We have agents. We have graphs. We store that information. We scrape the web. We qualify that
00:01:20.080 | with a proprietary process, and we deal with lawyers every day and understand exactly how they think and
00:01:26.400 | build these cases. And the cases I'm referring to would be like many people used a pharmaceutical
00:01:31.040 | product. That product has caused them harm. Science has proved that harm, and we can collect those
00:01:35.520 | people and collectively sue the pharmaceutical company. So we support the law firms that do that.
00:01:40.000 | And as I'm talking, and everyone here for a graph rad track can start to imagine that I'm starting to
00:01:43.600 | develop a bit of a schema there. I'm describing individuals. I'm describing products. Those products
00:01:47.520 | have ingredients. Those ingredients have concentrations. Those concentrations may have an ID number, and all of a
00:01:52.000 | sudden, you can start to imagine there is this large networked, schematized bit of data that has
00:01:58.240 | particular points in it that are very valuable and very visual and very useful to domain experts.
00:02:02.960 | So I'm going to start to use some definitions, because knowledge graphs have been around for a long
00:02:08.480 | time, and ABK would know that more than I would. But I started my PhD and wore my master's in graphs in
00:02:13.840 | 2016, and it was not nearly as popular as it is now, and it's fascinating to see how far it's come.
00:02:17.760 | But I do think it's important for me to define how we use them and how we think about them.
00:02:21.200 | Broadly, to me, graphs are relations. That's part of the visual element, and there's a back-end
00:02:27.280 | element as well. But it's the benefit of using graphs is that I can see what is connected to
00:02:32.320 | something else. I can be explicit about what is connected to something else, and I can do mass
00:02:36.480 | analytics on what is connected to something else. All the way from I can see it down to I can do large-scale
00:02:41.760 | analytics on it is the value of the relations. And when I use relations, it's not necessarily node to
00:02:47.040 | node. It can be node to node to node. It can be multi-hop. It can be as varied and as forked and as
00:02:51.920 | distributed as you want. This is why we use graphs in our process. Broadly, throughout the process of
00:02:58.880 | running this company and previously as an academic, this is what I think is easy about graphs. People look at
00:03:03.120 | them and go, well, that's fantastic. I have a great understanding of what this is. And someone else says,
00:03:07.120 | me too. And there isn't necessarily a consistency in what those two people just said. They may have a
00:03:11.760 | different understanding of what is represented. Broadly, throughout my career, these are the things
00:03:17.200 | that are difficult about graphs, right? And you can say that they're nodes connected to edges. You can say
00:03:22.640 | they're distributed. You can say they're backed up. There's a variety of ways in which people use the data
00:03:27.760 | that they have, the way they store it and the way they talk about it. And now, as graphs have become very
00:03:32.720 | necessary and consistent for things like graph rag, for things like structured data, etc., more and more
00:03:37.680 | people are coming to this relatively niche area previously that even at the time wasn't necessarily
00:03:42.720 | agreed upon what it was. So, I do like to define what it is we're using. So, graphs and multi-agent
00:03:48.800 | systems. These are the two things that I want to define as there's a variety of ways that people use them.
00:03:55.440 | So, this is how we use multi-agent systems, right? So, now, multi-agent systems are all the way from very
00:04:03.360 | specifically define what you're dealing with and chain those together and use an LLM to glue it all
00:04:08.320 | together. Or it is, in our case, break down a complicated white-collar workflow down into a specific set of
00:04:15.440 | steps that I can I/O test, right? And each of those steps have different requirements, different frequencies,
00:04:20.720 | different state, and that state can be controlled often, in our case, by a graph.
00:04:25.440 | This is why we like to use them. When we're building an application for the legal industry,
00:04:32.720 | I'm not sure if you guys know this, but lawyers don't really like when things are incorrect, right?
00:04:36.400 | It is basically the whole industry is make this very specifically correct and proper and definitely in
00:04:41.200 | the right language. So, when it comes to building applications, probabilistic large language models
00:04:46.320 | don't necessarily work for that just in isolation. I need to have a very specific control and structure
00:04:52.000 | and schema for the way that we build these systems, and I need to be able to test and be able to
00:04:56.640 | pinpoint exactly what is going right and wrong at any point in time.
00:04:59.520 | Here are some of the issues with that, right? And we've heard about multi -- well, at least I've heard
00:05:06.160 | about multi-agent systems a lot. I'm sure other people have as well. Sometimes the part in the workflow is
00:05:12.080 | much more important than the other part. Sometimes there's parts in the workflow I don't particularly
00:05:15.920 | care about. There are also agents in the world. Agents imply that these things are very capable,
00:05:20.960 | but I can write a bad prompt very easily, and all of a sudden I have a bad agent. So,
00:05:24.800 | when it comes to what is the agent that I trust, very few. We spend a lot of time guard railing as
00:05:30.880 | much as we possibly can. We spend time making so that the memory is not just immediate, but it's episodic.
00:05:36.000 | We spend time capturing the information state over time and then pruning that state. And again,
00:05:40.880 | to bring it back, capturing, expanding, pruning, structuring, and then querying state for us
00:05:48.080 | happens in a graphical format. Because the necessity of having the structure, having the extendability,
00:05:53.440 | and then having the ability to remove that extension is really important for us. And then finally,
00:05:58.480 | I'm trying not to make this too deep depth and too many numbers, but 95% accuracy for a single agent,
00:06:05.520 | I think, is a tall order at this point. Maybe people have entirely accurate agents. I'm very happy
00:06:12.080 | for you. I don't have that exactly right now. I have systems that I can put in place, like guardrails and
00:06:16.800 | humans in the loop, that can bring these agents to a point that it is accurate enough that people are
00:06:20.720 | willing to use them. However, five 95% accurate agents chained together sequentially, that's 77% expected
00:06:28.320 | accurate accuracy. That's not that many agents in a row. If you think about a workflow, that's five
00:06:33.280 | steps. And if I'm basically saying that if each of those five steps are 95% accurate, already quite a
00:06:38.560 | hard thing to ask, especially if there's an LLM involved. Now we're at 77% of the time, it gets to
00:06:43.360 | the end of that workflow in the way that I want. That is part of, probably, if I was to summarize my main
00:06:48.320 | problem with that. It would be decision making under uncertainty throughout the process of building
00:06:52.800 | these systems. That's the background. That's how we understand these systems. We use multi-agent
00:07:00.000 | systems and we're naturally skeptical. We use graphs every day and we have a natural skepticism of exactly
00:07:05.360 | how these things are stored and structured, but we use them specifically and consistently in the way
00:07:09.040 | that we like. So I am using the term agent because everyone's using the term agent. We build litigation
00:07:14.720 | agents. Litigation is the process of, well, I'm going to summarize, but we work with class action
00:07:19.680 | slash mass tort law. As I said before, get everyone together. They were harmed. Put that harm all in
00:07:25.280 | place and then sue a pharmaceutical company. Now, we don't do any of the litigating as a company or the
00:07:30.640 | suing, but we do support the lawyers who do that. We do that in a few different ways.
00:07:34.800 | Here is one of the ways that we look at the legal industry, right? Without exception, everything needs
00:07:43.360 | to be perfect, needs to be accurate, needs to be written in the correct way, right? There's also,
00:07:48.960 | once you have that correct format, creative arguments. The best lawyers are very, very, very
00:07:55.600 | detail-oriented and then very, very creative in the way that they can apply those details to a case.
00:08:01.040 | For example, there was an issue with Netflix and they were capturing data from their users, as they do
00:08:07.600 | and they should, and I'm a Netflix user and they capture my data and I appreciate it because they give me the
00:08:11.120 | better shows that I'd like to watch. However, there is a legal limit as to how much information they
00:08:15.120 | can capture from me, right? And you cannot surpass that legal limit, or you can, but then you can go
00:08:19.120 | into the process of litigation. Now, if you surpass that, there needs to be a precedent as to why someone
00:08:26.720 | could say, "You cannot capture this much information." And the particular precedent I'm referring to
00:08:31.440 | is many years ago, Blockbuster was sued by keeping too many details about the literal physical DVDs that
00:08:38.240 | people rented. That is a reasonably creative way to say, "Look, I remember that Blockbuster happened,
00:08:44.320 | and what Netflix is doing isn't that different. It may be in a digital format, it may be at a larger scale,
00:08:48.160 | it may be into an algorithm instead of someone who's recommending it. However,
00:08:51.840 | that is an interesting application of what I'm doing." So, these problems then, which is necessary
00:08:59.760 | accuracy and then creativity on top of that accuracy, and then all of that information is kept in separate
00:09:05.680 | places, and a lot of that creativity comes from the latent knowledge in the expert's head, starts to come
00:09:10.400 | to a bit of a fall when you say, "Well, I have these probabilistic agents that you could argue aren't
00:09:13.840 | that creative." Right? I have these agents that most of the time do a pretty good job and can be creative
00:09:20.720 | in a way that, frankly, can be quite frustrating, especially to a lawyer. So, this butts heads in
00:09:26.960 | terms of exactly how lawyers want to deal with this information. And again, I'm painting a very broad brush.
00:09:30.640 | I'm not a lawyer. My co-founder is. If anyone is a lawyer in the audience who's offended, I do apologize.
00:09:34.960 | But this is broadly what I've seen to be accurate. We help with legal discovery as well, right? Like I
00:09:42.240 | described before, there could be an unnamed pharmaceutical company. A pharmaceutical company's
00:09:45.600 | great, but they happen to have done some harm, right? And it is in their best interest to give
00:09:50.880 | all of the information to the law firm and describe exactly -- well, not exactly -- describe in as many
00:09:56.400 | ways as possible, "Here is 500 gigabytes of emails that don't matter. Go nuts." Right? Figure out exactly
00:10:03.840 | what happened at what point and bring up the information. Now, that is a challenge at the moment. A lot of the
00:10:08.880 | time it's manually reviewed. There are shortcuts and processes by necessity because a lot of these
00:10:13.120 | lawsuits are on a particular timeline. It is physically impossible to read all of the information
00:10:18.480 | that is given in the discovery of the processing of a lawsuit. However -- and this is just a generic
00:10:23.600 | graph I used because I'm not allowed to use the ones that I'm currently working on -- however, if you can
00:10:29.040 | take all of that information, you can extract the information and structure it in such a way that it is
00:10:33.200 | consistent, all of a sudden that mountain of emails becomes a lot of information I can immediately
00:10:38.560 | dismiss and a bunch of generally, genuinely useful information that I can look at. And not just that,
00:10:44.160 | when it comes to a graph, I can actually augment the information from discovery and then I can give
00:10:49.200 | that visual to the expert who can make an immediate decision. I'm going to loop back to the example I was
00:10:54.480 | working with, what I was describing before, which is the pharmaceutical example. So again, if ingredients are a
00:10:58.800 | certain concentration, that concentration is at a problem, that problem happened at a certain time,
00:11:02.720 | there is only going to be a few people in that graph of potentially millions of nodes that are
00:11:07.760 | a problem, right? In the same way that there are only a few people in that mountain of documents that
00:11:12.160 | were a problem. However, now I've changed the form factor such that I can specifically hone in on what
00:11:16.800 | matters, and not just hone in in a data-driven way, I can hone in in a visual way and natural language
00:11:22.080 | such that the lawyer who knows exactly what that natural language means, or the expert who knows exactly what that
00:11:27.600 | natural language means, can make a decision that's data-driven.
00:11:30.960 | There's also a process if we can build this information exactly, and I'm giving the fundamentals.
00:11:37.200 | This is a graph rag talk. We want to bring this graph in. The graph I just described is not that
00:11:41.440 | large. The graph I just described has a consistent schema, and the graph I just described can be
00:11:45.120 | relatively easily retrieved. I'm not going to say that retrieval is completely solved. I am going to say
00:11:50.480 | we have agents in production right now that lawyers can in natural language query and further understand the
00:11:56.400 | lawsuit and the individuals that they are representing.
00:11:58.400 | Now we get to case research. That was more discovery, right? A mountain of documents.
00:12:05.120 | Case research would be a lot of people used said product, and they are complaining about it online.
00:12:11.280 | And this is a lot of the value of our company and what we do. People can complain all the time. They can
00:12:16.640 | shout into the void of a niche subreddit, or they can go on Twitter, or they can be on a forum that they're used to.
00:12:21.280 | They can be in IRC. They can be wherever they want, right? But they're using similar language about a
00:12:25.440 | specific thing. And so when it comes to traditional case research, that information isn't really
00:12:30.400 | discovered. A lot of the time it happens through talking to another individual, subscribing to a
00:12:34.640 | newsletter, et cetera. How do people find the information?
00:12:37.840 | So, and this is a graphic I've taken from our website, which I promise looks significantly better than the
00:12:43.360 | slides that I make, but I tend to try and talk to them. Here is how case research, in our case, for our
00:12:50.480 | business, works. And that is we start and scrape the entire web. Now anyone can scrape the entire web.
00:12:56.240 | It's doable. It's a technical challenge, but it's doable. And you can scrape it at a frequency in the
00:12:59.600 | services, et cetera. What we do is scrape the web and then qualify the leads of that scraping. We filter
00:13:05.360 | down all of the information down to specifically what the individuals want. We have schemas that we work with
00:13:09.600 | particular law firms and lawyers. And those schemas get us down to just the information that they care
00:13:15.120 | about. And look, maybe there is, but right now, at least for me, there's no such thing as a perfect
00:13:20.560 | case. There's no such thing as a perfect lawsuit. It depends on the lawyer or the partner or the firm
00:13:25.040 | who's willing to take that on. So it is not a problem of best. It's a problem of specific and
00:13:29.760 | personalized. And that is where things like LLMs are particularly useful at the moment. That's where
00:13:34.480 | things like multi-agent systems are fantastic. That's where things like structured information and graphs
00:13:38.480 | all of a sudden, a different lawyer can have a different multi-agent system and a different graph
00:13:42.000 | that backs up their specific way that they like to work, as opposed to having a compromise
00:13:45.920 | previously on the way that everyone else liked to work to maybe hear something if they can. And from there,
00:13:50.800 | once we've honed down just to the signals that they care about, the qualified signals that are specific
00:13:54.960 | to them, that signal can then further generate a report, and that report can be entirely specific to the lawyer
00:14:00.800 | as well. So when it comes to report generation, again, a multi-agent system that's backed up by a schema, and that
00:14:05.520 | schema is consistent and pruned, and that schema looks like a controlled state with a graph that can
00:14:10.080 | build the report that the lawyer wants. And every report is going to be different, but the structure
00:14:14.000 | is going to be the same for each lawyer, and each lawyer has a different process. What I'm broadly
00:14:18.480 | describing is mass scraping the web down to a specific signal generated just for the lawyer. It's entirely
00:14:25.200 | personalized service that's been automated. And that is the process of what we do, and this is part of
00:14:30.880 | how we are able to manage and use state and graphs and multi-agent systems to bring the information
00:14:36.160 | together. Cool. I'm going to go through -- I think I have one case study that I want to describe, just
00:14:42.320 | conscious of time. This happens. It's not great. No one really wants it to. There may be situations in
00:14:50.320 | which there's a bunch of people who bought a car who really wanted it to catch fire. We don't necessarily
00:14:53.280 | deal with them. What we do find is that there are people who are driving their car, and it starts to smoke,
00:14:57.120 | and then it catches fire. That's not the behavior that they intended to happen. It was not on the
00:15:01.280 | brochure when they bought it. It's not what they want. Those people immediately go and complain, as
00:15:05.680 | they should. They go to government website. They go to carcomplaints.com. They're on a specific subreddit or
00:15:10.720 | forum. And once we can start to track that, which we can, and once we can start to scrape and then
00:15:16.080 | structure and then schematize and then analyze, we can start to basically build a density of complaints for a
00:15:20.960 | specific vehicle, for a specific year, for a specific problem. And that density is a combination of how many
00:15:26.160 | complaints multiplied by the velocity of complaints, so a certain amount per month over a number of
00:15:31.600 | months. All of a sudden, we get to the point where we're finding these leads particularly early. And
00:15:36.240 | now, as we're building models, we're starting to find these leads early and earlier, and that we don't
00:15:39.840 | necessarily need the velocity straight away. We can start to figure out what are the previous lawsuits,
00:15:44.400 | which were all public and very well documented, and exactly what happened in that process. And so,
00:15:50.080 | for a large law firm, maybe eight or nine months post people starting to complain, they can take that
00:15:55.600 | lawsuit on if they want to. For us, we can find it within about 15 minutes. And then generally,
00:16:00.560 | it takes probably a month for you to be confident that this is the signal that you want. And so we can
00:16:04.480 | find things significantly earlier. That process, again, scraping the web, filtering down, producing the
00:16:08.640 | specific report. This is an example that we did. And again, we deal with what the lawyers want. So this
00:16:14.080 | lawyer, again, he made the case that people's cars are catching flyers. They don't really want them
00:16:18.640 | to. Those are the cases that he would like to take on. It's of a certain amount of money. It's of a
00:16:22.480 | certain make and model. It's in a certain jurisdiction, et cetera. Those specific filters,
00:16:26.320 | that schema can be applied throughout the entire process. That's basically the graph. Each of these
00:16:30.560 | lawyers have a specific graph that they want. And not just that, they can filter and feedback that
00:16:35.920 | information. So it's not just a static graph. I mean, the benefit of a graph structure, at least one of the
00:16:41.520 | benefits of a graph structure, I should say, is that it's an extensible schema. And that I can update
00:16:46.320 | and I can query across and I can understand that information. So while we are dealing with RAG, I
00:16:51.440 | would say we have less of a chat RAG interface. Well, the lawyers definitely do appreciate that. A lot of
00:16:57.520 | what we have when it comes to RAG or retrieval augmented generation would be generating these reports.
00:17:01.920 | Because as much as a lawyer does want an answer, what they also want is the form factor they're used to.
00:17:06.080 | And so all of these graphs are consistently made and built each day. And then some subgraph from that
00:17:10.880 | broader monolithic structure is then brought in and composed into a report that a lawyer can action.
00:17:15.920 | Kind of what's next. And I'll talk about the future a little bit. I mean, what I described
00:17:22.080 | is kind of what we're doing. But this is what we're doing. Find law suits early, compensate harm,
00:17:27.440 | and then people can have that information if they want to. We're able to do this entirely technically.
00:17:31.680 | We're able to scrape the web structure, etc. We're able to iteratively build up a schema as we want to.
00:17:38.400 | So this is not just a Gen AI problem. And I think this is an important thing that I've seen around this
00:17:42.400 | conference and people may be seeing is that Gen AI is not better than machine learning. And LLMs are not
00:17:48.080 | you know, better than traditional ML systems. But there are situations in which one is fantastic and one is not.
00:17:53.920 | If you look at multi-agent systems, and again, I was previously an academic and multi-agent systems and no one ever
00:17:58.160 | listened to me. So this is a bizarre situation. But that when you used to structure the multi-agent systems together, somewhere along that workflow, you would have to stop
00:18:06.640 | or say this is not doable, because I cannot plug these two bits of information together. It's too probabilistic, or it's too random, or it's too inconsistent, or the way to describe it is not a binary feature, right?
00:18:16.640 | It is, I really just want to kind of type what I want, right? Now with LLMs, you can. But it's very much for us, not an LLM filtered system. It's an ML filtered system that LLMs have allowed us to pipe
00:18:28.640 | together such that you can actually provide value completely end to end, which I think was previously not doable. And for us, again, we've been using graphs for a long time.
00:18:36.640 | For us, the ability to iteratively build that graph, prune that graph, and every single report gets better, because we're able to manage the state, is why people like working with us, because we can consistently follow and track exactly what they want specifically.
00:18:50.640 | Cool. I think I'm just about at time, kind of got in early, but that's been the talk specifically around -- I'm happy to talk to anyone about the specifics, graph rag, et cetera, multi-agent systems, but that's how we use the process. Thank you very much.
00:19:03.640 | Thank you very much.
00:19:04.640 | Thank you very much.
00:19:05.640 | Thank you very much.
00:19:06.640 | Thank you very much.
00:19:07.640 | Thank you.
00:19:08.640 | We'll be right back.