back to index

LangChain Interrupt 2025 JP Morgan Building Ask D A V I D – Zheng Xue


Whisper Transcript | Transcript Only Page

00:00:00.040 | We're really excited about all the possibilities that ask David
00:00:03.480 | and potentially bring to that paper. So now we dive deeper into the technical
00:00:08.100 | magic behind Ask David. I'll turn it over to Jane who will walk through the notes and bolts of
00:00:13.020 | how we are making our vision or reality. And she might even let you know what Ask David
00:00:17.520 | stands for because I promise you I didn't name this out for myself. Thank you.
00:00:22.280 | Thank you David. So Ask David is a human-specific QA agent. So let's start with
00:00:29.000 | epidemiology analysis. First of all, we have the database of structured data. Those are the
00:00:35.000 | backcodes of many other running production systems. Prior to the introduction of an agent,
00:00:40.000 | users have access to the same data. But they have to navigate through different
00:00:44.000 | systems and manually navigate their information. An agent can introduce
00:00:49.000 | efficiency and integrated user experience. Next, we have unstructured data. As a bank, we
00:00:58.000 | have a vast amount of documentation, including emails, emails, presentations. With the rise
00:01:05.000 | of virtual readings, we also have increasing amount of video and audio recordings. How do we make
00:01:12.000 | for use of data information? The advancement of LRRM really bring in tremendous opportunity in this area. Lastly, as a research team, we have applied proprietary models and analytics, which are designed to really provide insights and visualization to help disseminate it. Previously, it will require a human expert to conduct this kind of analysis and offering a write-off survey. With the help of an agent, we can scale the
00:01:40.000 | inside generation and we can make our service available to more of our clients.
00:01:47.000 | Now imagine, being a financial advisor in a planning meeting, and your clients suddenly bring out the fund and ask you why it's terminating.
00:01:57.000 | Believe me, it's actually a very loaded question. So, in the past, you would reach out to our investment research team, talk with real debate, and then you figure out what's the strategy and the history of the fund and what's the reason behind it. What's the research about this fund? What are similar funds? How do I encourage these answers?
00:02:16.000 | specific for this fund? And you will come up with that presentation yourself manually. With the help of an agent, we can get access to the same data, analytics, insights, and visualization, writing or reading, enable the real-time decision-making. That is our vision of us, David. And you probably guessed it. David stands for data, analytics, visualization, insights, and decision-making data.
00:02:43.000 | So, this is our approach to build us, David, which is a multi-agent system.
00:02:50.000 | So, we found our supervisor agent, which acts as an officer in talks with other users, understand their intention, and try to delegate the task to one or more of the staff agents in the team. The supervisor agents have access to both short-term and long-term libraries, so that they can customize the user experience. It also knows when to invoke you in the loop to ensure the highest level of accuracy and automation.
00:02:51.000 | And also reliability.
00:02:52.000 | So, we found it.
00:02:53.000 | So, it's a multi-agent system.
00:02:53.000 | So, it's a multi-agent system.
00:02:53.000 | It's a multi-agent system.
00:02:54.000 | It's a multi-agent system.
00:02:55.000 | It's a multi-agent system.
00:02:56.000 | It's a multi-agent system.
00:02:57.000 | It's a multi-agent system.
00:02:58.000 | It's a multi-agent system.
00:02:58.000 | It's a multi-agent system.
00:02:59.000 | It's a multi-agent system.
00:03:00.000 | It's a multi-agent system.
00:03:01.000 | It's a multi-agent system.
00:03:02.000 | It's a multi-agent system.
00:03:03.000 | It's a multi-agent system.
00:03:04.000 | It's a multi-agent system.
00:03:05.000 | It's a multi-agent system.
00:03:06.000 | It's a multi-agent system.
00:03:07.000 | It's a multi-agent system.
00:03:08.000 | It's a multi-agent system.
00:03:09.000 | It's a multi-agent system.
00:03:10.000 | It's a multi-agent system.
00:03:11.000 | It's a multi-agent system.
00:03:12.000 | It's a multi-agent system.
00:03:13.000 | It's a multi-agent system.
00:03:14.000 | It's a multi-agent system.
00:03:15.000 | It's a multi-agent system.
00:03:16.000 | It's a multi-agent system.
00:03:17.000 | It's a multi-agent system.
00:03:18.000 | It's a multi-agent system.
00:03:19.000 | It's a multi-agent system.
00:03:20.000 | It's a multi-agent system.
00:03:21.000 | It's a multi-agent system.
00:03:22.000 | It's a multi-agent system.
00:03:23.000 | It's a multi-agent system.
00:03:24.000 | It's a multi-agent system.
00:03:25.000 | It's a multi-agent system.
00:03:26.000 | It's a multi-agent system.
00:03:27.000 | It's a multi-agent system.
00:03:28.000 | It's a multi-agent system.
00:03:29.000 | It's a multi-agent system.
00:03:30.000 | It's a multi-agent system.
00:03:31.000 | It's a multi-agent system.
00:03:32.000 | It's a multi-agent system.
00:03:33.000 | It's a multi-agent system.
00:03:34.000 | It's a multi-agent system.
00:03:35.000 | It's a multi-agent system.
00:03:36.000 | It's a multi-agent system.
00:03:37.000 | It's a multi-agent system.
00:03:38.000 | It's a multi-agent system.
00:03:39.000 | It's a multi-agent system.
00:03:40.000 | It's a multi-agent system.
00:03:41.000 | It's a multi-agent system.
00:03:42.000 | It's a multi-agent system.
00:03:43.000 | It's a multi-agent system.
00:03:44.000 | It's a multi-agent system.
00:03:45.000 | It's a multi-agent system.
00:03:46.000 | It's a multi-agent system.
00:03:47.000 | It's a multi-agent system.
00:03:48.000 | It's a multi-agent system.
00:03:49.000 | It's a multi-agent system.
00:03:50.000 | It's a multi-agent system.
00:03:51.000 | It's a multi-agent system.
00:03:52.000 | It's a multi-agent system.
00:03:53.000 | It's a multi-agent system.
00:03:54.000 | It's a multi-agent system.
00:03:55.000 | It's a multi-agent system.
00:03:56.000 | It's a multi-agent system.
00:03:57.000 | It's a multi-agent system.
00:03:58.000 | It's a multi-agent system.
00:03:59.000 | It's a multi-agent system.
00:04:00.000 | It's a multi-agent system.
00:04:01.000 | It's a multi-agent system.
00:04:02.000 | It's a multi-agent system.
00:04:03.000 | It's a multi-agent system.
00:04:04.000 | It's a multi-agent system.
00:04:05.000 | It's a multi-agent system.
00:04:06.000 | It's a multi-agent system.
00:04:07.000 | It's a multi-agent system.
00:04:08.000 | It's a multi-agent system.
00:04:09.000 | It's a multi-agent system.
00:04:10.000 | It's a multi-agent system.
00:04:11.000 | It's a multi-agent system.
00:04:12.000 | It's a multi-agent system.
00:04:13.000 | It's a multi-agent system.
00:04:14.000 | It's a multi-agent system.
00:04:15.000 | It's a multi-agent system.
00:04:16.000 | It's a multi-agent system.
00:04:17.000 | It's a multi-agent system.
00:04:18.000 | It's a multi-agent system.
00:04:19.000 | It's a multi-agent system.
00:04:20.000 | It's a multi-agent system.
00:04:21.000 | It's a multi-agent system.
00:04:22.000 | It's a multi-agent system.
00:04:23.000 | It's a multi-agent system.
00:04:24.000 | It's a multi-agent system.
00:04:25.000 | It's a multi-agent system.
00:04:26.000 | It's a multi-agent system.
00:04:27.000 | It's a multi-agent system.
00:04:28.000 | It's a multi-agent system.
00:04:29.000 | It's a multi-agent system.
00:04:30.000 | It's a multi-agent system.
00:04:31.000 | It's a multi-agent system.
00:04:32.000 | It's a multi-agent system.
00:04:33.000 | It's a multi-agent system.
00:04:34.000 | It's a multi-agent system.
00:04:35.000 | It's a multi-agent system.
00:04:36.000 | It's a multi-agent system.
00:04:37.000 | It's a multi-agent system.
00:04:38.000 | It's a multi-agent system.
00:04:39.000 | It's a multi-agent system.
00:04:40.000 | It's a multi-agent system.
00:04:41.000 | It's a multi-agent system.
00:04:42.000 | It's a multi-agent system.
00:04:43.000 | It's a multi-agent system.
00:04:44.000 | It's a multi-agent system.
00:04:45.000 | It's a multi-agent system.
00:04:46.000 | It's a multi-agent system.
00:04:47.000 | It's a multi-agent system.
00:04:48.000 | It's a multi-agent system.
00:04:49.000 | It's a multi-agent system.
00:04:50.000 | It's a multi-agent system.
00:04:51.000 | It's a multi-agent system.
00:04:52.000 | It's a multi-agent system.
00:04:53.000 | It's a multi-agent system.
00:04:54.000 | It's a multi-agent system.
00:04:55.000 | It's a multi-agent system.
00:04:56.000 | It's a multi-agent system.
00:04:57.000 | It's a multi-agent system.
00:04:58.000 | It's a multi-agent system.
00:04:59.000 | It's a multi-agent system.
00:05:00.000 | It's a multi-agent system.
00:05:01.000 | It's a multi-agent system.
00:05:02.000 | It's a multi-agent system.
00:05:03.000 | It's a multi-agent system.
00:05:04.000 | It's a multi-agent system.
00:05:05.000 | It's a multi-agent system.
00:05:06.000 | It's a multi-agent system.
00:05:07.000 | It's a multi-agent system.
00:05:08.000 | It's a multi-agent system.
00:05:09.000 | It's a multi-agent system.
00:05:10.000 | It's a multi-agent system.
00:05:11.000 | It's a multi-agent system.
00:05:12.000 | It's a multi-agent system.
00:05:13.000 | It's a multi-agent system.
00:05:14.000 | It's a multi-agent system.
00:05:15.000 | It's a multi-agent system.
00:05:16.000 | It's a multi-agent system.
00:05:17.000 | It's a multi-agent system.
00:05:18.000 | It's a multi-agent system.
00:05:19.000 | It's a multi-agent system.
00:05:20.000 | It's a multi-agent system.
00:05:21.000 | It's a multi-agent system.
00:05:22.000 | It's a multi-agent system.
00:05:23.000 | It's a multi-agent system.
00:05:24.000 | It's a multi-agent system.
00:05:25.000 | It's a multi-agent system.
00:05:26.000 | It's a multi-agent system.
00:05:27.000 | It's a multi-agent system.
00:05:28.000 | It's a multi-agent system.
00:05:29.000 | It's a multi-agent system.
00:05:30.000 | It's a multi-agent system.
00:05:31.000 | It's a multi-agent system.
00:05:32.000 | It's a multi-agent system.
00:05:33.000 | It's a multi-agent system.
00:05:34.000 | It's a multi-agent system.
00:05:35.000 | It's a multi-agent system.
00:05:36.000 | It's a multi-agent system.
00:05:37.000 | It's a multi-agent system.
00:05:38.000 | It's a multi-agent system.
00:05:39.000 | It's a multi-agent system.
00:05:40.000 | It's a multi-agent system.
00:05:41.000 | It's a multi-agent system.
00:05:42.000 | It's a multi-agent system.
00:05:43.000 | It's a multi-agent system.
00:05:44.000 | It's a multi-agent system.
00:05:45.000 | It's a multi-agent system.
00:05:46.000 | It's a multi-agent system.
00:05:47.000 | It's a multi-agent system.
00:05:48.000 | It's a multi-agent system.
00:05:49.000 | It's a multi-agent system.
00:05:50.000 | It's a multi-agent system.
00:05:51.000 | It's a multi-agent system.
00:05:52.000 | It's a multi-agent system.
00:05:53.000 | It's a multi-agent system.
00:05:54.000 | It's a multi-agent system.
00:05:55.000 | It's a multi-agent system.
00:05:56.000 | It's a multi-agent system.
00:05:57.000 | It's a multi-agent system.
00:05:58.000 | It's a multi-agent system.
00:05:59.000 | It's a multi-agent system.
00:06:00.000 | It's a multi-agent system.
00:06:01.000 | It's a multi-agent system.
00:06:02.000 | It's a multi-agent system.
00:06:03.000 | It's a multi-agent system.
00:06:04.000 | It's a multi-agent system.
00:06:05.000 | It's a multi-agent system.
00:06:06.000 | It's a multi-agent system.
00:06:07.000 | It's a multi-agent system.
00:06:08.000 | It's a multi-agent system.
00:06:09.000 | It's a multi-agent system.
00:06:10.000 | It's a multi-agent system.
00:06:11.000 | It's a multi-agent system.
00:06:12.000 | It's a multi-agent system.
00:06:13.000 | It's a multi-agent system.
00:06:14.000 | It's a multi-agent system.
00:06:15.000 | It's a multi-agent system.
00:06:16.000 | It's a multi-agent system.
00:06:17.000 | It's a multi-agent system.
00:06:18.000 | It's a multi-agent system.
00:06:19.000 | It's a multi-agent system.
00:06:20.000 | It's a multi-agent system.
00:06:21.000 | It's a multi-agent system.
00:06:22.000 | It's a multi-agent system.
00:06:23.000 | It's a multi-agent system.
00:06:24.000 | It's a multi-agent system.
00:06:25.000 | It's a multi-agent system.
00:06:26.000 | It's a multi-agent system.
00:06:27.000 | It's a multi-agent system.
00:06:28.000 | It's a multi-agent system.
00:06:29.000 | It's a multi-agent system.
00:06:30.000 | It's a multi-agent system.
00:06:31.000 | It's a multi-agent system.
00:06:32.000 | It's a multi-agent system.
00:06:33.000 | It's a multi-agent system.
00:06:34.000 | It's a multi-agent system.
00:06:35.000 | It's a multi-agent system.
00:06:36.000 | It's a multi-agent system.
00:06:37.000 | It's a multi-agent system.
00:06:38.000 | It's a multi-agent system.
00:06:39.000 | It's a multi-agent system.
00:06:40.000 | It's a multi-agent system.
00:06:41.000 | It's a multi-agent system.
00:06:42.000 | It's a multi-agent system.
00:06:43.000 | It's a multi-agent system.
00:06:44.000 | It's a multi-agent system.
00:06:45.000 | It's a multi-agent system.
00:06:46.000 | It's a multi-agent system.
00:06:47.000 | It's a multi-agent system.
00:06:48.000 | It's a multi-agent system.
00:06:49.000 | It's a multi-agent system.
00:06:50.000 | It's a multi-agent system.
00:06:51.000 | It's a multi-agent system.
00:06:52.000 | It's a multi-agent system.
00:06:53.000 | It's a multi-agent system.
00:06:54.000 | It's a multi-agent system.
00:06:55.000 | It's a multi-agent system.
00:06:56.000 | It's a multi-agent system.
00:06:57.000 | It's a multi-agent system.
00:06:58.000 | It's a multi-agent system.
00:06:59.000 | It's a multi-agent system.
00:07:00.000 | It's a multi-agent system.
00:07:01.000 | It's a multi-agent system.
00:07:02.000 | It's a multi-agent system.
00:07:03.000 | It's a multi-agent system.
00:07:04.000 | It's a multi-agent system.
00:07:05.000 | It's a multi-agent system.
00:07:06.000 | It's a multi-agent system.
00:07:07.000 | It's a multi-agent system.
00:07:08.000 | It's a multi-agent system.
00:07:09.000 | It's a multi-agent system.
00:07:10.000 | It's a multi-agent system.
00:07:11.000 | It's a multi-agent system.
00:07:12.000 | It's a multi-agent system.
00:07:13.000 | It's a multi-agent system.
00:07:14.000 | It's a multi-agent system.
00:07:15.000 | It's a multi-agent system.
00:07:16.000 | It's a multi-agent system.
00:07:17.000 | It's a multi-agent system.
00:07:18.000 | It's a multi-agent system.
00:07:19.000 | It's a multi-agent system.
00:07:20.000 | It's multi-agent system.
00:07:21.000 | It's a multi-agent system.
00:07:22.000 | It's a multi-agent system.
00:07:23.000 | It's a multi-agent system.
00:07:24.000 | It's a multi-agent system.
00:07:25.000 | It's a multi-agent system.
00:07:26.000 | It's a multi-agent system.
00:07:27.000 | It's a multi-agent system.
00:07:28.000 | It's a multi-agent system.
00:07:29.000 | It's a multi-agent system.
00:07:30.000 | It's a multi-agent system.
00:07:31.000 | It's a multi-agent system.
00:07:32.000 | It's a multi-agent system.
00:07:33.000 | It's a multi-agent system.
00:07:34.000 | It's a multi-agent system.
00:07:35.000 | It's a multi-agent system.
00:07:36.000 | It's a multi-agent system.
00:07:37.000 | It's a multi-agent system.
00:07:38.000 | It's a multi-agent system.
00:07:39.000 | It's a multi-agent system.
00:07:40.000 | It's a multi-agent system.
00:07:41.000 | It's a multi-agent system.
00:07:42.000 | It's a multi-agent system.
00:07:43.000 | It's a multi-agent system.
00:07:44.000 | It's a multi-agent system.
00:07:45.000 | It's a multi-agent system.
00:07:46.000 | It's a multi-agent system.
00:07:47.000 | It's a multi-agent system.
00:07:48.000 | It's a multi-agent system.
00:07:49.000 | It's a multi-agent system.
00:07:50.000 | It's a multi-agent system.
00:07:51.000 | It's a multi-agent system.
00:07:52.000 | It's a multi-agent system.
00:07:53.000 | It's a multi-agent system.
00:07:54.000 | It's a multi-agent system.
00:07:55.000 | It's a multi-agent system.
00:07:56.000 | It's a multi-agent system.
00:07:57.000 | It's a multi-agent system.
00:07:58.000 | It's a multi-agent system.
00:07:59.000 | It's a multi-agent system.
00:08:00.000 | It's a multi-agent system.
00:08:01.000 | It's a multi-agent system.
00:08:02.000 | It's a multi-agent system.
00:08:03.000 | It's a multi-agent system.
00:08:04.000 | It's a multi-agent system.
00:08:05.000 | It's a multi-agent system.
00:08:06.000 | It's a multi-agent system.
00:08:07.000 | It's a multi-agent system.
00:08:08.000 | It's a multi-agent system.
00:08:09.000 | It's a multi-agent system.
00:08:10.000 | It's a multi-agent system.
00:08:11.000 | It's a multi-agent system.
00:08:12.000 | It's a multi-agent system.
00:08:13.000 | It's a multi-agent system.
00:08:14.000 | It's a multi-agent system.
00:08:15.000 | It's a multi-agent system.
00:08:16.000 | It's a multi-agent system.
00:08:17.000 | It's a multi-agent system.
00:08:18.000 | It's a multi-agent system.
00:08:19.000 | It's a multi-agent system.
00:08:20.000 | So, everyone knows that compared to the traditional AI projects,
00:08:26.000 | CNI projects actually have a shorter development phase.
00:08:29.000 | But we do have a long evaluation phase.
00:08:32.000 | Right?
00:08:33.000 | So, our suggestion is to start early.
00:08:35.000 | Think about metrics.
00:08:36.000 | What kind of goals do you want to achieve?
00:08:38.000 | Right?
00:08:39.000 | We are in the financial industry.
00:08:40.000 | Obviously, accuracy is one of the most important things.
00:08:45.000 | And the continuous evaluation helps you get that confidence.
00:08:48.000 | You are improving day by day.
00:08:49.000 | So, there are additional tips I have over here based on our own experience of evaluation.
00:08:53.000 | As you can see on the screen, right?
00:08:55.000 | So, the dark blue bars over here are coming from the metrics of evaluation on main flow.
00:09:01.000 | And the big one is actually one example of our sub-agent.
00:09:05.000 | So, my tip number one is make sure you independently evaluate your sub-agents.
00:09:10.000 | Well, the key for evaluation is to find places to improve, right?
00:09:14.000 | These help you figure out what we think to improve your accuracy.
00:09:20.000 | Second point is, depending on how you design your agents, make sure you pick the right metrics.
00:09:27.000 | So, if you have a summarization, you may want to check whether your summarization is concise or not.
00:09:32.000 | So, concisiveness is one of the metrics you want to pick.
00:09:36.000 | If you are doing a true call, maybe you can have a trajectory evaluation instead.
00:09:41.000 | So, here is a common myth.
00:09:45.000 | I mean, especially if you are a developer, you talk about TDD.
00:09:49.000 | I think a lot of people say that I just don't do that.
00:09:51.000 | It's a lot of work, right?
00:09:52.000 | But it's not the same in the evaluation.
00:09:55.000 | You actually can start evaluation with or without long-truth.
00:10:00.000 | It doesn't matter.
00:10:01.000 | There are so many metrics beyond just accuracy.
00:10:04.000 | And each one of them will provide you some insight.
00:10:07.000 | And you know what?
00:10:08.000 | Once you start doing evaluation, you will have review.
00:10:11.000 | Once you start doing review, you actually can accumulate more of the long-truth examples.
00:10:16.000 | Lastly, we help large learning models help us judge in combination of human review.
00:10:25.000 | These automated solutions really help us scale without adding too much burdens to our human SME to review large amount of AI-generated answers.
00:10:35.000 | Talking about SME, our last lesson learned over here is about human SME in the loop.
00:10:41.000 | When you apply a general model to a specific event, usually you will get less than 50% of accuracy.
00:10:49.000 | But you can do a quick improvement like charting strategies.
00:10:54.000 | You can change your searching outcomes and then you can actually make improvement in engineering.
00:11:00.000 | That can get you to that 80% mark.
00:11:03.000 | From 80 to 90, we are using the workflow chains.
00:11:06.000 | We are creating the subgraph so that we can fine-tune certain kind of questions without impacting each other.
00:11:12.000 | Between 90% and 100%, that's what we call the last mile.
00:11:17.000 | And the last mile is always the hardest mile.
00:11:20.000 | In terms of GNI applications, it may not be achievable to get that 100% mark, right?
00:11:26.000 | So what do we do?
00:11:27.000 | So human SME in the loop is very important to us.
00:11:30.000 | Because we have billion dollars at stake and we cannot afford embarrassing.
00:11:36.000 | In other words, "Ask David" still consults with real David whenever you need it.
00:11:43.000 | In conclusion, three takeaways for you.
00:11:46.000 | Each way of us, evaluate early.
00:11:48.000 | Keep giving us the new loop.
00:11:50.000 | Thank you so much.
00:11:52.000 | Thank you very much.
00:11:53.000 | Thank you very much.
00:11:53.000 | Thank you very much.
00:11:54.000 | Thank you very much.
00:11:54.000 | Thank you very much.
00:11:55.000 | Thank you very much.
00:11:56.000 | Thank you very much.
00:11:56.000 | Thank you very much.
00:11:57.000 | Thank you very much.
00:11:59.000 | Thank you very much.