What I'm excited about is, we had this survey on augmented LLM a year ago, and all the idea is like, if you augment your LLM with something else, it can be a retriever, it can be search, it can be a tool, it can be a calculator, it can be a code execution, then you are not just distillating, like doing some data augmentation with your model, but you're actually adding some expert skills that possibly goes beyond the model weights.

For instance, if your model can calculate something it was wrong before, and now it has access to a calculator, and you can retrain your model on that, then you're learning something new. If your model didn't know something about LLM 2, it probably doesn't know a lot about LLM 3, but now if it can search online about it, and then you train the model on that, Then you have a positive feedback loop like what we call expert iteration.