GPT Related Papers, Code, and News

GPT
OpenAI
Author

Lawrence Wu

Published

April 4, 2023

There’s seemingly a firehose of development in the last month or so. I’ve been trying to keep up with the latest developments in GPT and related models. Here’s a list of papers, code, and news that I’ve found interesting. This is mainly for myself to have a reference, but I hope it’s useful to others as well. I was largely inspired by @osanseviero who created ml_timeline.

Papers

  • 2022-08-21 - Emergent Abilities of Large Language Models (paper, blog)
  • 2023-03-13 - Alpaca – Stanford’s CRFM group released a 1.5B parameter GPT-3 like model. They were the first to demonstrate you can get GPT-like performance using only 52k instruction-following data points. On the self-instruct evaluation set, Alpaca shows many behaviors similar to OpenAI’s text-davinci-003, but is also surprisingly small and easy/cheap to reproduce. I think one reason OpenAI dropped their pricing by 90% with GPT-4 is because they wanted to achieve wide distribution of their model.
  • 2023-03-15 - GPT-4 Technical Paper (paper) - highlights some of the amazing improvements GPT-4 has made over GPT-3
  • 2023-03-27 - GPTs are GPTs: An Early Look at the Labor Market Impact Potential of Large Language Models (paper) – Paper that identifies the occupations that have the highest exposure to automation by GPT. In related news, ResumeBuilder found 1 in 4 companies have already replaced workers with ChatGPT
  • 2023-03-22 - Sparks of Artificial General Intelligence (paper)
  • 2023-03-20 – Reflexion: an autonomous agent with dynamic memory and self-reflection (paper). A related post.
  • 2023-03-23 - AI Explained – GPT4 can improve itself (video) - Intro to Reflexion and HuggingGPT
  • 2023-03-30 - HuggingGPT: Solving AI Tasks with ChatGPT and its Friends in HuggingFace (paper, code) - Using a LLM as brain, HuggingGPT identifies what HuggingFace models to use to solve tasks. Notably Microsoft is calling this JARVIS.

Libraries / Tools

  • Github Copilot - I use Copilot in my IDE, VS Code and it’s dramatically improved my producitivity (10-20%?). More than that it makes coding less tedious and lowers the activiation energy for coding tasks. For example generating docstrings is trivial (and happens much more frequently!). And because the recommendations are inline, the developer’s ‘flow’ is not broken. I also moved from Jupyter Notebooks in a browser to using Jupyter in VS Code. Radek Omulski has a blog post for how to set this up. I do plan to try GenAI as well. I tried GenAI and it basically automatically sends all errors to ChatGPT and provides suggested corrected syntax to try in line in your Jupyter notebook. It actually can be a nice complement to Copilot.
  • LangChain - Building applications with LLMs through composability
  • llama_index - LlamaIndex (GPT Index) is a project that provides a central interface to connect your LLM’s with external data.
  • GenAI - generative AI tooling for IPython
  • marvin - Meet Marvin: a batteries-included library for building AI-powered software. Marvin’s job is to integrate AI directly into your codebase by making it look and feel like any other function.

Prompt Engineering

Prompt engineering is the process of creating prompts for LLMs. Essentially optimizing the input into LLMs.

  • What is Prompt Engineering - like how Googling became a skill (aka “Google-fu”), I think Prompt Engineering is an important skill to develop
  • awesome-chatgpt-prompts - A curated list of awesome ChatGPT prompts. I like “Act as a Linux Terminal” prompt.
  • Prompt Engineering Guide - “Motivated by the high interest in developing with LLMs, we have created this new prompt engineering guide that contains all the latest papers, learning guides, lectures, references, and tools related to prompt engineering.” Code: repo.

Output Parsers

Where prompt engineering works on the input to LLMs, output parsers work on the output.

  • Output Parsers - LangChain calls this “Output Parsers”. LangChain can return a list, JSON, any Python type (using Pydantic) and two other ways of handling outputs: OutputFixingParser and RetryOutputParser
  • Enforcing AI Format - Marvin calls this “Enforcing AI format”. Marvin can return a string, list of dicts, JSON or really any Python type (using Pydantic)

Predictions

  • 2023-04-01 - @AllenDowney predicts “The great majority of coding will be LLM-assisted, starting now.” (tweet, blog)