Large language models (LLMs) such as ChatGPT have demonstrated impressive capabilities in following instructions, reasoning over text, and maintaining natural language dialogue. However, training state-of-the-art LLMs remains highly costly in terms of computation and data, restricting their development to a few organizations and limiting domain-specific applications.

In this talk, I will give an overview of our group’s work on efficient machine learning methods aimed at improving both the sample and resource efficiency of LLM training, as well as on systematic approaches to evaluating pretrained models. I will present recent results on developing LLM architectures with subquadratic complexity and discuss our current effort to train a large state-of-the-art LLM for the German language.