Verta, Inc. Blog Are Small LLMs the Future?

LLMs are a transformative technology, but currently, their main limitation has been their size. The most powerful models have trillions of parameters and need trillions of tokens of training data. This makes LLMs very expensive to run, and puts the development of cutting-edge LLMs out of reach for all but a few resource-rich companies. It also raises responsible AI and ethical concerns, from model governance and control to energy consumption. For these reasons, we've been excitedly following the research into how to make LLMs smaller while still maintaining their capabilities. An important paper that recently came out in this space is Textbooks Are All You Need II: phi-1.5 technical report, by Microsoft Research. This paper explores how the authors built phi-1.5, a small LLM that achieves common sense reasoning benchmark results comparable to models that are 10x larger. We'll sum up our main takeaways here.