Tech companies are shifting their focus from building large language models (LLMs) to developing smaller ones (SLMs) that can match or even outperform them. Examples of large models include Meta’s Llama 3, OpenAI’s GPT-3.5 and GPT-4, while smaller models like Microsoft’s Phi-3 family and Apple Intelligence are gaining attention for their efficiency and practicality. SLMs consume less energy, can run on devices like smartphones, and are more accessible to smaller businesses and labs.
The performance gap between LLMs is narrowing, prompting tech companies to explore alternative avenues for upgrades. Recent tests by Microsoft showed that their smallest model, Phi-3-mini, with 3.8 billion parameters, rivaled larger models in certain areas due to its training dataset. While SLMs can achieve similar language understanding and reasoning as larger models, they are limited in storing factual knowledge. Combining SLMs with online search engines can address this limitation.
Comparisons have been made between SLMs and how children learn language, with the potential for reverse engineering efficient human-like learning at small scales to lead to improvements when scaled up to LLM scales. The efficiency of human learning remains a mystery, but researchers like Alex Warstadt suggest that understanding this process could result in significant advancements in AI technology.
Source link
Source link: https://www.inkl.com/news/no-one-knows-what-makes-humans-so-much-more-efficient-small-language-models-based-on-homo-sapiens-could-help-explain-how-we-learn-and-improve-ai-efficiency-for-better-or-for-worse