The content discusses a comprehensive evaluation comparing pre-translation with direct inference of PaLM2 on multilingual tasks. It highlights the improved performance of direct inference in the source language compared to pre-translation to English. The use of pre-translation has been a standard practice to address language bias issues in large language models (LLMs) due to skewed training data towards English. However, the study challenges the necessity of pre-translation with the introduction of powerful LLMs like PaLM2.
The evaluation includes discriminative and generative tasks across 108 languages, showing that PaLM2-L consistently outperforms pre-translation in 94 out of 108 languages. Direct inference is shown to be more efficient and effective in multilingual settings, unlocking linguistic authenticity and overcoming the limitations of pre-translation. The study also introduces the Language Ratio metric for a more nuanced understanding of LLM performance across languages.
While pre-translation shows superiority in some low-resource languages, the majority of languages benefit from direct inference with PaLM2. The findings suggest that the new generation of LLMs trained on multilingual datasets can handle communication across languages without the need for pre-translation in certain cases. The research is a joint effort of Verily AI and Google Research, with a commitment to ongoing research to improve LLM performance for all languages and promote inclusive multilingual communication.
Source link
Source link: http://research.google/blog/pre-translation-vs-direct-inference-in-multilingual-llm-applications/