ETH Zurich and EPFL to release a LLM developed on public infrastructure

Summary

Earlier this week in Geneva, around 50 leading global initiatives and organisations dedicated to open-source LLMs and trustworthy AI convened at the International Open-Source LLM Builders Summit. Hosted by the AI centres of EPFL and ETH Zurich, the event marked a significant step in building a vibrant and collaborative international ecosystem for open foundation models. Open LLMs are increasingly viewed as credible alternatives to commercial systems, most of which are developed behind closed doors in the United States or China. Participants of the summit previewed the forthcoming release of a fully open, publicly developed LLM — co-created by researchers at EPFL, ETH Zurich and other Swiss universities in close collaboration with engineers at CSCS. Currently in final testing, the model will be downloadable under an open license. The model focuses on transparency, multilingual performance, and broad accessibility. The model will be fully open: source code and weights will be publicly available, and the training data will be transparent and reproducible, supporting adoption across science, government, education, and the private sector. This approach is designed to foster both innovation and accountability. “Fully open models enable high-trust applications and are necessary for advancing research about the risks and opportunities of AI. Transparent processes also enable regulatory compliance,” says Imanol Schlag, research scientist at the ETH AI Center, who is leading the effort alongside EPFL AI Center faculty members and professors Antoine Bosselut and Martin Jaggi. Multilingual by design A defining characteristic of the LLM is its fluency in over 1000 languages. “We have emphasised making the models massively multilingual from the start,” says Antoine Bosselut. Training of the base model was done on a large text dataset in over 1500 languages — approximately 60% English and 40% non-English languages — as well as code and mathematics data. Given the representation of...