ETRI begins development of a 100B-scale large foundation model

ETRI, South Korea’s leading government-funded research institute, is establishing itself as a key research entity for securing nation’s artificial intelligence sovereignty.

ETRI announced that it was selected as one of five national research teams as part of the NC AI Consortium in the independent AI foundation model development project, promoted by the Ministry of Science and ICT, and began full-scale AI research in August.

This selection is particularly significant as it solidifies ETRI’s status as a national research institute and secures its representation in the research community.

ETRI is participating in a consortium led by NC AI to carry out the project “Development of a Scalable Multimodal Generative Foundation Model for Industrial AI Transition.”

This project is a core project to implement large-scale artificial intelligence (AI) models that can be utilized in industrial settings, and its goal is to enhance multi-modal AI technology capable of integrating and processing diverse data such as text, voice, and images. By securing applicability focused on industrial sites, ETRI is expected to accelerate the innovation of AI across major industries such as manufacturing, medical, education, and culture.

Currently, ETRI has established a stable research infrastructure necessary for large-scale model development based on data support from the National Information Society Agency (NIA) and GPU resource support from the National IT Industry Promotion Agency (NIPA). Through this, the pre-training of a 100B (100 billion parameters)-scale model is being carried out smoothly, and it is expected to contribute to domestic technological independence in the development of ultra-large AI models based on proprietary technology in the future.

This project is a national strategy initiative aimed at strengthening AI competitiveness. It aims to independently secure Large Language Models (LLMs) and Multi-modal AI technologies and cultivates foundational technologies that can be utilized across all sectors, including industry, the public sector, and academia.

ETRI’s Intelligence Information Research Division is actively incorporating core technologies from its own national projects that have been carried out so far into this research.

The Language Intelligence Research Section has been applying the conceptual understanding and inference capabilities of the language model “Eagle”, developed through “Research on Artificial Complex Intelligence Core Technology”, and the sparse adapter-based persistent learning technology of the “Technologies for Maintaining Up-to-Date Generative Language Models” to large-scale models to enhance the model’s currency and efficiency.

The Embodied Integrated Intelligence Research Section is developing Multi-modal AI models focused on audio and video, based on existing research achievements such as “Technology for Assessing Degenerative Brain Function Decline” and “Multi-Speaker Conversation Modeling Technology,” and plans to expand this into general-purpose foundational models.

The Visual Intelligence Research Section is also conducting research to simultaneously enhance vision-language fusion generation performance and AI safety, based on the technology accumulated through text-based image generation model “KOALA” and visual language question-answering model “Ko-LLaVA.”

Oh-woog Kwon, assistant vice president of ETRI’s Intelligence Information Research Division and the project manager, stated, “Since being selected as the only government-funded research institute to represent the nation, ETRI has overcome initial resource acquisition challenges and has been steadily pre-training a 100B-scale model. Through a close collaboration with NC AI, we aim to lead the industrial AI transformation by developing a scalable and highly reliable multimodal foundation model and contributing to strengthening global top-tier technological capabilities.”

ETRI is using this opportunity to advance its roadmap, which connects technology development, large-scale pre-training, and industrial implementation. With the development of a 100B-scale model supported by NIA and NIPA yielding remarkable results, it is expected to mark an important turning point in South Korea’s pursuit of independent, ultra-large-scale AI technology capabilities.

ETRI is expected to play a pivotal role in transforming South Korea into a global AI technology powerhouse by leading the innovation of the AI ecosystem and the intelligent transformation of industries nationwide.

1) AI Foundation Model: AI core infrastructure that serves as a foundation for quickly and efficiently developing specialized AI in various fields using general-purpose artificial intelligence models trained using various large-scale data such as language, images, and audio.

2) 100B (100 billion parameters)-scale: A parameter is a unit of intelligence that AI models adjust through learning. The more parameters there are, the more complex patterns and contexts they can remember and understand. This is the baseline for large-scale foundation models capable of human-level language understanding, inference, and creative generation, achieving performance on par with top-tier models currently under development by global big tech companies.

3) Large Language Model (LLM): An artificial intelligence model that can understand and generate human language by learning vast amounts of text data. The most famous model is the GPT series, which is a core technology foundation for the AI ​​revolution. The expanded foundation models based on this are accelerating the development of artificial intelligence.

4) Multimodal AI Technology: An artificial intelligence technology that understands and processes different types of data, such as text, images, audio, and video. This AI technology aligns representations across various modalities to perform comprehensive judgment and inference, implementing multisensory cognitive abilities similar to humans.

5) Language model “Eagle”: The Korean language-centric language model developed by ETRI (planned to transition to a multimodal foundation model after 2026). Designed to strengthen Korean language, mathematics, and quantitative inference capabilities while focusing on lightness and efficiency, it can be easily used in research projects. Currently, the 1B, 3B, and 6.7B models are available on the website.
(https://huggingface.co/collections/etri-lirs/eagle-series)

6) Sparse adapter: An adapter technique that updates only a small subset of parameters instead of the full parameter set of a large foundation model, thereby increasing learning efficiency and reducing memory and computational costs by adjusting only a small number of active connections (parameter paths) in the entire model.

7) KOALA: A lightweight text-to-image generation AI model based on knowledge distillation that generates high-resolution images quickly and efficiently from text input and optimizes stable diffusion technology to reduce computational costs while providing high-quality results.

(https://huggingface.co/spaces/etri-vilab/KOALA)

8) Ko-LLaVA: A Korean-based vision-language multimodal model that understands images and texts together, performs question-and-answer in Korean, and enhances visual language understanding capabilities by learning the LLaVA structure with Korean data.

(https://huggingface.co/spaces/etri-vilab/Ko-LLaVA)

###

About Electronics and Telecommunications Research Institute (ETRI)

ETRI is a non-profit government-funded research institute. Since its foundation in 1976, ETRI, a global ICT research institute, has been making its immense effort to provide Korea a remarkable growth in the field of ICT industry. ETRI delivers Korea as one of the top ICT nations in the World, by unceasingly developing world’s first and best technologies.