The Data Lifecycle in LLM Development

GitHub Copilot · Data Infrastructure and Generative AI · Bởi Super Admin

The development of an LLM follows a general data lifecycle—problem framing, data collection, preprocessing, training, and deployment—that underpins how systems like GitHub Copilot are built.

Understanding how GitHub Copilot handles data starts with understanding the general flow of building a large language model (LLM). While the exact steps can vary and some details remain opaque, the development cycle generally includes problem framing, data collection, data preprocessing, training of the model, and model deployment and monitoring.

Problem Framing

The first step in creating an AI model is setting a goal—determining what problem is to be solved. For an LLM used in AI coding systems, this goal is often ambitious; OpenAI's stated mission, for example, is to ensure that artificial general intelligence (AGI) benefits all of humanity. Building these models can be extremely expensive and requires huge amounts of data, along with highly qualified data scientists, data engineers, and architects, whose compensation can reach seven figures. A growing trend is toward small language models (SLMs), which are a fraction of the size of LLMs but can still be powerful and fast, enabling better real-time interactions. SLMs are usually open-source, offering more transparency and customization, and Meta's Llama series SLMs have proven quite capable.

Data Collection

Data collection for an LLM is fairly straightforward in concept: it involves collecting publicly available data from the internet, a process made easier by modern scraping techniques and relatively cheap storage. However, substantial amounts of data sit behind paywalls, such as content from The New York Times, the Wall Street Journal, and the Washington Post, so major LLM developers have been licensing this type of content to train their models. There is also user-generated data, since major LLM developers hire employees to create content, and applications like ChatGPT and Claude generate user feedback that becomes a source of data.

Data Preprocessing

Data is always messy, especially when the dataset includes most of the internet, so data preprocessing is used to improve quality. The first stage is cleaning, which involves removing outliers, duplications, and missing information—a tedious process that companies like Scale AI specialize in, using large numbers of employees along with sophisticated automation systems. Next comes filtering, which strips harmful, toxic, or illegal content from the dataset. Another important step is tokenization, which turns data into a form that transformer models can process.

Training, Deployment, and Monitoring

Training the model involves processing the dataset with AI algorithms so it can learn patterns, focusing on how to generate content such as text, audio, code, or video. Because the dataset is unlabeled, this uses unsupervised learning, with the transformer model analyzing information to understand underlying relationships. The first version of training is usually far from useful, requiring ongoing fine-tuning, including Reinforcement Learning from Human Feedback (RLHF), where people evaluate the LLM's responses. Training can take months, is expensive, and requires chips like GPUs and TPUs—reflected in multibillion-dollar investments in OpenAI and Anthropic that were largely made as access to GPUs, TPUs, and cloud computing rather than cash. Once vetted, the model is deployed into production, typically on large cloud platforms like Azure, AWS, or Google Cloud, which provide the necessary capacity, state-of-the-art systems, scalability, and stringent security and compliance protections.

Key takeaways

  • LLM development generally follows five stages: problem framing, data collection, data preprocessing, training, and deployment/monitoring.
  • Small language models (SLMs) are a growing, often open-source alternative to LLMs offering speed, transparency, and customization.
  • Data collection combines public internet scraping, licensed paywalled content, and user-generated data.
  • Preprocessing includes cleaning, filtering harmful content, and tokenization, while training relies on unsupervised learning and RLHF fine-tuning using costly GPU/TPU resources.
0.0 (0 đánh giá)

Thảo luận / Bình luận (0)

Chưa có thảo luận nào cho bài viết này. Hãy gửi bình luận đầu tiên nhé!

Vui lòng Đăng nhập để tham gia thảo luận.