From Text to Numbers: Tokens, Chunking, and Embeddings

AWS Certified AI Practitioner · Fundamentals of Generative AI · By Super Admin

Generative AI models convert language into numbers through tokenization, chunking, and embeddings, enabling them to understand and generate coherent text.

Before a generative AI model can produce human-like responses, it must first convert human language into a form a computer can understand. This transformation happens through a series of steps that turn raw text into numerical representations, starting with tokenization and ending with embeddings.

Tokenization: Breaking Text into Tokens

When text such as "Gen AI is fascinating" is entered into a model, the model does not read the sentence all at once. Instead, it uses tokenization to break the text into smaller, more manageable units called tokens. These tokens can be entire words, subwords, or even single characters, depending on what the model's tokenizer algorithm determines is necessary. This can be imagined as a box of Lego bricks, where each brick is a token and the entire sentence is the assembled structure; working with smaller bricks lets the model handle unfamiliar words or word variations more flexibly. For example, a subword-based tokenizer might break "Tokenization simplifies language processing" into the tokens "Token," "ization," "simplifies," "language," "processing," and "." — splitting "Tokenization" into "Token" and "ization" because the full word may be less frequently encountered, allowing the model to handle unseen words using familiar components. Tokens are the atomic units in Gen AI, and the number of tokens directly impacts how long it takes to analyze or generate text: longer texts have more tokens, which can raise computational costs and slow down response times. Token limits, such as 4096 or 8192 tokens, often define how much text can be fed into or received from the model in a single interaction.

Chunking: Grouping Related Tokens

Once text is broken into tokens, the model may group related tokens together through a process called chunking. If tokenization is like snapping Lego bricks apart, chunking is like gathering the pieces that form meaningful sub-assemblies. For instance, the phrase "Gen AI" might be grouped as a single conceptual unit because it frequently appears together in various contexts, helping the model understand that it refers to a specific type of technology rather than a random assortment of words. By preserving these logical groupings, chunking allows the model to better capture relationships among tokens, leading to more coherent and contextually appropriate responses.

Embeddings: Converting Tokens into Vectors

Tokens and chunks mean nothing to a computer unless they are converted into numerical form, and this is the role of embeddings, which convert each token into a vector. A vector is a quantity with direction and magnitude, a concept often used in physics; in Gen AI, a vector can be thought of as an "address" in a mathematical space with hundreds or even thousands of coordinates, unlike the familiar x, y, and z axes. Each token—such as "king," "queen," "computer," or "laptop"—is assigned a unique vector in this high-dimensional map. Words with similar meanings or uses tend to cluster near each other, while dissimilar words lie far apart, similar to a galaxy of words where related terms form constellations, such as "king" and "queen" orbiting one region while "computer" and "laptop" cluster in another. By converting tokens into vectors, embeddings enable the AI model to recognize semantic relationships even when words don't appear in the exact same context, forming the foundation of the model's ability to understand nuance and generate coherent sentences, whether summarizing text or crafting new sentences.

Figure: Illustration depicts the process of converting raw text input through tokenization process, resulting in embedding vectors of sectors.

Key takeaways

  • Tokenization breaks text into tokens—words, subwords, or characters—that serve as the atomic units processed by Gen AI models.
  • Token count affects computational cost and response time, and models have token limits (e.g., 4096 or 8192 tokens) per interaction.
  • Chunking groups related tokens together to preserve context and logical relationships, improving coherence of model output.
  • Embeddings convert tokens into high-dimensional vectors, allowing models to capture semantic relationships between words based on proximity in vector space.
0.0 (0 ratings)

Discussion (0)

No discussion on this article yet — be the first to comment.

Please sign in to join the discussion.