AI Concepts category
Model Fundamentals
Attention Mechanism
Attention combines information from available representations using input-dependent weights.
Autoregressive Language Model
An autoregressive language model generates a sequence by predicting each next token from the tokens that precede it.
Cross-Entropy
Cross-entropy measures the expected negative log probability a model assigns to outcomes under a target distribution.
Embedding Model
An embedding model converts inputs into vectors whose geometry reflects learned patterns of similarity.
Entropy
Entropy measures how spread out the probabilities are across possible outcomes.
Foundation Model
A foundation model is pretrained broadly enough to serve as a reusable base for multiple downstream applications or adaptations.
Hallucination
A hallucination is false or unsupported content generated without an adequate basis in the model's input or available evidence.
Hyperparameter
A hyperparameter is a chosen setting that controls a model's design or training, rather than a value learned as a model parameter.
Language Model
A language model assigns probabilities to token sequences or token choices so it can score, predict, or generate language.
Large Language Model (LLM)
A large language model is a neural language model trained at substantial scale, with no fixed threshold for what counts as large.
Large Multimodal Model (LMM)
A large multimodal model works with more than one type of data, such as text, images, audio, or video.
Log Probability
A log probability is the logarithm of a token or sequence probability, used to inspect or score model continuations.
Logit
A logit is a model's raw score for a candidate token before those scores become probabilities.
Masked Language Model
A masked language model learns to recover selected tokens by using the surrounding context on both sides.
Mixture-of-Experts
A mixture-of-experts model combines expert subnetworks through a routing mechanism, often using only a subset for each input or token.
Model Parameter
A model parameter is a numerical value in a model, such as a weight or bias, that is typically learned during training.
Model Size
Model size is usually reported as the parameter count, which helps estimate resource needs but does not rank model quality.
Open-Weight Model
An open-weight model makes its trained parameters available to download and run, subject to its license.
Perplexity
Perplexity turns average next-token prediction loss into a score for how well a language model predicts a text sample.
Post-Training
Post-training adapts a pretrained model toward selected tasks and behavior.
Pretraining
Pretraining is the initial training phase that produces a pretrained model before later task- or behavior-specific adaptation.
Restricted-Weight Model
A restricted-weight model allows access to its trained parameters under limits on who can obtain them or how they may be used.
Self-Supervised Learning
Self-supervised learning derives training targets from the data rather than requiring a separate label for each example.
Tokenization
Tokenization converts input into discrete token identifiers that a model can process.
Transformer Architecture
A transformer is a neural-network architecture built from attention, feed-forward layers, and repeated processing blocks.
Vocabulary
A vocabulary is the fixed set of token entries and IDs a tokenizer shares with a model.