AI Concepts category
Model Fundamentals
Core concepts behind modern foundation models and how they work.
Attention Mechanism
Attention combines information from available representations using input-dependent weights.
Autoregressive Language Model
An autoregressive language model generates a sequence by predicting each next token from the tokens that precede it.
Foundation Model
A foundation model is pretrained broadly enough to serve as a reusable base for multiple downstream applications or adaptations.
Mixture-of-Experts
A mixture-of-experts model combines expert subnetworks through a routing mechanism, often using only a subset for each input or token.
Tokenization
Tokenization converts input into discrete token identifiers that a model can process.
Transformer Architecture
A transformer is a neural-network architecture built from attention, feed-forward layers, and repeated processing blocks.