AI Concepts category
Fine-Tuning & Model Adaptation
Adapter
An adapter is a small trainable module added to a pretrained model so it can learn a new task while most base weights stay frozen.
Direct Preference Optimization
Direct Preference Optimization trains a model from preferred and rejected responses using a direct preference objective.
Fine-Tuning
Fine-tuning continues training a pretrained model on additional data to adapt its behavior for a downstream need.
Full Fine-Tuning
Full fine-tuning adapts a pretrained model by updating all or effectively all of its existing parameters.
LoRA
LoRA adapts a model by learning low-rank weight updates while keeping the original weights frozen.
Model Distillation
Model distillation trains a student model to reproduce selected behavior of a teacher model.
Parameter-Efficient Fine-Tuning
Parameter-efficient fine-tuning adapts a model by training a small subset of parameters or additional parameters instead of updating the full model.
Preference Fine-Tuning
Preference fine-tuning adapts a model using comparisons between candidate responses to the same input.
Prompt Tuning
Prompt tuning learns task-specific input embeddings while keeping a pretrained model's weights frozen.
Proximal Policy Optimization
Proximal Policy Optimization updates a model's response policy using reward feedback while discouraging large policy changes.
QLoRA
QLoRA trains LoRA adapters with a frozen, four-bit quantized base model to reduce fine-tuning memory use.
Reinforcement Learning from Human Feedback
Reinforcement Learning from Human Feedback uses human preference information to define a reward signal and optimize model behavior.
Reward Model
A reward model is a learned model that scores candidate outputs or intermediate steps, often using preference data.
Supervised Fine-Tuning
Supervised fine-tuning adapts a pretrained model using input-output examples with target responses.