Keep holdout cases out of prompt examples

Your support ticket classifier picks the wrong label for “I was charged twice”. You add the message to the prompt with the correct billing label. If the message stays in the holdout, the prompt gives away the answer on the next run.

Keep two fixtures. Use tuning.jsonl to edit the prompt. Once you’ve chosen a prompt, score it on holdout.jsonl. Put all variants of a ticket in the same fixture. Keep the original TicketId when you edit or export a case.

In the .NET eval runner, use only the tuning cases you check below as few-shot examples. Keep hard-coded examples out of the prompt template.

Check for overlap before the first model call. Compare the original TicketId and the exact Input text. Report the Id of each matching holdout row.

var tuningTicketIds = tuningCases.Select(c => c.TicketId)
    .ToHashSet(StringComparer.Ordinal);
var tuningInputs = tuningCases.Select(c => c.Input)
    .ToHashSet(StringComparer.Ordinal);

var overlappingCaseIds = holdoutCases
    .Where(c => tuningTicketIds.Contains(c.TicketId)
        || tuningInputs.Contains(c.Input))
    .Select(c => c.Id)
    .ToArray();

if (overlappingCaseIds.Length > 0)
{
    throw new InvalidOperationException(
        $"Holdout overlap in cases: {string.Join(", ", overlappingCaseIds)}");
}

Matching TicketId values catch rewrites of the same ticket. The input check catches exact copies even if you assign a new TicketId. A paraphrase with a different TicketId can slip through. Review likely duplicates when you split the data.

If a holdout failure leads to a prompt edit, move that ticket and its variants into tuning. Even after the move, the revised prompt reflects what you learned from the holdout. The same applies if you use the holdout score to choose a revision. In either case, use a fresh holdout to measure final performance.