Definition
Functional correctness describes whether an output satisfies a task’s required behavior when executed or applied. The evaluation applies checks derived from the task’s requirements and compares the observed results with the expected behavior. It does not require the output to use the same wording or implementation as a reference answer.
This approach is useful when several outputs can solve the same problem. Generated code, queries, and tool arguments can look different while producing the same valid result.
Simple example
Suppose a model must write a C# function that converts strings such as 1h 30m into a TimeSpan. The evaluator compiles the generated code and runs tests for valid values, extra whitespace, malformed input, and boundary cases.
One solution may use a regular expression while another splits the input into tokens. Both pass if they satisfy the declared behavior. A response that closely resembles the reference implementation still fails if it does not compile or returns the wrong value for a test case.
Why it matters
Text similarity can penalize valid solutions simply because they take a different path. Functional checks measure the behavior that the application depends on instead.
That makes functional-correctness checks useful for regression testing generated code and other outputs with executable contracts. A team can change a prompt, model, or tool without requiring every successful output to stay textually identical. Failures are also concrete: the output did not compile, a test case returned the wrong result, or a required constraint was violated.
One important nuance
A test-based assessment of functional correctness is only as strong as its checks. Passing five unit tests proves that the output passed those five tests, not that it works for every valid input. Weak checks can miss hard-coded answers, unwanted side effects, poor performance, or unsafe behavior.
Use representative cases, boundary conditions, and hidden tests where appropriate. Run untrusted generated code in an isolated environment. Treat maintainability, security, and performance as separate evaluation concerns unless the executable contract tests them explicitly.