Evaluating AI features before production

Generative features can change their answers with small changes in inputs, context or model versions. A handful of successful demonstrations does not show how the system behaves across real users and awkward edge cases. Product teams need a repeatable way to compare behaviour before launch and after changes. Write down the user promise Describe what the feature does, what it does not do and which decisions remain with the user. Choose measures that reflect that promise: task completion, correction effort, source accuracy, safe refusal or escalation. Avoid a single vague score such as intelligence. Different risks require different acceptance thresholds. Build a representative test set Collect realistic examples from intended workflows, with privacy safeguards and appropriate review. Include common tasks, rare but important cases, ambiguous questions, missing information, adversarial prompts and inputs from different devices or language styles. Keep examples separate from the tuning process when possible so the evaluation can reveal overfitting. Score more than the final answer Inspect whether the system used the correct context, followed permission boundaries, communicated uncertainty and selected the right next action. For tool-using features, check each tool call and the consequences of partial failure. Human reviewers need a rubric and calibration so two reviewers interpret quality consistently. Make evaluation part of release Run the same set against the current baseline and each proposed change. Review regressions by task type and user group, not just the overall average. Use a limited rollout with monitoring and a rollback plan when impact is significant. Production feedback can reveal new cases, but it does not replace pre-release checks. Keep the evaluation set versioned, governed and refreshed as the product and risks change.