The AI answer sounded right. Then someone opened the source.

Imagine a procurement manager asking an internal AI assistant whether a supplier’s insurance is still valid. The answer arrives in seconds: yes, with a document link. She opens it. The policy expired last month.

This is an illustrative scenario, but it captures a familiar problem: an answer can look carefully researched while missing the fact that matters. The document exists. The wording sounds reasonable. The conclusion is still wrong.

NIST’s Generative AI Profile identifies confidently incorrect output as a distinct risk and recommends verifying sources and citations during testing and ongoing monitoring. The practical question for a business is whether the evidence actually supports the answer being used.

For this supplier check, a useful response would show the insurer, policy number, expiry date and page reference. It would distinguish “a certificate was found” from “the certificate is current”. A missing renewal should produce a request for the renewal, rather than a reassuring guess.

That changes the reviewer’s job. Instead of rereading a whole file to discover what the assistant meant, the manager checks a few explicit claims against the relevant passages.

Dates deserve special attention. A folder can contain a signed agreement, an unsigned revision and an old email discussing a proposed change. The most recently uploaded file is not necessarily the document in force.

Our suggested pilot starts with cases the team already finds difficult: a blurred scan, conflicting dates, a missing page and two suppliers with similar names. Have a knowledgeable colleague record the expected result before the assistant sees the test set.

Record different failures separately. Did the system miss the relevant document? Misread it? Use an outdated version? Draw a conclusion the text did not support? A single average accuracy number can conceal all four.

Keep a separate set of examples for the final check. Repeatedly tuning the assistant on every test case can make the demonstration look better without showing how it handles unfamiliar work.

There is another trap. A document might contain text telling the assistant to ignore its rules or reveal information. OWASP describes this as indirect prompt injection when instructions arrive through external content. Finding a document should never give its author permission to control the assistant.

Test that boundary, and test access permissions too. Ask the same question as employees with different roles. A correct answer that exposes a restricted contract is still a failed result.