AI & information

Test what an AI-written regular expression should reject

Define a filename rule, expected matches and near misses before using a generated pattern on real files.

An AI gives you a pattern for finding PDF filenames. It accepts your first example, so it looks ready. But does it also accept a backup file ending in .pdf.bak, or a name with an extra character where the dot should be? A positive example alone cannot answer those questions.

Write the requirement before the pattern

For this exercise, accept exactly the prefix invoice_, followed by three digits from 0 to 9, followed by .pdf, with the shown lowercase spelling. Match the whole filename, without a folder path. These are example requirements, not rules for all PDF files.

Tell the assistant which tool will run the expression. In Python, re.search looks for a match somewhere in a string, whereas re.fullmatch requires the whole string to match. The dot also has a special meaning unless made literal. Syntax and options can differ in other tools.

Decide these expected results before testing:

  • invoice_007.pdf: accept.
  • invoice_07.pdf: reject; only two digits.
  • xinvoice_007.pdf: reject; an extra prefix.
  • invoice_007Xpdf: reject; no literal dot.
  • invoice_007.pdf.bak: reject; extra trailing text.
  • Invoice_007.PDF: reject under this case-sensitive requirement.

Keep matching separate from changing files

Run the pattern against sample strings and compare every result with the list. Ask the assistant to explain any mismatch and rerun all cases after a change, not just the one it repaired. NIST describes how generated answers can confidently depart from the request; a convincing explanation is not the test result.

Only after the selection is right should you consider renaming or moving files. First produce a list of proposed changes, inspect it and keep a recoverable copy. Correct matching does not check whether two renamed files would collide or whether the selected files are the ones you meant to change.

Sources & further reading

  1. NIST AI 600-1 — Generative Artificial Intelligence Profile, section 2.2
  2. Python documentation — re: Regular expression operations

Sources support the technical explanations. Examples and suggested checks are editorial guidance.