Check the created Jira stories for hallucinations

Hallucination monitoring · innovation-group · po-c81217ee

completedcost $0.3388203

[po-eval-monitor running] Pipeline started — https://github.com/N-iX-GenAI-Value-LAB/observability-server/actions/runs/34426976535
[po-eval-monitor running] Phase: scan — reading backlog from run po-d1fcf6ec and fetching Jira baseline KAN-140
[po-eval-monitor running] Phase: verify — KAN-140 not found; using run intent as baseline. Tracing all concrete claims in backlog to intent source.
[po-eval-monitor running] Phase: verdict — 2 HIGH, 7 MEDIUM, 9 LOW flags identified; verdict: blocked-advisory

blocked-advisory

Findings

9 low2 high7 medium

highinvented-number

a GET request to that URL within 3600 seconds returns HTTP 200Signed URL TTL of 3600 s (1 hour) is not stated in the intent. Drives security policy and email UX; wrong value would be a product defect.

highinvented-number

maximum retry count (3 attempts)Retry ceiling of 3 not stated in the intent. Determines when the schedule owner is notified of failure; wrong value affects incident response.

mediumfabricated-fact

role != account_managerThe exact role identifier account_manager is not in the intent (which says account managers). If the system role key differs, the auth guard would break.

mediumunsupported-claim

columns [invoice_id, client_name, date, amount, status, currency]CSV column schema not specified in intent. Presented as a testable fact; a different actual schema would cause all CSV ACs to fail.

mediumunsupported-claim

server returns HTTP 413HTTP 413 as the over-limit error code not stated in intent. Choice of status code is a contract decision that affects client error handling.

mediumunsupported-claim

recipient email pre-filled with my account emailEmail pre-fill behavior not stated in intent; presented as a concrete testable condition that could be wrong.

mediumunsupported-claim

API returns HTTP 409 ... A schedule with identical settings already existsDuplicate-schedule detection and HTTP 409 response not in intent. Introduces a business rule not authorized by the PO.

mediuminvented-number

retry is scheduled within 300 seconds300 s retry window not in intent. Sets an SLA for failure recovery that is not authorized.

mediuminvented-number

export job is triggered within 60 seconds of next_run_at60-second trigger latency SLA not in intent. This is a platform/infra commitment not stated by the PO.

lowunsupported-claim

column C is Date (Excel date type format YYYY-MM-DD), column D is Amount (numeric 2 decimal places)XLSX cell-type specs not in intent; reasonable domain assumption but unlabeled.

lowinvented-number

button shows a checkmark icon for 2000 ms2-second success feedback duration not in intent.

lowinvented-number

server takes longer than 10,000 ms to respondClient-side 10 s timeout not in intent; intent specifies a server-side 5 s performance requirement.

lowinvented-number

button shows a loading spinner for no more than 3000 ms3-second spinner cap on offline state not in intent.

lowunsupported-claim

server returns HTTP 410 GoneHTTP 410 for expired links not stated in intent; reasonable convention but unlabeled assumption.

lowinvented-number

Export XLSX button returns to its enabled state within 500 ms500 ms UI recovery timing not in intent.

lowinvented-number

5 concurrent export requests each for 10,000 rowsTest sizing (5 concurrent, 10k rows each) not in intent; invented benchmark parameters.

lowunsupported-claim

next next_run_at is set to last_run_at + 86400 secondsDaily cadence arithmetic (86400 s) not in intent; DST and timezone handling assumptions are embedded here without label.

lowunsupported-claim

file named invoices_<YYYY-MM-DD>.csvFilename pattern not in intent; reasonable convention but presented as testable fact.