Reporting

Key	Value
Status	Active
Owner	QA Automation
Updated	2026-03-26
Scope	Slack alerts, run summaries, periodic reports, and recovery communication

Reporting is where the system becomes readable. A good report answers three questions quickly:

what happened
how worried should we be
what should happen next

That sounds simple, but it took a lot of iteration to get there.

Reporting Outputs

Output	Audience	Typical Use
failure alerts	QA, on-call, engineers	immediate action
success summaries	operators and channel followers	reassurance and trend awareness
visual notifications	teams watching the visual suite	structural UI signal
weekly reports	managers and maintainers	broad operational summary
monthly reports	stakeholders	trend and stability review
recovery replies	anyone following a failure thread	closure after a fix

Slack Failure Alerts

Failure alerts are now designed to be more human and less robotic.

Main Principles

Principle	What It Looks Like
calm headline	headline says what matters, not just “FAILED”
short first post	stats and failed tests, not a forensic dump
verdict-first thread	`Assessment`, `Why`, `Next`
clustering	repeated failures with one cause are grouped
confidence language	`Post-fix, watching`, `Likely flaky`, `Infra/CI suspicion`, and similar labels

Why This Matters

Operators need to know whether to escalate, rerun, ignore, or watch. Generic classifier text does not help much when a run is noisy.

Success Summaries

Success messages are intentionally smaller. Their job is to confirm the run landed cleanly and leave useful links behind without taking over the channel.

Weekly And Monthly Reports

Periodic reports are the long-view counterpart to failure alerts.

Report Type	What It Tries To Show
weekly	what changed this week, where the risk is, and whether the trend is improving
monthly	slower moving stability and quality patterns

Recent direction:

plain-language executive summary
root-cause rollups, not only symptom counts
week-over-week context instead of isolated numbers
better separation between recurring noise and material regressions

What A Weekly Report Contains

Section	What It Shows
overall pass rate	percentage across all suites for the past 7 days
failure breakdown	per-site and per-suite failure counts
top failing tests	tests that failed most often during the week
top flaky tests	tests with mixed pass/fail outcomes
trend direction	whether pass rate improved, held steady, or declined versus the prior week
root-cause rollup	failures grouped by incident or category rather than listed one by one

What A Monthly Report Contains

Section	What It Shows
30-day pass rate	overall quality trend over the month
stability patterns	which sites and suites are most stable or most volatile
recurring failures	incidents that appeared repeatedly over the month
recovery rate	how quickly failures were resolved after they first appeared

Schedule

Report	When	Channel
weekly	Monday morning (GitLab schedule)	`#qa-reports`
monthly	1st of each month (GitLab schedule)	`#qa-reports`

Triggering Reports Manually

Command	What It Does
`npm run report:weekly`	generates and posts a 7-day report
`npm run report:monthly`	generates and posts a 30-day report
`npm run data:weekly`	pulls weekly summary data without posting

PDF output is saved to test-results/reports/.

Reporting Data Flow

%%{init: {'theme':'base', 'themeVariables': {'primaryColor': '#4a90d9', 'primaryTextColor': '#fff', 'primaryBorderColor': '#2c6fad', 'lineColor': '#555', 'fontFamily': 'sans-serif'}}}%%
flowchart LR
    OS["OpenSearch\ncncqa_tests-*"] --> GEN["Report generator\nscripts/ci/generate-weekly-report.ts"]
    HIST["test-results/history/\nlocal run records"] --> GEN
    GEN --> PDF["PDF report\ntest-results/reports/"]
    GEN --> SLACK["Slack post\n#qa-reports"]

The generator pulls run records from OpenSearch when credentials are available, or falls back to local history files. It produces both a PDF artifact and a Slack post in the same run.

Slack Channel Setup

Variable	Purpose
`SLACK_REPORTS_CHANNEL`	target channel for weekly and monthly reports (default: `#qa-reports`)
`SLACK_BOT_TOKEN`	enables PDF attachment upload and thread replies
`SLACK_WEBHOOK_URL`	fallback delivery path if bot token is not set (message only, no PDF)

Recovery Replies

Recovery replies are one of the most useful newer reporting features because they stop failure threads from becoming dead ends.

Recovery Reply Does	Why It Helps
confirms a failure has stopped repeating	reduces uncertainty
links back to the original thread	keeps context together
records the recovery in history	improves future confidence

Visual Notifications

The visual suite needs separate reporting because its failure modes differ from text-first suites.

Important reporting rule:

a missing baseline is a configuration problem
a screenshot diff is a visual regression candidate
those should not be reported as the same thing

Report Consumers

Consumer	What They Usually Need
QA operator	what failed and what to do next
engineer	enough evidence to reproduce or fix
team lead	whether the problem is broad or isolated
stakeholder	trend and operational stability, not every low-level detail

Good Reporting Style

lead with the verdict
prefer one clear action over five vague suggestions
separate symptom from cause when confidence is low
group shared incidents instead of repeating near-identical paragraphs
avoid flooding the first message with low-level selectors or stack details

run completes
notifier reads merged results
history and incident services add context
Slack gets a short main alert
investigation thread carries the deeper reasoning
if the failure stops repeating, the thread can receive a recovery reply later

Need	Go To
notifier commands	CLI Reference
root-cause and symptom labels	Failure Categories
Slack and service wiring	Integrations