Automated WCAG tests: what they can and can't do

Automated WCAG tests are the fastest way to find barriers – and the most over-sold. “WCAG test tool, one click, done” is a promise the technology can’t keep. This article explains what automated tests reliably find, what they fundamentally can’t do, why a green result is not proof of conformance, and how to use them honestly. For the bigger picture of what WCAG is, see our WCAG guide.

What automated WCAG tests are

An automated test inspects a page’s code programmatically and checks it against WCAG rules. Violations are flagged – in seconds, and across many pages at once. That makes these tools strong at catching errors early, right after a change. Well-known examples are axe, WAVE, Lighthouse and Pa11y.

What they reliably find

Automation works well wherever there are clear, programmable rules: missing alt attributes, contrast values that can be calculated, missing form labels, a missing language declaration, certain structural errors in the heading hierarchy. These are real, valuable findings – and they hit the most common error types.

What they can’t do – and why

This is the honest core. Measured against the WCAG success criteria, an automated tool can only meaningfully judge about a third – roughly 16 of the 50 criteria at level AA. The rest requires human judgement.

There’s a nuance here that honesty demands: if you instead measure the number of individual issues found, the rate is higher – Deque cites around 57 % for axe-core – because common error types occur in large numbers. But that is something different from criteria coverage, and it is certainly not proof of conformance.

The reason: a tool can’t judge whether an alt text actually makes sense (only that one exists at all), whether an image is decorative or informative, whether the focus order is logical, whether a text is written clearly, or whether the purpose of a link is clear from its context. An automated test sees that an alt attribute is present and lets it pass – even if it says “image123.jpg” or describes a steak as bread.

Why “green” doesn’t mean “conformant”

A passed automated test is necessary, but not sufficient. The WebAIM Million study found automatically detectable errors on over 96 % of home pages in 2024 – so automation finds a lot, but “no automatically detected errors” still leaves open the roughly two-thirds of criteria it can’t check at all.

On top of that, conformance per level is all-or-nothing: a single unmet AA criterion means the page is not AA-conformant. A high score is therefore not a conformance certificate. And selling automation as full compliance is exactly the mechanism behind the overlay con – covered in detail in accessibility overlays.

The tools at a glance

• axe (Deque) – the engine behind many tools, available as a browser extension and for automated pipelines.
• WAVE (WebAIM) – shows findings visually, directly on the page.
• Lighthouse – built into the Chrome developer tools, with an accessibility audit.
• Pa11y – for automated workflows and ongoing monitoring.

All are useful for a quick first assessment – none is a substitute for manual review.

Testing honestly with automation

The right role for automation is clearly defined: the quick first assessment plus ongoing monitoring that flags new errors early when something changes – combined with manual review for the large remainder. What that interplay looks like is described in how to test accessibility.

An honest tool is open about what it can and can’t check. It surfaces the real problems in the source code so you can fix them – instead of hiding them so the score looks green. That’s exactly what Welcoma’s upcoming WCAG scanner is built for. The dishonest pattern is the opposite: paper over the problems and show a green seal.