Comparing Alfa with ACT-Rules and WAI test cases

Alfa is an automated accessibility checker developed by Siteimprove.

This repository runs Alfa against the official test cases from ACT Rules and WCAG 2 test rules.

Neither Alfa nor Siteimprove are endorsed by the W3C, the Web Accessibility Initiative or the ACT rules Community group.

See a summary of the implementations reports:

• for Alfa fully automated;
• for Alfa semi-automated.

Cloning

This repository uses github submodules. Clone with

$ git clone --recursive git@github.com:Siteimprove/alfa-act-r.git

If you've already made a shallow clone, initialise submodules with

$ git submodule update --init

Remark: We've seen problem with using the Github repository of act-tools as a dependency, using a git submodule smoothes it out.

Installation

$ yarn install
$ yarn build

Installing Alfa packages need authentication to the Github repository for the @siteimprove scope.

Usage

Update test cases

To download the latest version of the test cases, build the project then:

$ yarn fixtures

To download the old test cases, use instead

$ yarn fixtures old

Generate reports

To run Alfa against the test cases:

$ yarn test

This generates a report for both the automated and assisted implementations. Committing the report and pushing it upstream will update the implementation summary on the corresponding website.

To run test for a single implementation (often useful for debugging), use:

$ yarn [testAssisted | testAutomated]

Using yarn [strictAssisted | strictAutomated] instead will fail the tests (and generate errors) for benign mismatches (e.g. "Passed" vs "Inapplicable"). This can be useful for investigating and marking these correctly…

Generate summaries

Once the reports have been generated, a summary of the implementation can be generated with

$ yarn summary

The summaries are shown at:

• for Alfa fully automated;
• For Alfa semi-automated.

These summaries are mostly the same than the implementation report hosted on WAI ACT rules implementation reports but contain a bit more details, especially for missing stuff.

Old test cases

In order to test the old test cases (hosted as a Github page), use

$ yarn ava "**/old*" [-- --strict]

this is mostly deprecated now and only the new test cases should matter.