W3C ACT Implementation

Overview - Accessibility Conformance Testing (ACT)

The W3C established a community of developers of accessibility evaluation methodologies and testing tools and some accessibility Subject Matter Experts (SMEs). The community establishes and documents ACT Rules for testing the conformance of WCAG and ARIA. These test rules address automated, semi-automated, and manual testing, but since the Checker is primarily an automated test tool, that is the primary interest. ACT’s goal is to make accessibility testing more transparent, and thus reduce confusion caused by different interpretations of the standards (WCAG Success Criteria). ACT members write & share with the community and implement ACT Rules into their products and services. They also test their tool against the ACT Rules and measure consistency and accuracy as reported in their ACT Implementation reports. ACT is not intended for end-users of accessibility tools, web developers, content authors, QA testers, etc. For more overview and background information, see Accessibility Conformance Testing (ACT) Overview.

ACT Rules

1. All ACT rules are listed by the standard or specification that they cover, including WCAG, ARIA, Proposed Rules, and beyond.

2. All ACT rules follow a review and publishing process.

3. Each ACT rule has a 6 character Rule ID, description, applicability, expectations, assumptions, the WCAG criteria it maps to (reports against), and example test cases.

4. Each rule’s Test Cases include Passing examples, Failing examples, and Inapplicable examples

   • See an example - Rule ID: 2779a5 - HTML page has non-empty title [w3.org/WAI/standards-guidelines/act/rules/2779a5/].

   • The ACT Rule ID is found in the URLs, such as:

     • Rule 97a4e1 description URL: e.g. w3.org/WAI/standards-guidelines/act/rules/97a4e1/

     • Rule 97a4e1 test case URL: e.g. w3.org/WAI/content-assets/wcag-act-rules/testcases/97a4e1/….html

Checker mapping to ACT

The ACT community describes how implementations, such as the Checker, should approach the Mapping to Rules. Individual Checker rules may test and report against (map to) one or more ACT rules by identifying the ACT rule's ID (6 character) in its checker_rule_name_.ts file using the act: [ _act_6_character_rule_ID_here_ ], declaration:

• See example img_alt_valid.ts, this Checker rule maps to act: "23a2a8" at around line 53 in the rule’s logic code.

• See example aria_id_unique.ts where a single Checker rule maps to two ACT rules: act: ["59796f", "6a7281"],

• See example aria_graphic_labelled.ts where a single Checker rule maps to multiple ACT test cases depending on the Fail reasons: act: [{ "7d6734": {"Pass_0": "pass", "Fail_1": "inapplicable", "Fail_2": "fail", "Fail_3": "inapplicable" } }],. These more complex mappings may be due to our rules having different scope than the related ACT rule, or our rule split across multiple ACT rules.

• See example aria_img_labelled.ts where a single Checker rule maps to multiple ACT Rules and multiple test cases:

act: ["23a2a8", { "7d6734": {
                  "Pass_0": "pass",
                  "Fail_1": "inapplicable",
                  "Fail_2": "fail",
                  "Fail_3": "inapplicable"
                 }
      }],

• The Checker reports the individual ACT test case results with one of the following values in the outcome:
  • earl:passed: A node in the test case passed the rule
  • earl:failed: A node in the test case failed the rule
  • earl:inapplicable: The rule is inapplicable in the test case
  • earl:cantTell: The tool could not be sure whether or not a node passed of failed the rule
  • earl:untested: The rule was not executed on the test case

ACT Results

1. See .../accessibility-checker/test-act-w3/main.js in the Checker repo that uses Puppeteer to run the ACT test cases and report the results.

   • The data sent to W3C ACT is ...@next/act-report-v2.json.
   • The isPartOf is the mapping (reporting) to WCAG that the Checker reports in the results and ACT will use to match the results.
   • How the Success Criteria #'s (scID's) are defined is in the ...accessibility-checker-engine/src/v4/sc-urls.json. Some examples:
     • "isPartOf": ["WCAG2:identify-input-purpose" is defined by "scId": "WCAG2:non-text-content" that maps to "num": "1.1.1".
     • "isPartOf": [ [] ] is defined by ARIA and HTML as "scId": [], so those map to None by ACT.
   • The scID's, such as "WCAG2:non-text-content" were originally defined by ACT at github.com/act-rules/act-tools/blob/main/src/data/sc-urls.json.

2. See the ACT Test Results (act-report-v2.txt) Artifact created from Actions by each PR build that details the results of running the Checker against the ACT rule Test Cases. The results file is available in Actions for debugging and QA purposes.

   1. Each ACT rule is listed, each test case is listed, and if there is a mismatch, then there is a message, such as “Expected passed, but returned earl:failed”

   2. If no Checker rule maps to an ACT rule, then a message: “? No checker rules“ is listed under that ACT rule.

   3. See sample ACT Test Results (act-report-v2.txt) file saved to Box. But, it is best to view the Artifact file from Actions for the most current results.

ACT Implementation report

• After each deployment, the Checker sends the test results to ACT. The data is published using the EARL+JSON-LD data format.

• The “official results” are publicly available a day later from the ACT Implementation Reports page when ACT formats it into the common-looking report. See the Equal Access Accessibility Checker linked from that page. Results from other test tools are also publicly available. Note the date in the report. Ignore the Checker version in the report until ACT fixes the issue.

Background for contributors

The following statements help clear up the purpose and high level understanding of these related topics and specs:

• The ACT Community uses the Git repo at act-rules.github.io/rules. The JSON file that indexes all the rules is at act-rules.github.io/testcases.json.

• The Evaluation and Report Language (EARL) is a machine-readable format for expressing test results (not the rules) that tool developers use to send their results to ACT.

• EARL is an RDF vocabulary (not rules & not results), the terms of which are defined by Evaluation And Report Language (EARL) 1.0 Schema.

• Some of the terms defined by EARL are being used by the Accessibility Conformance Testing (ACT) Rules Format (v 1.0 Oct 31 2019).

• The ACT Rules Format defines a format for writing accessibility test rules (not the results). This format is intended to enable a consistent interpretation of how the test was / is to be done ... and promote consistent results between complementary and competing 3rd party engines in reporting accessibility testing. Example tools include Axe-core, SortSite, Alpha (SiteImprove), etc. As of 2022 it is not clear yet if/when WebAIM's WAVE engine plans to publish an ACT implementation report.  

• Accessibility Conformance Testing (ACT) Overview - introduces the ACT Rules Format, lists the ACT Rules formally published by W3C, Who ACT is for, and Who develops ACT rules.

• Each ACT rule has the following sections: these links go to an example rule: ARIA required context role): Applicability, Expectation, Assumptions, Accessibility Support, Background, Test Cases, Glossary, Implementations, Change log.

• ACT Community Git repo also includes an additional section called Acknowledgements of who created / submitted the rule (see example at act-rules.github.io/rules/ff89c9#acknowledgments

• Both the ACT Community Git repo and the W3C list of ACT Rules contain an Implementation section for each rule showing which tool supports that particular ACT rule. For example, see ACT rule 97a4e1 Button has non-empty accessible name.

• The Checker does not list the ACT rules it implements in its own documentation. See the Equal Access Accessibility Checker implementation report on the ACT website.