Assertions Discussion

[rachaelbradley]

We will be discussing Assertions for the next few weeks in the working group meetings.

Reference: Assertions Slide Deck

[rachaelbradley]

How WCAG 3 Define Procedures?

From WCAG 3: A procedure may be limited to guidance:

• approved by AGWG and listed in WCAG,
• that references publicly published guidance, or
• that meets criteria specified in WCAG
• or it may be any process that an organization uses to improve accessibility.

How should WCAG 3 define the procedures that Assertions reference?

[mbgower]

assertions are not repeatable or verifiable

On 7/23 call, I stated that I believe that while that may be true, the outcome may be able to be assessed. For example, if someone has an assertion for a process their team uses to arrive at clear wording, and their resulting headings are wordy and poor, one cannot verify the team has done their process, but can point to the outcome to show that either they did not carry it out, or their process sucks.

[bruce-usab]

On call 7/23, I shared my opinion that the current definition of assertions could include activities frequently included in U.S. legal settlement agreements such as consultation with advocacy organizations. From that perspective, some assertions have some ability to be verified.

[mbgower]

it may be valid to state that the assertion cannot be verified from the page's outcome but that the team may be able to show evidence that their process was carried out. I think this is supported on slide 7 of the assertion deck.

WCAG recommends maintaining additional information that an organization can use to improve or validate procedures and assertions.

[GreggVan]

Good finesse. It doesnt say anyone would have access to them -- or that they would have to show them-- but that it would be good to keep such records "to improve or validate procedures and assertions."

[shawnthompson]

Wasn't there talk early on about maybe adding links to documentation in metadata about assertions? I vaguely remember the idea being brought up and thought it was an interesting idea.

[GreggVan]

Key to the concept of assertions is that the ONLY thing to test is that an assertion was made. There is no requirement for a testable outcome (or we should be using the outcome as the measure - not the assertion).

Also note that we can give points for an assertion for higher levels -- but unless we can confirm that companies can make assertions (in the 2000-2008 period we were told that many large companies were not allowed to by legal departments) we would not be able to include assertions as requirements for conformance at the basic (or any regulated) level.

A key part of our architecture then has to be structuring the guidelines so that testable / requireable provisions are right along with recomendations for doing non-testable but just as important things.

[bruce-usab]

Maybe an example which might be point for call today:

Assertion: We have taken steps to ensure we are using plain language. We test our assumptions as described by Federal plain language guidelines
.

[rachaelbradley]

Summary of Questions from Assertions Template Exercise

1. What level should assertions be placed?

• For example, assertions may reference guidance such as a style guide or testing such as AT testing or usability testing that covers multiple guidelines/outcomes.
• Should the assertion be mentioned both at the outcome and a higher level
• Should we frame assertions so they refer to a section of a larger document (example: Guidance that addresses alt text instead of a style guide that addresses alt text)

2. How can a person or organisation with a small website attain the highest level of accessibility?

• E.g. simple website done by an expert may not need internal guidance in the same way that teams do.
• Related question: If a small website/organization meets all the criteria at each level, is that comparable to the assertion?

3. How are assertions tested?
   -Validating an assertion only tests that an assertion was made and correctly stated/documented. Sometimes testers will see sites that are so bad it is obvious the assertion is false. How is that handled?
   -If the documentation that supports an assertion is not public, how are assertions enforced or validated?

4. Can WCAG 3 require an accessibility statement?

• If yes:
  -- What should in included in order to make it useful?
  -- Should they only be required if the organization is using assertions?
  -If no:
  --There seems to be some consensus that an assertion must be made in a publicly available location. Would that location then be in an accessibility conformance report?

[rachaelbradley]

Summary of Discussion from 27 August 2024 meeting

From a documentation perspective, the group was generally in favor of integrating assertions at multiple levels, alongside requirements. Exactly how they are handled will be dependent on conformance decisions but the process we will continue to follow for the near future is building them into decision trees and then look later to see if they can be grouped. We then discussed how assertions might fit into conformance.

Please reply to the new threads below for the follow up questions:

• Should assertions be available to meet a conformance level? Opinions varied widely during the discussion. Some members felt they should be visible but never used to meet conformance at any level, some members felt that they the option to use assertions to meet any conformance level should be left open, and some felt that assertions should not be used at the most basic level of conformance but should be available as one way to meet higher levels of conformance.
• Do assertions need to be made in a place that is viewable by people who have access to the system or only in a conformance statement that is provided by the author? The group generally agreed that assertions should be documented as part of conformance statements but supporting documentation should not be required required to make a conformance statement.

[detlevhfischer]

1. My view is that assertions should be kept clearly separate from the result of any outcome testing and not enter any score. So they would not be on the same level but in another compartment.
2. Since the level of outcome would be unrelated to any assertions made, there is no problem for smaller organisations that don't have or need a formal process.
3. An assertion template may at least link to a doc (policy, process description or styleguide) that substantiates the assertion. If no such link exists, readers can draw their own conclusions. No test of content can reliably substantiate the assertion (because the fact that a site's content indeed demonstrates what is being asserted may be due to factors other than the documented process).
4. My feeling is an accessiblity statement is on the level of policy, not standard.

[GreggVan]

Regrets on the call tomorrow -- conflict came up
here are my thoughts quickly

What level should assertions be placed?
For example, assertions may reference guidance such as a style guide or testing such as AT testing or usability testing that covers multiple guidelines/outcomes.

• should not be required at any level we want picked up in regs
• but YES definitely want them viewable and seen right along with our most basic requirements !

Should the assertion be mentioned both at the outcome and a higher level

• yes -- mentioned at basic outcomes level -- as ways to get points for doing better to get higher level rating

Should we frame assertions so they refer to a section of a larger document (example: Guidance that addresses alt text instead of a style guide that addresses alt text)

• not sure I understand but I think I know. I think it should be specific. If in larger doc - refer to section.

How can a person or organisation with a small website attain the highest level of accessibility?
E.g. simple website done by an expert may not need internal guidance in the same way that teams do.

• in some ways it could be much easier to. in some ways harder
• we should include a mix of both so it is level playing field

Related question: If a small website/organization meets all the criteria at each level, is that comparable to the assertion?

• do not know what that means

How are assertions tested?
-Validating an assertion only tests that an assertion was made and correctly stated/documented. Sometimes testers will see sites that are so bad it is obvious the assertion is false. How is that handled?

• well - if the assertion is that you did training -- a bad site does not mean that you didn't. You may just have bad designers or designers that don't care.
• an assertion is just that. If you make it -- you pass - no matter what the site looks like. (Otherwise it is an outcome you are looking for and not an assertion

-If the documentation that supports an assertion is not public, how are assertions enforced or validated?
-require that it is public to be an assertion
-you can't enforce an assertion. it is or it isnt.

Can WCAG 3 require an accessibility statement?

• in WCAG 2.0 we were told that some companies would not do that. which meant they could not ever conform. which meant - why try.
• instead we said what was required if you did.

If yes:
-- What should [we] included in order to make it useful?
-- Should they only be required if the organization is using assertions?

• I think no - that might discourage assertions.

-If no:
--There seems to be some consensus that an assertion must be made in a publicly available location. Would that location then be in an accessibility conformance report?

• it could but many companies have confidential accessibility conformance reports and only show them when required as part of an RFP to avoid critique and criticism.

[detlevhfischer]

but YES definitely want them viewable and seen right along with our most basic requirements!

My assumption was that what organisations will want or be able to assert will depend on their own processes (human remedying, automatic checks, AI, whatever) and will come in various kinds of granularity (from addressing one outome, to several, to pledging continued conformance the the whole lot) so I fail to see the point of any kind of prefab assertion baked into the basic requirements.

[rachaelbradley]

Should assertions be available to meet a conformance level?

Opinions varied widely during the discussion. Some members felt they should be visible but never used to meet conformance at any level, some members felt that they the option to use assertions to meet any conformance level should be left open, and some felt that assertions should not be used at the most basic level of conformance but should be available as one way to meet higher levels of conformance.

Please reply to this thread with your position and reasoning or react to responses with thumbs up or thumbs down.

[gundulaniemann]

Assertions should not be part of the decision tree, as a strict separation between testable and non-testable outcomes is needed.
The example assertions we have seen so far ask for guidance/guidelines. These are usually internal to a company, thus they cannot be retrieved by 3rd party testers. As a consequence, they are not suitable for any conformance claim.

[GreggVan]

Yes but not the base level -- or the level that we want to have required in regulations UNLESS we determine that all companies can make assertions. Assertions are different than just doing something. It is like obeying a law (stopping at stop signs) and legally asserting that you stop at stop signs. Everyone can do #1 but a lawyer would not let you assert that you do.

we don't want the basic level of conformance include something that a company can't/won't do - or else they can never even try to conform at that level.

[alastc]

Chair hat off - I'm concerned that we might run into equity problems if we cannot use an assertion at the baseline level.

For example, if there is a language based requirement for people with cognitive impairments that can only be addressed by following a process. It could be as important (to end-users) as things which are more objectively tested.

Why shouldn't that be at the baseline level?

Chair hat on - I also appreciate the concern that we don't want to include things at a baseline level that can't be used by some organisations. I think we need to get an answer to that concern sooner than later.

[GreggVan]

Agree with this

But there is a corollary. If some orgs can't do it -- it can't be in a regulation. Or won't be adopted in regulation. And we don't want out base level to not be adaptable......

So again - a key question we need to determine -- "Are assertions something that can be required? That all companies could reasonably do? So far I know know if we have any reliable conclusion. Just old information (my information is old - but seems like it MIGHT (and might NOT) still hold and wishes and hopes and fears. Would be good to get some input from companies and lawyers at regulators? Tough question. We would need a bunch of legal opinions that agree? unghh

[ChrisLoiselle]

Hi @rachaelbradley using https://www.w3.org/WAI/planning/statements/generator/#create as an example, the "statement" provides what standards were applied, conformance status, "your efforts" which then has a subset of organizational measures, which could be interpreted as "assertions" by a company or org of "what they've done". I understand @nitedog and others were on the project team on the WAI statement generator, so perhaps pulling those individuals in on how we can reformat or utilize what WAI has already done to insert assertions in to that project would be beneficial so we don't lose all that previous work that seems to resemble these efforts. In the use case of using https://www.w3.org/WAI/planning/statements/generator/#create the statement wouldn't be used to meet a conformance level but provide a statement on what you've done to get to the level you are stating you've achieved per your assertions and efforts , methods, outcomes of testing, applicable outcomes analyzed, etc.

[rachaelbradley]

Do assertions need to be made in a place that is viewable by people who have access to the system or only in a conformance statement that is provided by the author?

The group generally agreed that assertions should be documented as part of conformance statements but supporting documentation should not be required required to make a conformance statement.

Please reply to this thread with your position and reasoning or react to responses with thumbs up or thumbs down.

[detlevhfischer]

It is good to document assertions on top of conformance statements. The more visible the better. It seems a matter of policy not standardisation though. For example, public bodies in Europe are required to have accessibility statements.
I can even see a situation where some provider's conformance statement documents partial non-conformance ("due to the dynamic nature of content, not all images have appropriate alt text") BUT asserts that there is a process to add alt text after some time interval, or populate images with AI-generated alt text which may or may not be correct, but purportedly "better than nothing". This could be done by linking from the conformance statement to the relevant assertion. They'd still be separate, but connected.

[gundulaniemann]

Assertions should be in a central place, like the Accessibility statement.

[GreggVan]

Not sure what your first part means. You mean that they would have to put the assertion on every page that a person might visit so that they would be viewable by people who access the site on any page? Or do you mean that there is an accessibility statement someplace on the website (example.com/accessibility) where they can view it? (versus only having it available as a legal document if you request it?)

Can you provide a little bit more information about what the two choices are? Or if you have three or more?

[alastc]

(Chair hat off) I'd like to re-phrase the question a bit in order to answer: Should the documentation behind an assertion be publicly available, available to trusted 3rd parties, or always private?

I think from a WCAG point of view, we should only ask for the assertion, yes/no. That someone is making an assertion should be part of a conformance / accessibility statement.

However, all the documentation behind that should not be required by us.

We can provide informative information on how to assess that you are meeting an assertion, and what documentation you should gather as evidence.

The use of that documentation is then up to organizations and their regulators. It could be required to be public from the regulator, or it might be required if there is some legal action. However, I think it would undermine the use of assertions to require the documentation be public, as it is likely to include non-public information such as internal policies and processes.

[ChrisLoiselle]

@alastc

_Re: We can provide informative information on how to assess that you are meeting an assertion, and what documentation you should gather as evidence.

The use of that documentation is then up to organizations and their regulators. It could be required to be public from the regulator, or it might be required if there is some legal action. However, I think it would undermine the use of assertions to require the documentation be public, as it is likely to include non-public information such as internal policies and processes._

1. I believe what we'd need to determine, if we haven't already, is whether or not assertions are in fact testable. @GreggVan has mentioned before, I believe, that they aren't. @GreggVan correct me if I mis-read that in another discussion/issue.
2. If assertions are not , then how would one assess they are meeting an assertion?
3. Is that the company self validating and verifying internally that they are about to make an assertion publicly and then :
4. Making sure they've gathered and have internally (but won't ever share that documentation publicly), the information needed to make that assertion?
5. Or is what you are stating that the we, AG, would provide information on how to assess the assertion itself that the company made, from a customer or potential customer perspective?
   I've heard both sides of assertions on W3C calls, as to whether or not these would be testable, and I feel group consensus would be beneficial to understand how we'd proceed with these moving forward.