Edit

Assessing the Web Sustainability Guidelines (WSG) on automated testing

One of the issues with WCAG is how much of it is open to interpretation. Let’s see how the new Web Sustainability Guidelines fare in that regard, and specifically how much potential there is to create fully automated testing tools.

For the Web Content Accessibility Guidelines, there have been long-standing efforts to reduce the need for interpretation. For automated tools in particular, the most prominent effort is the ACT Rules project.

Wouldn’t it be neat if the WSG got this right from the start? That’s where discussions of Manual vs. automated testing come in.

My results

TL;DR; I produced two documents:

My classification

Here is how I personally rated the SCs (and how many SCs I found for each rating):

  1. Static analysis (6): Potential to write automated code checks that would run in CI / developer IDEs. Example: jsx-a11y/alt-text
  2. Automated (35): “Runtime” analysis – potential to inspect the product with automated browsing or equivalent and detect issues. Example: Axe
  3. Manual, quantitative (16): Likely manual auditing but with potential to follow a set scoring algorithm. Possible to create semi-automated tools to help with auditing. Example: Tab stops testing
  4. Manual, qualitative (50): Manual auditing with an element of interpretation. Can be done with publicly available information, reproducibility of findings a possible concern. Example: WCAG SC 3.2.4 Consistent Identification
  5. Consulting (125): Manual auditing requiring interpretation and internal knowledge of the project / organisation. Cannot be audited without behind-the-scenes access.

Comparison with @marvil07-adapt

After converting this classification to the one by @marvil07-adapt, here is how we differ in the comparison table:

  • Same assessment for 69 guidelines
  • Almost the same for 21 (“partial” vs. yes or no)
  • Different for 3 (“yes” vs “no”)

Mapping from my classification to yes/no/partial (internal). I also used “partial” if a guideline was a mixture of “yes”, “no”, or “partial”.

  • Static analysis → yes
  • Automated → yes
  • Manual, quantitative → partial
  • Manual, qualitative → no
  • Consulting → no – internal

Differences

@marvil07-adapt @thibaudcolas Guideline
no – internal yes 3.4 Apply Tree Shaking To Code
no – internal yes 3.17 Manage Dependencies Appropriately
no – internal yes 3.20 Avoid Using Deprecated Or Proprietary Code

Here are my thoughts on the three guidelines where we differ:

  • For 3.4 Apply Tree Shaking To Code – I think this is automate-able because though the guideline is called “tree shaking”, the SC is only about eliminating dead code in a wider sense. We can’t quantify whether the project makes an effort to identify and eliminate dead code, but we can definitely quantify the amount of dead code, which seems good enough to me to consider this automate-able.
  • For 3.17 Manage Dependencies Appropriately – this all seems automate-able enough to me. Which dependencies are included can be checked with different methods, and same for how actively they are used.
  • For 3.20 Avoid Using Deprecated Or Proprietary Code – I considered this automate-able at least for code sent client-side (HTML/CSS/JS/SVG/etc).

Partial matches

Note when I reviewed the potential for automation, my focus was primarily on auditing a website or app with no internal knowledge. So for most SCs that require internal knowledge I rated them as “consulting” / “no - internal”, even if there could be automation. There are a few exceptions, such as 3.22 Use The Latest Stable Language Version.

I didn’t assess our differences here in much detail. At a high level:

  • For 2. User-Experience Design, I seem to think there’s much less room for automation
  • For other areas, it’s mixture of results with no clear pattern
@marvil07-adapt @thibaudcolas Guideline
partial – editorial no 2.7 Avoid Unnecessary Or An Overabundance Of Assets
partial no 2.8 Ensure Navigation And Way-finding Is Well-structured
partial no – internal 2.9 Respect The Visitor’s Attention
partial no 2.11 Avoid Manipulative Patterns
partial no 2.21 Support Non-Graphic Ways To Interact With Content
partial – internal yes 3.3 Use Code-splitting Within Projects
partial yes 3.8 Use HTML Elements Correctly
yes – editorial partial 3.9 Resolve Render Blocking Content
partial – editorial, semantics yes 3.12 Use Metadata Correctly
partial – editorial yes 3.13 Adapt to User Preferences
partial no 3.14 Develop A Mobile-first Layout
partial – internal yes 3.16 Ensure Your Scripts Are Secure
partial – editorial yes 4.2 Optimize Browser Caching
no – editorial partial 4.4 Use Error Pages And Redirects Carefully
no – editorial partial 4.7 Maintain a Relevant Refresh Frequency
no – editorial partial 4.9 Enable Asynchronous Processing And Communication
yes partial 4.10 Use Edge Computing
partial – internal no – internal 5.5 Estimate A Product Or Service’s Environmental Impact
no – internal partial 5.19 Use Justice, Equity, Diversity, Inclusion (JEDI) Practices
no – internal partial 5.22 Promote Responsible Emerging Technology Practices
partial – internal no – internal 5.27 Define Performance And Environmental Budgets

Audit types by success criterion

This is my full review of the SCs, with the audit types I found for each.

Audit type Success Criterion
  2. User-Experience Design
  2.1 Undertake Systemic Impacts Mapping
consulting External Variables
  2.2 Assess And Research Visitor Needs
consulting Identify And Define
consulting Visitor Constraints
consulting Barriers And Access
consulting Barrier Removal
  2.3 Research Non-visitors Needs
consulting Non-Human Impact
  2.4 Consider Sustainability In Early Ideation
consulting Wireframes And Prototypes
  2.5 Account For Stakeholder Issues
consulting Human-Centered Brainstorming
consulting Ecological Brainstorming
  2.6 Create a Frictionless Lightweight Experience By Default
consulting Performance By Default
manual, quantitative Efficient Paths
manual, qualitative Patterns For Efficiency
manual, qualitative Distraction-Free Design
manual, qualitative Eliminate The Non-Essential
manual, quantitative User-Initiated Actionable Content
  2.7 Avoid Unnecessary Or An Overabundance Of Assets
manual, qualitative Decorative Design
  2.8 Ensure Navigation And Way-finding Is Well-structured
manual, qualitative Navigation And Search
manual, qualitative Navigable Sitemaps
manual, qualitative New Content
  2.9 Respect The visitors Attention
manual, qualitative Respecting Attention
consulting Avoid Distraction
manual, qualitative Avoid Attention-keeping
  2.10 Use Recognized Design Patterns
manual, qualitative Design Patterns
  2.11 Avoid Manipulative Patterns
manual, qualitative Dark Patterns
manual, qualitative Using Advertisements
manual, qualitative Page Tracking
  2.12 Document And Share Project Outputs
consulting Deliverables Reusability
consulting Deliverables Documentation
  2.13 Use A Design System To Prioritize Interface Consistency
consulting Design System
  2.14 Write With Purpose, In An Accessible, Easy To Understand Format
manual, qualitative Write Clearly
manual, qualitative Content Formatting
static analysis Search Engine Optimization (SEO)
  2.15 Take a More Sustainable Approach To Image Assets
automated Need For Images
automated Optimize Images
static analysis Lazy Loading
manual, qualitative Sizing And Deactivation
consulting Management And Usage
  2.16 Take a More Sustainable Approach To Media Assets
automated Need For Media
automated Optimize Media
manual, qualitative Labels And Choice
consulting Management And Usage
  2.17 Take a More Sustainable Approach To Animation
manual, qualitative Need For Animation
manual, qualitative Avoid Overburdening
manual, qualitative Control Animation
  2.18 Take a More Sustainable Approach To Typefaces
static analysis Default Typefaces
static analysis Font Optimization
  2.19 Provide Suitable Alternatives To Web Assets
manual, quantitative Open Formats
manual, quantitative Font Subsetting
automated Alternative Text
manual, quantitative Audio Alternatives
manual, quantitative Video Alternatives
  2.20 Provide Accessible, Usable, Minimal Web Forms
manual, qualitative Form Simplicity
manual, qualitative Form Functionality
  2.21 Support Non-Graphic Ways To Interact With Content
manual, qualitative Alternative Interactions
  2.22 Give Useful Notifications To Improve The visitors Journey
manual, qualitative Notification Justification
manual, qualitative Notification Control
manual, qualitative Prompts And Responses
  2.23 Reduce The Impact Of Downloadable Or Physical Documents
manual, quantitative Printing Documents
manual, quantitative Optimize Documents
manual, quantitative Labels And Choice
  2.24 Create A Stakeholder-focused Testing & Prototyping Policy
consulting New Features And Perspectives
consulting Resourcing And Viability
consulting Training And Onboarding
consulting Testing And Validation
  2.25 Conduct Regular Audits, Regression, And Non-regression Tests
consulting Regular Issue Testing
consulting Non-Regression Tests
consulting Regression Tests
  2.26 Analyze The Performance Of The Visitor Journey
consulting Measurement And Compliance
  2.27 Incorporate Value Testing Into Each Major Release-cycle
consulting Usage Changes
  2.28 Incorporate Usability Testing Into Each Minor Release-cycle
consulting Usability Testing
  2.29 Incorporate Compatibility Testing Into Each Release-cycle
consulting Compatibility Policy
consulting Maintaining Compatibility
manual, quantitative Frequent Testing
consulting Mobile Friendly
consulting Progressive Web Application’s (PWA)’s
  3. Web Development
  3.1 Identify Relevant Technical Indicators
consulting Performance Goals
  3.2 Minify Your HTML, CSS, And JavaScript
automated Minify Code
  3.3 Use Code-splitting Within Projects
automated Code Splitting
  3.4 Apply Tree Shaking To Code
automated Remove Redundancy
  3.5 Ensure Your Solutions Are Accessible
automated Accessibility Compliance
static analysis Enhancing For Accessibility
consulting Electronic Inequalities
manual, qualitative Carbon Aware Design
  3.6 Avoid Code Duplication
consulting Remove Or Simplify
consulting Iteration Over Recreation
consulting Organize Code Arrangement
  3.7 Rigorously Assess Third-party Services
consulting Assess Third-parties
consulting Third-party Implementation
consulting Libraries And Frameworks
consulting Self-Hosting
consulting Avoiding Dependency
  3.8 Use HTML Elements Correctly
automated Semantic Code
static analysis Avoid Non-standard Code
manual, qualitative Custom Code
  3.9 Resolve Render Blocking Content
automated Asynchronous Code
manual, qualitative Priority Loading
  3.10 Provide Code-based Way-finding Mechanisms
automated Metadata And Microdata
automated Search Engines
manual, qualitative Accessibility Aids
  3.11 Validate Form Errors And External Input
manual, qualitative Error Validation
manual, qualitative Label Elements
automated Allow Paste
  3.12 Use Metadata Correctly
automated Required Elements
automated Meta Tags
automated Structured Data
  3.13 Use CSS Preference And Media Queries
automated CSS Preference Queries
  3.14 Develop A Mobile-first Layout
manual, qualitative Mobile-First
manual, qualitative Responsive Design
manual, qualitative Alternative Browsing
  3.15 Use Beneficial JavaScript And Its API’s
consulting Beneficial JavaScript
manual, qualitative API Requests
  3.16 Ensure Your Scripts Are Secure
automated Script Security
  3.17 Manage Dependencies Appropriately
automated Dependency Management
automated Dependency Necessity
automated Dependency Updates
  3.18 Include Files That Are Automatically Expected
automated Expected File Formats
  3.19 Use Plaintext Formats When Appropriate
automated Beneficial File Formats
  3.20 Avoid Using Deprecated Or Proprietary Code
automated Deprecated Code
automated Outdated Code
  3.21 Align Technical Requirements With Sustainability Goals
consulting Identify Requirements
consulting Optimized Methodology
consulting Expandability Considerations
consulting Interface Impact
  3.22 Use The Latest Stable Language Version
automated Versioning
  3.23 Take Advantage Of Native Features
manual, qualitative Native Over Custom
  3.24 Run Fewer, Simpler Queries As Possible
manual, qualitative Database Queries
  4. Hosting, Infrastructure And Systems
  4.1 Choose A Sustainable Hosting Provider
consulting Monitor Metrics
consulting Equipment Longevity
consulting Recycling Waste
automated Renewable Electricity
consulting Remaining Emissions
  4.2 Optimize Browser Caching
automated Server-side Caching
automated Offline Access
  4.3 Compress Your Files
automated Server-side Compression
automated Media Compression
  4.4 Use Error Pages And Redirects Carefully
manual, quantitative Error Pages
manual, qualitative Redirection
  4.5 Limit Usage Of Additional Environments
consulting Unused Environments
  4.6 Automate To Fit The Needs
consulting Automate Tasks
consulting Necessitate Tasks
consulting Automated Scaling
consulting Security Tooling
  4.7 Frequency For Refresh Is Relevant To Visitor Needs
manual, quantitative Refresh Frequency
  4.8 Be Mindful Of Duplicate Data.
consulting Data Backups
  4.9 Enable Asynchronous Processing And Communication
consulting Batch Processing
automated Protocol Usage
  4.10 Use Edge Computing
automated Content Delivery Network’s (CDN’s)
automated Sustainability Commitment
manual, qualitative Local Servers
manual, quantitative Avoid Dynamic Resources
  4.11 Use The Lowest Infrastructure Tier Meeting Business Requirements
consulting Lowest Requirements
  4.12 Store Data According To Visitor Needs
consulting Reduce Redundancy
consulting Expiration Dates
consulting Classify And Tag
consulting Justify Storage
consulting Optimize Logging
consulting Compress Storage
  5. Business Strategy And Product Management
  5.1 Have An Ethical And Sustainability Product Strategy
manual, quantitative Statement Availability
manual, qualitative Achievements And Compliance
manual, qualitative Governance Over Time
consulting Onboarding New Members
manual, qualitative Documentation
manual, qualitative Renewable Showcasing
  5.2 Assign A Sustainability Representative
consulting Ecological Referee
  5.3 Raise Awareness And Inform
consulting Inform And Train
consulting Active Participation
  5.4 Communicate The Ecological Impact Of User Choices
manual, qualitative Impact Communication
  5.5 Estimate A Product Or Service’s Environmental Impact
consulting Life-cycle Analysis
consulting Competitor Analysis
  5.6 Define Clear Organizational Sustainability Goals And Metrics
manual, qualitative Sustainability Goals
  5.7 Verify Your Efforts Using Established Third-party Business Certifications
consulting Certification Achievement
consulting Certification Maintenance
  5.8 Implement Sustainability Onboarding Guidelines
consulting Training Materials
consulting Progress Incentivisation
consulting Negative Variables
  5.9 Support Mandatory Disclosures And Reporting
consulting Policies And Practices
manual, qualitative Impact Report
manual, qualitative Standards And Policies
consulting Impact Reduction
  5.10 Create One Or More Impact Business Models
consulting Theory Of Change
  5.11 Follow A Product Management And Maintenance Strategy
consulting Management And Maintenance
consulting Planning Strategy
consulting Resourcing Products
consulting Resource Measurement
  5.12 Implement Continuous Improvement Procedures
consulting Continuous Improvement
consulting Agile Reviews
consulting Iterative Consideration
consulting Functionality Decisions
consulting Security Updates
consulting Skills And Maintenance
  5.13 Document Future Updates And Evolutions
consulting Feature Changes
  5.14 Establish If A Digital Product Or Service Is Necessary
consulting Sustainable Development Goals
consulting Creation Evaluation
consulting Avoid Duplication
consulting Obstacle Consideration
  5.15 Determine The Functional Unit
consulting Life-cycle Assessment
  5.16 Create A Supplier Standards Of Practice
consulting Vetting Potential Partners
consulting Collaborative Measurement
consulting Informative Partner Promotion
  5.17 Share Economic Benefits
consulting Living Wage
consulting Incentivisation
consulting Employee Benefits
consulting Legislation Advocation
  5.18 Share Decision-making Power With Appropriate Stakeholders
consulting Decision-Making
  5.19 Use Justice, Equity, Diversity, Inclusion (JEDI) Practices
consulting JEDI Practices
manual, quantitative Accessibility Policy
consulting JEDI Training
consulting JEDI Improvements
consulting JEDI Legislation
  5.20 Promote Responsible Data Practices
manual, qualitative Privacy Policy
consulting Data Ownership
consulting Data Protection
  5.21 Implement Appropriate Data Management Procedures
consulting Outdated Content
manual, qualitative Data Controllers
  5.22 Promote Responsible Emerging Technology Practices
manual, quantitative Emerging Technologies
consulting Disruptive Technology
consulting Technology Legislation
  5.23 Include Responsible Financial Policies
consulting Fuel Divestment
consulting Responsible Finance
  5.24 Include Organizational Philanthropy Policies
consulting Philanthropy Policy
consulting Success Criterion Voluntary Work
  5.25 Plan For A Digital Product Or Service’s Care And End-Of-Life
consulting End-of-life Care
  5.26 Include E-waste, Right-to-repair, And Recycling Policies
consulting E-Waste Policy
consulting Recycling And Repairing
consulting Refurbishment Strategy
  5.27 Define Performance And Environmental Budgets
consulting Environmental Budget
consulting Performance Budget
consulting Measurable Improvements
consulting Capacity And Maintenance
  5.28 Use Open Source Tools
consulting Open Source Policy
consulting Collaboration
consulting Contribution