Comparing Results from Tracker 1.0 and Tracker 2.0

Tracker 1.0 focused on HTTPS/SSL configuration so this comparison concerns only web results.

Validation of results was performed with the SSL Labs server tester.

Data used in the comparison was current as of May 17 2021 and may not reflect current results.

Domains

Tracker 1.0 (via https-everywhere.canada.ca) has results for around 3000 domains, while at this time Tracker 2.0 is monitoring 194. There are 136 domains for which both have results. When looking at 4 key metrics (HTTPS enforcement, use of HSTS, use of weak cipher suites, and certificate validity), 70 of these domains have differing results between the two.

HTTPS Enforcement

There are 4 domains for which the two versions of Tracker differ on detection of HTTPS enforcement. 3 of the differences are a result of Tracker 2.0 having a stricter definition of what it means to "enforce" HTTPS (less tolerant of redirects). The remaining domain was a match until 13/05/2021, when it became unreachable to Tracker 2.0's scanners. This issue is being investigated.

This result is not especially notable because all of the domains that could be checked still redirect to HTTPS as required and so could be said to enforce HTTPS. Tracker 2.0's HTTPS criteria should be modified to consider them compliant while still providing feedback that the redirect could be performed in a manner better aligned with best practices.

HSTS

There are 5 domains for which the two versions of Tracker differ on detection of HSTS implementation. In 3 of these cases Tracker 2.0 was unambiguously correct, with Tracker 1.0 reporting one false negative and two false positives (a positive result meaning detection of non-compliance). Of the remaining two domains, one produced different results when checking for HSTS with different methods and the other is the unreachable domain mentioned above.

Weak Cipher Suites

There are 44 domains for which the two versions of Tracker differ on detection of weak cipher suites in use. In 40 out of 44 cases Tracker 2.0 successfully detected the use of weak ciphers where 1.0 did not. Of the remaining 4, 2 were domains Tracker 1.0 falsely detected as using weak ciphers when they were not, and 2 domains were unreachable.

If this sample of 136 domains is representative of overall performance, Tracker 1.0 has a false negative rate of around 29.5% and a false positive rate of around 1.5% with respect to detection of weak ciphers. This suggests Tracker 1.0 has wrongly detected no weak ciphers in use for approximately 890 domains when extrapolating to its full set of 3038 monitored domains. It should however be noted that our sample of 136 domains is small compared to the population and not randomly selected. Nevertheless, this demonstrates that Tracker 2.0 is significantly more trustworthy in detecting the use of non-compliant cipher suites.

Certificates

Note: Although documentation is unclear, it appears Tracker 1.0 only evaluates the signature algorithm used on a domain's certificate. Tracker 2.0 primarily relies on this as well, although it does add checks for expired or self signed certs. Per guidance, it is also required to check if a certificate has been revoked and if the name on the certificate matches the domain request.

There are 33 domains for which the two versions of Tracker differ on detection of certificate validity. In 11 cases Tracker 2.0 was correct in detecting no issues (within the metrics scanned) where Tracker 1.0 falsely detected problems. In 12 cases Tracker 2.0 correctly found issues where 1.0 did not due to improved detection of incomplete cert chains. 4 domains, all belonging to the same organization, are unreachable seemingly only to Tracker 2.0's scanners. This issue is under investigation. The remaining 6 were unreachable for verification.

With the same disclaimers as above, this suggests Tracker 1.0 has a false negative rate of around 8.8% and a false positive rate of 8.1%, corresponding to approximately 270 and 240 domains, respectively, across 1.0's 3038 monitored domains.

ITPIN 2018-01 Compliance Determination

Both versions of Tracker are intended to monitor compliance with the ITPIN on HTTPS implementation. Tracker 2.0 differs in the determination of compliance in that it deems a domain non-compliant if it does not implement HSTS preload. This is in line with best practices but is not mentioned in the relevant policy documents so it should not be used to determine compliance status. This issue was resolved on May 26 2021.

Since a domain is compliant if it is passing in the above metrics, Tracker 2.0's more accurate detection of weak ciphers and certificate issues will mean fewer domains are considered compliant.