test runner / deployment: launching test runner from broadcast - ATSC3

[jpiesing]

It would be good to have the ability to launch the WMAS (and DPCTF) test runner from broadcast signalling.
This issue is for ATSC3. HbbTV is #62.
This would be the following;

• PCAP pointing at the CTA cloud-hosted WMAS instance(s)
• The ability for a a locally hosted WMAS (and DPCTF) instance to generate a PCAP pointing at itself.

The PCAP may need to point at a Smart TV friendly landing page - see #61 .

[bobcampbell-resillion]

As noted in the other thread, Eurofins is prepared certainly to provide the 1st and can look at the second, but not until we've finished the latest batch of ATSC 3 materials for the NEXTGEN logo, for which these tests were not deemed a priority. [The NEXTGEN TV logo does not require these tests at present, because it still needs to agree what the pass criteria would be. So making a PCAP isn't the challenge for getting these tests adopted by that program...]

[cta-source]

Key points from discussion with our ATSC3 consultant:

... using “A3” for ATSC 3.0 and “STV” for smart TV, with “A3TV” as “ATSC3.0 Smart TV”… and “A1” for ATSC 1.0 …

• About AVC/AAC specifically:
o Typically, both A3TVs and A1TVs can handle AVC/AAC streaming media a la WAVE.
o However, an A3TV may not support AVC/AAC delivered via an ATSC 3.0 RF emission as the ATSC 3.0 standard does NOT include support for AVC and AAC.
o The A3 NEXTGEN tests do not test for AVC and AAC codecs in either its ‘Recommended’ or ‘Approved’ test categories.
• Wave and the ATSC3.0 modulator – architectural issues with RF transmission of non-ATSC streams
o CTA created an A1 test stream in 2013 to test A1 receivers, via a modulator.
o Several STVs tested did not support AVC/AAC delivered via ATSC 1.0’s 8VSB RF emission; some did.
o That observation was reconfirmed recently after NAB asked CTA about TVs supporting AVC/AAC (for temporary channel sharing, during the ATSC 1.0 to 3.0 transition).
• Because there is no requirement to support WAVE tech via the RF path, there is no guarantee that the path exists despite the presence of individual components.
• Engineering the RF stream for WAVE on A3TVs is not trivial. Depending on bandwidth and other use case requirements, the modulator requires specific configuration file(s) to properly construct an ATSC 3.0 valid emission.
• Crafting a PCAP stream that carries WAVE-related content is feasible but may not work on some systems due to lack of design-side requirements, as mentioned above.

Bottom line: The ATSC 3.0 modulator project for WAVE should have significant buy-in from ATSC experts and TV makers. It is rather different from HbbTV’s use case and may not be compelling to the intended user group.

A separate note: The ATSC community is a little shy about the test runner concept. Some effort about explaining what we've done to make it simple (and free) would perhaps be more helpful than a modulator track, in my opinion.

[jpiesing]

Key points from discussion with our ATSC3 consultant:

I started commenting line by line but ended up repeating myself. I think these comments are based on perhaps two major misunderstandings.

Firstly this issue is in the WMAS project, not the DPCTF project. The DPCTF project uses the test runner from WMAS. If ATSC3 would reference the WMAS test suite then the ability to run the test runner in the ATSC3 browser would be needed for WMAS even if it would (initially) not be used for DPCTF.

Secondly the comments seem to address the issue of AVC/AAC in the RF. This has never been proposed & is not what this issue is about. This issue is about a PCAP including the URL of the test runner. That's it. It would not need to include any video or audio at all unless that's a requirement for a compliant stream.

[bobcampbell-resillion]

It would not need to include any video or audio at all unless that's a requirement for a compliant stream.

Agreed; the launch PCAP AV requirements are independent of WAVE tests - and need to be ATSC 3.0 Broadcast compliant, and include a URL in the HELD to launch the WMAS tests in an ATSC 3.0 user agent. This is very simple, we have just been asked to prioritise other Broadcast-centric tests right now...

A/344:2021 references CTA-5000-B (although CEB 32.8:2020 references CTA-5000-A we can presume they mean to be in sync), so WMAS tests launched via an ATSC 3.0 compliant RF signal into an ATSC 3.0 user agent are potentially "in scope" for testing an ATSC 3.0 receiver. However, the challenges are not creating the PCAP. While some of the comment above may be based on some misunderstandings, this point is no less important to address:

Bottom line: The ATSC 3.0 modulator project for WAVE should have significant buy-in from ATSC experts and TV makers. It is rather different from HbbTV’s use case and may not be compelling to the intended user group.

[bobcampbell-resillion]

Practically what is the URL a static PCAP ought to point to to launch the tests for the A/344:2021 referenced "Web Media API Snapshot 2019"

I assumed it was https://webapitests2019.ctawave.org/ but at time of writing that gives me a cert invalid error then a directory listing...

[jpiesing]

Practically what is the URL a static PCAP ought to point to to launch the tests for the A/344:2021 referenced "Web Media API Snapshot 2019"

I assumed it was https://webapitests2019.ctawave.org/ but at time of writing that gives me a cert invalid error then a directory listing...

@louaybassbouss ?

TLS server certificate expired - see #65

[louaybassbouss]

Practically what is the URL a static PCAP ought to point to to launch the tests for the A/344:2021 referenced "Web Media API Snapshot 2019"
I assumed it was https://webapitests2019.ctawave.org/ but at time of writing that gives me a cert invalid error then a directory listing...

@louaybassbouss ?

TLS server certificate expired - see #65

confirm there is an issue with the certificate will provide a fix soon.

EDIT:
@bobcampbell-eurofins the URL is https://webapitests2019.ctawave.org/wave/ (wave/ was missing in your URL)

[bobcampbell-resillion]

Sorry for the delay, but we have a PCAP available pointing to https://webapitests2019.ctawave.org/wave/ we believe, which we'll liaise with Louay to make available.

[JohnRiv]

One thing to note, https://webapitests2019.ctawave.org/wave/ currently hosts the WMAS2018 tests. The WMAS2019 tests are currently in development, so that page will be updated in the coming weeks to be the 2019 tests.

[bobcampbell-resillion]

As discussed, probably best not to publicise the PCAP until the 2019 tests are actually profiled else this will cause confusion if people see the 2018 tests and their results change in future due to a different list; but when they are we'd like permission to post the PCAP to the wiki and provide some guidance notes around its use...

[cta-source]

No objection from CTA, if the chairs agree to posting. Mike Bergman

[bobcampbell-resillion]

I think I'm waiting for confirmation the 2019 tests are suitable to verify against receivers that implement an ATSC 3.0 user agent? See also #66

[JohnRiv]

@bobcampbell-eurofins the test suite is ready for verification

[bobcampbell-resillion]

We've checked and the basic "run all the tests" seems to work now via the PCAP we created which launches the 2019 runner/tests in an ATSC 3.0 user agent/application environment. But as discussed at length in the ad-hoc joint call, devices can't run/pass all the tests as some aren't viable on these platforms.

We'll look out for the subset that we all agreed was necessary and can validate that when its available? Let us know if we can help otherwise.

[JohnRiv]

When we have the subset of tests available, this can be tested

[JohnRiv]

In the HATF call on September 27th, we determined that since the subset of tests is now available (https://github.com/cta-wave/WMAS-subset) and it is possible launch the test suite via ATSC3 HELD pointing at the test suite, this issue can be closed.

/cc @bobcampbell-resillion if you have any further comments we can reopen if needed