improve instructions on being able to access test runner from local network

[jpiesing]

I apologise if there's something hidden but ...

The documentation is silent about making the test runner visible on the local network that it's host is connected to.

There seem to be a number of options for this and none are mentioned.

• Docker runs in some kind of bridged mode where IP addresses from the local network are used for the interface between docker and the host
• Some form of port forwarding
• The network architecture is such that devices on the local network have a route to the test runner in its docker instance

I think this is one of the places where docker in WSL2 (without D. Desktop) is different from docker on Linux. Docker in WSL2 (without D.D) has an extra level of virtualization and potentially an extra network between it and the local network.

If some form of port forwarding is used then the instructions need to be clear that the IP address entered in the DUT must be the IP address of the PC hosting docker and not the IP address the docker instance is bound to.

[jpiesing]

For example, this article suggests some magic is needed when running docker in WSL2 without D.D. if the test runner is to be accessible from a DUT on the local network.

[jpiesing]

@afrawei has got caught by this.

@FritzHeiden Please can you comment on the following when docker runs inside WSL2 without D.D.

1. What IP address should be used in the host_override statement? The IP address of the inside of WSL2 (i.e. ifconfig -a in the bash window) or the IP address of the host PC on its network (i.e. ipconfig) in PowerShell?

2. Do port forwarding rules have to be setup on the host PC something like this ...

netsh interface portproxy add v4tov4 listenport=8000 listenaddress=0.0.0.0 connectport=8000 connectaddress=172.30.28.51
netsh interface portproxy add v4tov4 listenport=8443 listenaddress=0.0.0.0 connectport=8443 connectaddress=172.30.28.51
netsh interface portproxy add v4tov4 listenport=8444 listenaddress=0.0.0.0 connectport=8444 connectaddress=172.30.28.51
netsh interface portproxy add v4tov4 listenport=9000 listenaddress=0.0.0.0 connectport=9000 connectaddress=172.30.28.51
netsh interface portproxy add v4tov4 listenport=80 listenaddress=0.0.0.0 connectport=80 connectaddress=172.30.28.51

If yes, are these the correct and complete set of ports to forward?

(172.30.28.51 is the IP address of the eth0 interface inside WSL2 as printed by ifconfig in bash.)

3. @yanj-github Which IP address should be used in the OF configuration file, the one of the PC on its local network or the one inside of WSL2 (172.30.28.51 in this example).

[yanj-github]

@jpiesing I believe OF is co-located with test runner inside WSL. I think default value test_runner_url = http://localhost:8000 should work. If not, user should use WSL IP address.

Just to add on, I have tried to use default test_runner_url = http://localhost:8000 on WLS it work fine on our end.
I'd check if test runner is running by "docker ps" and try connect it on web browser.
It might be also worth checking ipv6 some systems by default use ipv6 instead of ipv4. If so disable ipv6.

[gitwjr]

localhost works fine for me. Tried to change to docker IP 172.17.0.1 (found using powershell ifconfig). After changing config.json to that address, closing the terminal, stopping and restarting docker service, and entering "http://172.17.0.1:8000/_wave/index.html" in Chrome it returns "This site can’t be reached172.17.0.1 took too long to respond." using port 8000. Same with Firefox. Changed back to local host, opened new terminal, restarted docker and it worked fine.

[afrawei]

I have my "test_runner_url" setting (inside WSL2) as IP address from local PC + ":8000". Now my environment is up and running after Jon marked out a few lines in hbbtv.js as a temporary workaround

[gitwjr]

Tried to run a test on mobile devices. The mobile devices (Android phone or iPad) will not connect to localhost. They both connect to IP 10.0.0.125. However Chrome on PC will only connect to locallost, not IP (have tried setting config.json to both. Only localhost works. When I run localhost on PC (with config.json set to localhost, and start a session (with mobile using :8000, Iit starts. I then enter the first 8 characters of the token on PC, configure the session and hit enter. The mobile devices say cannot connect to localhost. Seems the redirect to mobile is to localhost. But PC will not connect to IP.

[afrawei]

@gitwjr Let me share what I did and see if it works for you

1. Check the IP address <x.x.x.x> from WSL2 by ifconfig.

2. Open Windows Powershell with admin right, and enter below
   netsh interface portproxy add v4tov4 listenport=8000 listenaddress=0.0.0.0 connectport=8000 connectaddress=<x.x.x.x>
   netsh interface portproxy add v4tov4 listenport=8443 listenaddress=0.0.0.0 connectport=8443 connectaddress=<x.x.x.x>
   netsh interface portproxy add v4tov4 listenport=8444 listenaddress=0.0.0.0 connectport=8444 connectaddress=<x.x.x.x>
   netsh interface portproxy add v4tov4 listenport=9000 listenaddress=0.0.0.0 connectport=9000 connectaddress=<x.x.x.x>
   netsh interface portproxy add v4tov4 listenport=80 listenaddress=0.0.0.0 connectport=80 connectaddress=<x.x.x.x>

3. Check IP address on your Windows PC (physical IP) and fill it in config.json
   Now when you want to connect to TestRunner from other devices, use IP address from 4.
   Whenever your physical IP is changed, same change shall be adopted in config.json

4. In observation-config.ini file, you can use IP address from 1 so you don't need to modify it when physical IP is changed

If you have firewall issue, I found useful websites for reference
Windows WSL 2 can't ping host machine
Access a web server which is running on WSL (Windows Subsystem for Linux) from the local network

[louaybassbouss]

Here is how it works on our side step-by-step:

1. In Windows Powershell run:
   ipconfig and copy the local IP address of your windows host e.g. 192.168.1.23 (lets call this <host_ip>). This is the IP you should use in "host_override": "<host_ip>" (see config.json)

2. In Windows Powsershell run:
   wsl hostname -I and copy the (first) IP address e.g. 172.24.202.133 (lets call this <wsl_ip>)

3. In Windows Powershell (must be run as Admin by Admin User!!!!) run:
   netsh.exe interface portproxy add v4tov4 connectport=8000 connectaddress=0.0.0.0 listenport=8000 listenaddress=<wsl_ip>

Please make sure you use the right IP address (<host_ip> & <wsl_ip>) in the right command/config.

VERY IMPORTANT!!

In some situations, it may happen that the <host_ip> (the IP of the windows host in local network) is unknown for the WSL distribution (it happens to us when the local IP is 192.168.X.X but not for this local IP 10.X.X.X). This is why the host_check.sh script (which runs inside docker inside WSL) fails to ping the <host_ip> (this is I think what happens on @gitwjr side, but not on @afrawei side). For this reason, we recommend to change it to warning and not raise an error. This means you need to change check-host.sh#L21 from exit 1 to exit 0 (if this fixes the issues on your side, we will provide a PR).

After this change, you need to:

1. rebuild the image: ./build.sh
2. recreate container: docker-compose up --force-recreate

Please follow these steps and let us know if issue could be solved on your side.

[gitwjr]

I made the above change to host_check.sh (its called check-host.sh) in my file: "-" not "_"). The test now runs in my PC browser at P 10.0.0.125 instead of just localhost. I can also open the link on my phone. However, when I I try 10.0.0.125:8000/_wave/configuration.htlm on the PC and run a test the phone comes back "can't connect". It is still redirecting to localhost on the phone. If I run a test on my PC using 10.0.0.125 the test runs at localhost and the results pages shows http://localhost:8000/_wave/finish.html?token=3277e188-3944-11ef-8f5d-0242ac110002 when complete. Appears the phone still cannot connect to localhost which is where the test actually runs.

[louaybassbouss]

I made the above change to host_check.sh (its called check-host.sh) in my file: "-" not "_"). The test now runs in my PC browser at P 10.0.0.125 instead of just localhost. I can also open the link on my phone. However, when I I try 10.0.0.125:8000/_wave/configuration.htlm on the PC and run a test the phone comes back "can't connect". It is still redirecting to localhost on the phone. If I run a test on my PC using 10.0.0.125 the test runs at localhost and the results pages shows http://localhost:8000/_wave/finish.html?token=3277e188-3944-11ef-8f5d-0242ac110002 when complete. Appears the phone still cannot connect to localhost which is where the test actually runs.

ist "host_override": "10.0.0.125"? please make sure to rebuild the image ./build.sh and the recreate container docker-compose up --force-recreate

[gitwjr]

I did both. And changed the config.json to IP from localhost. Before the change IP didn't work on the PC browser, only localhost.

[gitwjr]

But phone still gets redirected to localhost after opening and running the test and can't connect to localhost.

[louaybassbouss]

But phone still gets redirected to localhost after opening and running the test and can't connect to localhost.

yes localhost on phone will not work. can you please paste the content of you config.json?

[gitwjr]

{
"browser_host": "web-platform.test",
"alternate_hosts": {
"alt": "not-web-platform.test"
},
"wave": {
"aliases": [],
"results": "./results",
"timeouts": {
"automatic": 100000,
"manual": 20000000
},
"enable_results_import": false,
"web_root": "/_wave",
"persisting_interval": 20,
"api_titles": [],
"host_override": "10.0.0.125"
}
}

[louaybassbouss]

I seems be that the docker image which uses localhost in still in docker cache. can you remove the old docker container and create again (run ./build.sh and then docker-compose up --force-recreate).

[gitwjr]

I closed terminal adn opened a new one. Then stopped dpctf and restarted it. Now running on phone. Results show pass. I didn't record it so no OF analysis.

[gitwjr]

I had already rebuilt the container after making the change to host-check.sh. Appears the change doesn't fully flow through until close and reopen. Thanks.

[gitwjr]

Tried to run a test using my PC to host the test an my iPad as the mobile DUT. It appears to start on the iPad, goes to "test about to start", then to "s: waiting; a: initialize". showing 10.0.0.125 in the browser window. It sits there until it comes back "session complete" except results are 2 of 2 tests timed out. No flashing QR codes or video playing. Its an older iPad mini 2, model #ME2772LL/A, software version 12.5.7 so it may not be able to run it? Apple no longer updating it.

[gitwjr]

Tried on iPhone SE. Same result as iPad. But it works on my Pixel 6a.

[louaybassbouss]

@gitwjr on iOS/iPadOS you need to make sure that MSE is supported. iPhone SE does not support MSE in Safari Browser.

[FritzHeiden]

extended the readme to explain how to make the test runner accessible from other devices when using windows+wsl #106

[gitwjr]

With the change either localhost or my host IP address work on the PC. As expected, only host IP works on Pixel 6a. From my perspective this issue has been resolved.

[louaybassbouss]

@afrawei you added also the three ports, we think these are not necessary and on our side and @gitwjr, the installation work without them. Can you please remove them and check if you installation still work. The TR only uses 8000 (for http) and 8443 (for https).

netsh interface portproxy add v4tov4 listenport=8444 listenaddress=0.0.0.0 connectport=8444 connectaddress=<x.x.x.x>
netsh interface portproxy add v4tov4 listenport=9000 listenaddress=0.0.0.0 connectport=9000 connectaddress=<x.x.x.x>
netsh interface portproxy add v4tov4 listenport=80 listenaddress=0.0.0.0 connectport=80 connectaddress=<x.x.x.x>

[afrawei]

@louaybassbouss Yes, I have removed them since you suggested, only port 8000 is remained, and it works properly

[louaybassbouss]

@afrawei thanks for the update. you can add 8443 as well, if you later want to run https tests (in this case, you need to configure a domain and corresponding SSL certificate). The documentation is updated and should reflect all the steps for Windows WSL https://github.com/cta-wave/dpctf-deploy?tab=readme-ov-file#configure-access-to-the-test-runner. @jpiesing if you are happy with it, feel free to close the issue.

[gitwjr]

@louaybassbouss @FritzHeiden @afrawei
I am bring up a clean install pc for Mike/CTA. Clean new Windows 10 install. I installed everything (WSL2, Ubuntu 22.04, Docker/Docker-compose, Git, etc.) and built the TR. Works fine on LocalHost but not using my home network or a router with nothing else connected to it IP address. IP config when connected to my home network shows

Ethernet adapter vEthernet (WSL):
Connection-specific DNS Suffix . :
Link-local IPv6 Address . . . . . : fe80::14cf:94e2:b53c:7a93%39
IPv4 Address. . . . . . . . . . . : 172.19.208.1

Wireless LAN adapter Wi-Fi:
IPv4 Address. . . . . . . . . . . : 10.0.0.64

For the router (no internet connection) I get
Ethernet adapter vEthernet (WSL):
IPv4 Address. . . . . . . . . . . : 172.19.208.1

Wireless LAN adapter Wi-Fi:
IPv4 Address. . . . . . . . . . . : 192.168.0.

Running either of these I get failed to connect. I ran netsh.exe... for port 8000 and for port 8443, Of course I changed config.json each time I changed the connection. All work for localhost. Neither work using host IP address (e.g. 10.0.0.64 or 192.168.0.

[gitwjr]

@louaybassbouss @FritzHeiden @afrawei I am bring up a clean install pc for Mike/CTA. Clean new Windows 10 install. I installed everything (WSL2, Ubuntu 22.04, Docker/Docker-compose, Git, etc.) and built the TR. Works fine using LocalHost but not using my home WiFi network or a new router with nothing else connected to it. IP config when connected to my home network shows

Ethernet adapter vEthernet (WSL): Connection-specific DNS Suffix . : Link-local IPv6 Address . . . . . : fe80::14cf:94e2:b53c:7a93%39 IPv4 Address. . . . . . . . . . . : 172.19.208.1

Wireless LAN adapter Wi-Fi: IPv4 Address. . . . . . . . . . . : 10.0.0.64

For the router (no internet connection) I get
Ethernet adapter vEthernet (WSL): IPv4 Address. . . . . . . . . . . : 172.19.208.1

Wireless LAN adapter Wi-Fi: IPv4 Address. . . . . . . . . . . : 192.168.0.64

Running either of these host IP addresses I get failed to connect. I ran netsh.exe... for port 8000 and for port 8443, and I changed config.json and stopped/restarted docker each time I changed the connection and IP address. Both work for localhost. Neither work using host IP address (e.g. 10.0.0.64 or 192.168.0.64. I can use either of them (with the corrected IP local and WSL addresses) on my computer with no problems. Something about the new computer does not let me run anything but localhost.

My computer uses 172.29.128.1 for WSL, 192.168.0.13 for the new unconnected wifi router and 10.0.0.22 for my home router.

[gitwjr]

Accidentally hit close. See above for comment.

[afrawei]

@louaybassbouss
There is a mistake in the deploy page about Windows WSL proxy configuration.
The wsl-ip address shall be assigned to connectaddress instead listenaddress.

[gitwjr]

Thanks. I figured that out although I did it a different way (thanks to Chatgpt).
netsh interface portproxy add v4tov4 listenport=8000 listenaddress=0.0.0.0 connectport=8000 connectaddress=172.19.210.158

And added my network IP address:
netsh interface portproxy add v4tov4 listenport=8000 listenaddress=10.0.0.64 connectport=8000 connectaddress=172.19.210.158

As for the WSL address, I get different onees depending on whether I run ipconfig or wsl hostname -I
Which one should I be using? It also appears to change. At least one of them. They are now 172.29.128.1 (ipconfig) and 172.29.130.45 (wsl hostname -I

I have both computers working now. However, when I run with the CTA computer as host and my computer as DUT, I get a single good run but subsequent runs begin and then immediately go to the green screen showing $: error; a: play; ct: 0;. Then it goes to $: finished; a: play: ct:0. Then to the white "Session Complete". with QR code. I ran it a number of times (using both my computer and my phone as DUT). I tried stopping dpctf, stopping docker, and stopping dpctf and docker and then restarting with the same result. I then closed the terminal and and reopened it and ran the test. Worked once and then I got the same result with green screen with both my computer and with my phone. Tried closing everything (docker/terminal) and then restarting. This time it simply went to the green screen. Any thoughts?
I need to get this running reliably so I can return the computer in working order to Mike. I will run more tests tomorrow (Sunday) and update.

[gitwjr]

@louaybassbouss @afrawei @yanj-github
Update: I have both computers up and running and can do so reliably as well as do an analysis of the video recordings done on both. i.e. can run either as host with the other as DUT. I ca also run it on my phone but I can't record anything since my phone is my camera. I still get a green screen failure (on both) but subsequent runs appear to work without making any changes to the setups. Weird but at least its working reliably as of today (after the initial green screen error). Any thoughts on why ipconfig gives a different wsl ip using ipconfig vs wsl hostname -I?
I also created a firewall rule to open port 8000. Not sure if that was one of the reasons it started working. Any thoughts?
Also, as noted above, I ran netsh to listen to my host ip as noted above. Also not sure if that is necessary. Thoughts?
My observation results show some failures for duration and missing frames but the lighting was not optimal so I will re-run when the sun is not shining as brightly.
Looks like the deploy "netsh" instructions need to be updated. I need to make a recording of building the TR and OF and I'd like to be able to refer to Deploy when doing it so sooner is better since I have to get it done early this week.

[gitwjr]

@louaybassbouss @afrawei @yanj-github
Further testing: Restarted docker on CTA machine (host) to my PC (DUT) and ran tests. Failed error/green screen 3x in a row. 4th time it ran to end and did so 3x more without failure. No changes to CTA machine between this afternoons tests (it was running fine before I stopped docker (dpctf) and this evening when I restarted dpctf and it failed 3x, then passed 4 in a row. Something strange going on since nothing has changed on either machine all day. The green screen (immediate error) seems to occur after stopping/restarting docker on the host. I am not running docker on the DUT. Just the Chrome browser"10.0.0.64:8000/_wave /index.html" using host machine's IP as the host IP. And once it starts working it seems to continue to work until I stop docker and then come back later to run more tests. Repeat... Any thoughts?

[gitwjr]

@louaybassbouss @afrawei @yanj-github
Reran tests this am. Kept getting green screen failures. When this happens I typically stop the test and delete the session to save some time. Failed 4x but on the 5th test I let it run to timeout (140 seconds) even though the DUT was stuck at the green screen. The next test worked. It appears that stopping the test and deleting the session leaves something in docker that causes subsequent tests to fail. Letting the host run the test to timeout. DOES THIS MAKE SENSE?
The next test failed with green screen. I didn't let it go to timeout. 3 more went green screen. I let the next go to timeout and the next one I ran was successful. Appears the solution is let the test run to timeout when it fails. Does not explain why it fails on occasion when nothing has been changed but at least there appears to be a way to clear it. INPUT?

[gitwjr]

Ran another test following the successful one above. Went green screen. Let it timeout and the next 3 worked. Again, nothing changed at any point. Green screen appears either random or after sitting for a few minutes after a test. But letting it run to timeout fixes it at least for subsequent tests run soon after.

[jpiesing]

@gitwjr You explain a lot about how killing & restarting docker (etc) don't fix the green screen problem. Did you exit and restart the browser on the PC that was acting as the DUT? Does it make any difference if you use a different browser implementation on the PC that is acting as the DUT? It is certainly possible that something in the integration between the browser and the OS / video decoder isn't properly cleaned up.

[gitwjr]

I typically leave a Chrome session with my "working" tabs open and open a separate session for testing. I rarely close the main session unless restarting the computer. I will try to "catch" a failure and see if closing and reopening Chrome fixes it without letting the failed test run to completion. This am I ran a test (random access of presentation) and it failed but slightly differently. It ran to frame 501, sat there, and then went to video ended event fired with test workflow failure. Since it ran to completion the next test passed.

[jpiesing]

I don't think I've ever interrupted a test run, I always let them run to completion which might explain why I've never seen this particular failure mode.

[gitwjr]

I only interrupt it when it fails. Otherwise, I always let it go to completion. I just ran a test and it failed, stalling at frame 1. This time it was only around 1-1.5 hours between tests and I had not stopped docker/dpctf or closed the terminal on the host (DUT is not running docker/dpctf). I stopped the test before completion (which never leads to a successful follow up test). I then closed all Chrome sessions, restarted Chrome and immediately ran another test. As always it too stalled at frame 1 but this time I let it run to completion. Then I immediately ran another test without restarting Chrome. It passed as it always does when I let the stalled/failed test run to completion. So it is not related to closing Chrome. The reason for the initial failures is not clear or obvious. Seems to be related to time between tests but there could be other things at play although I did not open, close, or change anything on the host computer. I was using the DUT to try doing some screen recordings for the instruction video. To test a different browser is time consuming since it seems to take time (hours) before the failure reappears. Whatever is causing the failure, the firing of the video ended event is the main clue. That always fixes it for subsequent tests until I let the 2 systems sit without testing for a period of time. And it always fails the first test after leaving them running overnight (although never with docker running since a system restart/crash seems to blow up the daemon and/or containers/images and require a rebuild).

[jpiesing]

I only interrupt it when it fails.

How do you interrupt it? Which device do you do that on?

[gitwjr]

I use "Stop" on the host screen and then "Delete Session". I have tried ""Pause" then Stop then Delete as well. No difference. It is only fixed when I let it run to completion. That always fixes it.

[gitwjr]

Just ran a test (Chrome) and it failed. Retested on Chrome. Failed. Retested on Edge. Failed. Retested on Firefox and it passed. I stopped the Chrome and Edge tests using Host Stop and Delete session. The Firefox test passed. Will retest to see if Firefox always works after a failed test. Same process: Test and fail on Chrome, stop, delete. Retest using Firefox.

[gitwjr]

Ran another test on Chrome after a period and it failed same way at frame 1. Did not let it run to completion to "preserve" the failure. Ran same test using firefox and it ran fine. Since the last time I did this I tried both Chrome (after shutdown) and Edge and both failed follow up tests and then Firefox worked it may be an obscure issue with Chromium? I believe both Chrome and Edge are based on chromium.

[gitwjr]

Sorry for all the updates but I keep testing to gain data. Chrome test failed again on buffer underrun and recovery at Frame 1 and local random access of a wave presentation at frame 501. Did not run to completion. Then ran buffer underrun and recovery on Firefox and it ran fine. So pretty consistent Chrome appears to be part of the problem but it clears when failed test runs to completion. Firefox doesn't seem to fail. I'll test more with Firefox to be sure. If it continues to be good, I think we should recommend Firefox at least for WSL.

[FritzHeiden]

@louaybassbouss There is a mistake in the deploy page about Windows WSL proxy configuration. The wsl-ip address shall be assigned to connectaddress instead listenaddress.

fixed with #110

[gitwjr]

@louaybassbouss @yanj-github @jpiesing
After more tests with Firefox, it appears to have a similar problem with audio tests on 2-device setups at times.
Summary: Testing done on 2 different computers, identical Test Suite build, 2-device setup (both PCs) with either as Host.

• When using Chrome (and Edge), the tests (I've run it with many different tests) sometimes fail at frame 1 (non-audio tests), or frame 501 (audio test). Letting tests with this type of error run to completion clears the error/failures for subsequent test runs.
• Firefox does not exhibit this error for non-audio testing but it does fail (freezes) consistently with audio tests (frame 501). Letting it run to completion does not clear the error. Subsequent tests with Chrome pass.
• Chrome (once error cleared by allowing freezing test to run to completion) passes audio and non-audio testing on subsequent tests.
• Recommendation: Add note that if a test freezes on a Chromium browser, allow it to run to completion. Then retest.
• For non-chromium browsers that repeatedly freeze, recommend they try a different browser.
• Although this was done on 2 different Windows/WSL PCs (one Windows 10, other Windows 11 PRO) with the same (updated) versions of WSL, Ubuntu, Docker, I can't say if it is consistent across all PCs or when testing with a TV on 2-device setups. I don't have the equipment to do such testing.
• I see the change to netsh/Port 8000 has been made. That part of the issue is now fixed.

I added the following note to my WSL instructions. "NOTE: There have been reported cases where tests stall/freeze on e.g. frame 1 or other frames. Allowing stalled test to run to “Session completed” often clears this error for subsequent test runs. If the error persists, try running the tests on a different browser."

As I don't see any fix for the error/freeze problem noted above, I recommend closing this issue, assuming something along the lines of the above note is added, if there are no further comments.