Feature Request: Use Kodi's proxy

[extremelyonline]

What Problem Does This Solve?

I am sorry for re-posting this issue, which is related to #883. Upon further testing, it seems that Kodi's proxy is not being used:

• On the router, I blocked all internet traffic of a TV except LAN traffic
• On another computer in the local network, I set up a SOCKS proxy server
• After configuring Kodi to use the SOCKS server, Kodi can reach the internet (e.g. updating Addons, etc.) but not when using the Youtube plugin.

#841(comment) indicate that the proxy settings works. However, the user is running a local SOCKS server on the same machine to bypass DPI.

Would someone please test whether the proxy settings work when using the Youtube app?

---

Suggest a Solution (optional)

It would be great if the addon would be able to follow Kodi's system-wide proxy settings (#32). A menu to set a proxy for the addon only is another method.

This may be of help: YouTube addon don't use Kodi's proxy #795

---

[MoojMidge]

Can you describe what the intended purpose of the proxy is?

As mentioned in the original issue you opened, implementing something like this in the plugin would only be of use for the plugin listings and nothing else i.e. playback would be unaffected.

The use case in #841 is the opposite of this i.e. the Kodi proxy settings are used for playback.

[extremelyonline]

The purposes include getting around geo-blocking or allowing Youtube access when the TV itself is quarantined from the internet.

I'll do more testing as I am not sure if the plugin is indeed using the Kodi proxy settings currently

[MoojMidge]

The purposes include getting around geo-blocking

Using a proxy in the plugin won't really work for this. The playback would/should use the Kodi proxy settings. If it doesn't, that is an issue that needs to be raised with InputStream.Adaptive or Kodi itself.

allowing Youtube access when the TV itself is quarantined from the internet

This would work I suppose, but only for the listings of this plugin. Would still need to set the Kodi proxy settings for everything else.

I'll do more testing as I am not sure if the plugin is indeed using the Kodi proxy settings currently

Note as well that even if it isn't, the ability to actually set a proxy within the plugin may not actually work so well, due to issues with how Python requests handles system proxies in sessions, compared to individual requests. I have no way to test this either.

[pnperehod]

I have a more weird case. Proxy works well with all the youtube videos except for live streams.

[MoojMidge]

Please refer to the comment directly above your own.

Not planning on doing anything about this for the reasons previously identified. AFAIK there is only one potential scenario where something like could be useful, but it is not clear from anyone asking for this whether it is even actually required.

If you think there is a need for this, providing a full debug log with cURL component logging enabled would be the first step.

[Liqianyu]

I can confirm that the YouTube add-on does not “fully follow” the Kodi proxy settings, hereafter referred to as YT add-on.
The YT add-on does not follow the Kodi proxy settings. Simply put, if the add-on uses proxy settings. Then it should get stuck on misconfigured/invalid proxy settings (or better yet, it should indicate a network issue and alert the user when a proxy is set)
When playing videos and live streams, the video stream has to follow the Kodi proxy settings. So if the proxy is misconfigured\invalid, then it will get stuck when playing the video stream (and will eventually prompt a playback failure). I'm not quite sure if this is due to ISA or Kodi itself following the proxy settings, is the YT add-on calling ISA?

What are the benefits of YT add-ons following the Kodi proxy settings exactly?
For most parts of the world, the biggest use is to bypass geographic region restrictions. This is not in line with the Kodi community guidelines, so some developers are not interested in this at all either. However, YouTube is still blocked in some countries around the world.
For more information, please refer to
https://en.wikipedia.org/wiki/Censorship_of_YouTube#Countries_where_access_to_YouTube_is_currently_blocked
These countries and territories are home to more than 1 billion internet users who are unable to access YouTube directly.
So if the YT add-on can fully follow the Kodi proxy settings, it will be very helpful for users in these countries and regions.

I hope the developers will take the time to investigate and improve the proxy support for YT Add-ons and I can help with testing.
The best implementation would be to provide a proxy option in the settings of the YT add-on that would allow the add-on to follow, ignore, or even allow users to configure their own proxy (independent of the Kodi proxy settings).

[MoojMidge]

🤷

#884 (comment)

[Liqianyu]

🤷

#884 (comment)

In the latest v7.1.0 Beta 1, the YT add-on footprint still won't use the Kodi proxy settings.
I grabbed the logs and if the network can't connect directly (proxy can) to Google, then it can't log in.

Kodi Logs
https://paste.kodi.tv/uliqelevos.kodi

[MoojMidge]

Unfortunately you are probably the wrong person to be able to provide the necessary logs.

What I was trying to say is that even if proxy support was added to the plugin, video playback would be unaffected, which makes adding proxy support a somewhat useless endeavour.

Need a log of someone who uses a proxy, and who can reach the Google/YouTube servers normally, to also get a debug log, with cURL component logging enabled, of the video playback.

Let's see if someone else can help out here.

[Liqianyu]

It's not a big deal, I can access Google\YouTube normally via VPN on a router device, which is transparent to the client so this doesn't affect the client, and then configure the HTTP\Socks proxy on the Kodi to capture the add-on logs.

This is the log captured as you requested
https://paste.kodi.tv/noromideqi.kodi
Some extraneous content and IP addresses were removed for privacy reasons.

@extremeonline says he's blocking internet traffic to the TV (except for the local LAN), so to me that equates to no access to Google\YouTube.
I don't know how you see it, but if the add-on can't connect to Google\YouTube, it can't get the video listings at all right? In my tests, it can't log in at all, so why is it pointless to say that the YT add-on supports proxies? Proxy support allows YT add-ons to connect to Google\YouTube through a proxy.
It seems to me that the problem is that the YT add-on doesn't use Kodi's proxy settings, which is of course a feature enhancement request. The add-on can use\not use the Kodi proxy settings, currently the add-on does not. This is up to the add-on and not Kodi, as discussed at xbmc/xbmc#17729.

So the video playback part is not controlled by the YT addon? Their proxy support needs to be fed to the corresponding addon maintainer right, I don't know much about Kodi addon development. But after my testing, the video\live streams in YT add-ons are using Kodi proxy settings.

Anyway, I refer to the code change @hak0 mentioned jdf76/plugin.video.youtube#795 (comment)
The test YT add-on will use hardcoded proxy settings that can be accessed through a proxy without a direct connection to Google\YouTube. So the add-on just needs to add the configure proxy option inside the settings (or use the Kodi proxy settings).

https://forum.kodi.tv/showthread.php?tid=369592
If possible, maybe the add-on should allow to ignore the Kodi proxy settings? Of course video streaming can't be controlled by YT addons.

[MoojMidge]

so why is it pointless to say that the YT add-on supports proxies?

Because if the playback can't use the configured proxy then what is the point? And it is not even certain whether plugin specific settings are needed.

In xbmc/xbmc#17729 the OP produced a log showing that the Kodi proxy settings are not being used for listings.

In https://forum.kodi.tv/showthread.php?tid=369592 the OP is asking how to bypass the proxy settings, presumably because they do work for the listings.

Other people commenting on this issue have also indicated the same while others indicate the proxy is not being used, but no one has provided a log until now, and I simply don't have enough information or time to spend on something that may prove to be useless. Perhaps Kodi on different platforms behaves differently with respect to using the Kodi or system proxy?

Your log indicates that cURL is using the proxy, but that the initial manifest request from InputStream.Adaptive is not going via the proxy.

For VOD using MPEG-DASH this is not a problem since the manifest is begin served from a local server created by the plugin.

For live streams and non-adaptive VOD this may be a problem, and possibly why @pnperehod was saying live streams didn't work.

after my testing, the video\live streams in YT add-ons are using Kodi proxy settings.

Can you also get a log showing playback of a live stream?

Can you also describe the proxy setup you are using and how have you configured the Kodi proxy settings?

[Liqianyu]

Because if the playback can't use the configured proxy then what is the point? And it is not even certain whether plugin specific settings are needed.

But the add-on controls logging in and getting YouTube playlists, so it's part of the overall proxy support. And if I understand correctly, it's controlled by the add-on until the video actually starts playing. As I mentioned with @extremelyonline, if there is no direct access to Google\YouTube (via the Kodi proxy access Google\YouTube), then the add-on won't work at all.

In xbmc/xbmc#17729 the OP produced a log showing that the Kodi proxy settings are not being used for listings.

I think that's correct, although that was the old Kodi and add-ons version. But his test results are the same as what we are observing now.

In https://forum.kodi.tv/showthread.php?tid=369592 the OP is asking how to bypass the proxy settings, presumably because they do work for the listings.

I don't know if @toml272 actually observed that YouTube add-ons have use proxies. Perhaps he's just confused YouTube add-ons with cURL. Based on my testing, the current Kodi version of cURL does use the Kodi proxy. So it wouldn't be wrong to say that YouTube traffic goes through the proxy.

I tested it on both Android and Microsoft UWP platforms. There are differences in Kodi functionality between platforms, but I did not observe a difference in agent behavior.

For live streams and non-adaptive VOD this may be a problem, and possibly why @pnperehod was saying live streams didn't work.

I don't really understand what non-adaptive VOD stands for exactly. But for now, I modified the requests.py and it looks like all traffic goes through the proxy. no problems with VOD or live streaming.

Anyway, here's the log of the add-on playing live.
https://paste.kodi.tv/kareqizuva.kodi

The proxy protocol is SOCKS5 with remote DNS resolving and the server is on the LAN. No other special settings.
https://kodi.wiki/view/Settings/System/Internet_access#Use_proxy_server

By the way, if I turn Use InputStream.Adaptive off in the add-on's advanced settings, or set Use for live streams to HLS\Multi-stream HLS, it reports an error when playing the live stream. Does this mean its deprecated? Setting Adaptive HLS seems to work well to get the highest bitrate. And MPEG-DASH has a slightly lower bitrate compared to that, but you get a 360p VP9 format. But I get very low resolution with MPEG-DASH on the UWP platform. For example 144p and 240p, could this be a limitation or bug unique to the UWP platform?

[MoojMidge]

if there is no direct access to Google\YouTube (via the Kodi proxy access Google\YouTube), then the add-on won't work at all.

As mentioned previously the plugin doesn't control proxy use for playback. What is the point of listing a video that can't be played?

But for now, I modified the requests.py and it looks like all traffic goes through the proxy. no problems with VOD or live streaming.

Might need to stick with that for now until some additional logs are provided to better understand what is actually happening on various platforms.

Does this mean its deprecated?

No, it means that Kodi is loading the manifest rather than InputStream.Adaptive.

As mentioned there is too much contradictory information to spend time working on something like this.

[Liqianyu]

As mentioned previously the plugin doesn't control proxy use for playback. What is the point of listing a video that can't be played?

Current tests show that YouTube video streams will be loaded through the Kodi proxy, so if the Kodi proxy is configured. It will be possible to play the video normally.

All in all, thank you and all the maintainers of this add-on. I very much understand the limited time to meet high priorities first. Adding proxies to the add-on itself may be a relatively niche need.
However, I think we can keep the issue open for discussion and see if anyone else can get involved.

[MoojMidge]

Current tests show that YouTube video streams will be loaded through the Kodi proxy, so if the Kodi proxy is configured. It will be possible to play the video normally.

To be clear I will try to add this in when I can, as your logs identify that doing so won't be waste of time.

However there is behaviour that you yourself have observed that, and that other people have commented on, that indicates this may not work properly across a range of different installations or for different videos, which is why I'm trying to get logs.

There appear to be certain circumstances where ISA or Kodi don't use the system proxy or Kodi proxy settings, and I'm trying to understand why this happens.

By the way, if I turn Use InputStream.Adaptive off in the add-on's advanced settings, or set Use for live streams to HLS\Multi-stream HLS, it reports an error when playing the live stream.

Can you get a log of this?

[pnperehod]

I don't really understand what non-adaptive VOD stands for exactly. But for now, I modified the requests.py and it looks like all traffic goes through the proxy. no problems with VOD or live streaming.

Can you share this changes?

[Liqianyu]

@MoojMidge
Thank you for the time and effort you put into this.
I think it's possible that it varies from video to video (live, VOD) and from platform to platform, but it needs to be better and more investigated.
As always, I capture logs on the Android TV platform. kodi version for Omega latest nightlies.
https://paste.kodi.tv/mipecipubi.kodi
The log shows ffmpeg[0x27799afe98]: [tls] Error decoding the received TLS packet.
It looks like the failure is in ffmpeg, so this has nothing to do with the add-on right?
I'm not sure if this is due to my Kodi version, configuration, or even network.
Hopefully someone else can repeat my test.

@pnperehod
Sure, but this is just a crude implementation used to demonstrate that add-ons are easy to proxy through Kodi in my opinion. The add-on maintainer has more to think about.

    # add my hard-coded proxy settings here
    proxies = {
      'http': 'http://ip:port',
      'https': 'http://ip:port',
    }
    response = None
    try:
        response = self._session.request(method, url,
                                         params=params,
                                         data=data,
                                         headers=headers,
                                         cookies=cookies,
                                         files=files,
                                         auth=auth,
                                         timeout=timeout,
                                         allow_redirects=allow_redirects,
                                         proxies=proxies,
                                         hooks=hooks,
                                         stream=stream,
                                         verify=verify,
                                         cert=cert,
                                         json=json)
        if not getattr(response, 'status_code', None):
            raise self._default_exc[0](response=response)

[pnperehod]

To my surprise, live streams began to work again through the proxy with 7.1.0-beta2!

[MoojMidge]

@Liqianyu - do you have access to any logs on your proxy server to identify what connections are made through it? It seems like the manifests requests use the proxy via curl but the actual MPEG-TS segments are being loaded using ffmpeg directly and is not using your proxy. If you can check what the proxy server is doing then it would be good to confirm this.

@pnperehod - if you want anything to happen with this then you need to provide debug logs of it working or not working. Any change that you may have seen is unrelated to any change in the plugin because no such changes have been made.

[Liqianyu]

@MoojMidge
Yes, I use a web development and proxy tool on iOS. So viewing logs & capturing traffic is easy.
So I can confirm that all manifests requests and video traffic are using the Kodi proxy.
I captured the logs again and enabled curl and ffmpeg detailed logs.
Kodi (21.1 (21.1.0) Git:20240912-29b3dc5a65)
plugin.video.youtube v7.1.0+beta.3
Playing the same live video, the first playback fails using Multi-stream HLS, the second playback fails using HLS, and the third playback succeeds using Adaptive HLS.
https://paste.kodi.tv/usawokifaw.kodi

@pnperehod
In my previous tests, I observed that the live streams were indeed using the Kodi proxy, is it possible that this has something to do with your setup of YouTube add-ons and Kodi?
I can confirm again. The live streams both MPEG-DASH and Adaptive HLS clearly show that they are using the Kodi proxy settings.
Of course it is important to note, as I provided the debugging information. I am testing on Android TV and Kodi 21.1.

[MoojMidge]

So I can confirm that all manifests requests and video traffic are using the Kodi proxy.

Playing the same live video, the first playback fails using Multi-stream HLS, the second playback fails using HLS, and the third playback succeeds using Adaptive HLS.

Can you confirm that the first two attempts to play also went through your proxy completely? As mentioned before the Kodi logs seems to indicate that they actual MPEG-TS segments are not using the proxy settings which is why it fails. You would see a number of index.m3u8 files being requested through your proxy, followed by a seg.ts file, which doesn't appear to go via your proxy and which results in playback failing.

Playback through InputStream.Adaptive seems to work though, and that is the preferred method for playback so this should work for most types of streams.

[Liqianyu]

@MoojMidge
In the last test, I remember that the Kodi agent was not turned on. So I don't think the issue is network related.
This test I noticed that every time I play a live stream and then switch the live stream format in the add-on, it uses the original format once again (the one before the switch) and uses the format I currently have set the next time.
I'm not quite sure what the difference is between the four types? Any advantages or disadvantages? It seems to me that using MPEG-DASH works well.

[MoojMidge]

In the last test, I remember that the Kodi agent was not turned on. So I don't think the issue is network related.

Strange, will chalk it up to some issue with how ffmpeg is handling the https traffic compared to curl.

This test I noticed that every time I play a live stream and then switch the live stream format in the add-on, it uses the original format once again (the one before the switch) and uses the format I currently have set the next time.

This is likely because the list is created using the settings that have already been set. When changing the settings, the list needs to be re-created to use the new settings.

I'm not quite sure what the difference is between the four types? Any advantages or disadvantages? It seems to me that using MPEG-DASH works well.

HLS - uses ffmpeg with the main stream which can be slow to load and has limited stream selection options
Multistream HLS - uses ffmpeg with all available variant streams and is even slower to load as a result
Adaptive HLS - uses InputStream.Adaptive to load the HLS manifest
MPEG-DASH - uses InputStream.Adaptive to load the MPEG-DASH manifest

Which is better depends on what YouTube provides, what this plugin can extract, and what InputStream.Adaptive supports. Currently the available MPEG-DASH live streams are very low quality, however older versions of InputStream.Adaptive don't fully support the HLS streams.

Where possible using InputStream.Adaptive will provide the best experience, and currently the HLS streams offer the best quality, so Adaptive HLS is the current default selection depending on Kodi version.

[MoojMidge]

@Liqianyu - can you test if the following test version allows both Kodi and custom proxy settings to work?

https://github.com/MoojMidge/plugin.video.youtube/archive/refs/heads/master.zip

Options available in Settings > Advanced > Internet > HTTP proxy

[Liqianyu]

@MoojMidge
Here's a quick test that seems to work well. Of course I didn't test all proxy protocols.
If the PySocks dependency is not installed manually, the YouTube add-on just reports an error when Kodi sets up the Socks proxy.

It's worth noting that if you use a custom proxy. Neither the video nor the manifest will be proxied, which is expected right?
If the Kodi proxy is enabled and the HTTP proxy is set to disabled within the add-on. I don't observe manifest traffic, but I do see video traffic (*.googlevideo.com).

[Liqianyu]

Which is better depends on what YouTube provides, what this plugin can extract, and what InputStream.Adaptive supports. Currently the available MPEG-DASH live streams are very low quality, however older versions of InputStream.Adaptive don't fully support the HLS streams.

Where possible using InputStream.Adaptive will provide the best experience, and currently the HLS streams offer the best quality, so Adaptive HLS is the current default selection depending on Kodi version.

So when using HLS with Multistream HLS, it won't be used even if the InputStream.Adaptive option is on, right?
After testing it looks like Adaptive HLS is indeed the best option available. Has the best picture quality and loading speed in most cases.
MPEG-DASH is an open source protocol, maybe more scalability in the future? And HLS with Multistream HLS might be suitable for lower versions of Kodi\ISA?It seems that there are some compatibility issues with FFmpeg in the latest Kodi versions and some platforms.

[MoojMidge]

@Liqianyu - As I mentioned before, any proxy settings in this plugin will only be of use for browsing within the plugin.

It will have no effect on downloading or playing anything, either for manifests or video segments. That is all handled by Kodi itself. If you need to use a proxy to watch the videos, then this will need to be set in the Kodi proxy settings.

Why that sometimes appears to work, and sometimes doesn't, can only be determined by people providing logs of both scenarios, but at this point I don't intend to look into this any further as there wasn't much input aside from yourself, even if it doesn't really impact you. It is outside the scope of this plugin anyway.

If the PySocks dependency is not installed manually, the YouTube add-on just reports an error when Kodi sets up the Socks proxy.

Yes, this is a problem that I will need to look into further. Will need to trigger the PySocks installation manually if a socks proxy is used.

So when using HLS with Multistream HLS, it won't be used even if the InputStream.Adaptive option is on, right?
After testing it looks like Adaptive HLS is indeed the best option available. Has the best picture quality and loading speed in most cases.
MPEG-DASH is an open source protocol, maybe more scalability in the future? And HLS with Multistream HLS might be suitable for lower versions of Kodi\ISA?It seems that there are some compatibility issues with FFmpeg in the latest Kodi versions and some platforms.

Yes, that is a reasonable summary.

[MoojMidge]

Prompt for pysocks installation added in https://github.com/MoojMidge/plugin.video.youtube/archive/refs/heads/master.zip