TVP extractor redacted for new url schema and extract method #32028

bibiak1 · 2023-04-11T13:07:20Z

Please follow the guide below

You will be asked some questions, please read them carefully and answer honestly
Put an x into all the boxes [ ] relevant to your pull request (like that [x])
Use Preview tab to see how your pull request will actually look like

Before submitting a pull request make sure you have:

Searched the bugtracker for similar pull requests
Read adding new extractor tutorial
Read youtube-dl coding conventions and adjusted the code to meet them
Covered the code with tests (note that PRs without tests will be REJECTED)
Checked the code with flake8

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

I am the original author of this code and I am willing to release it under Unlicense
Where I am not the original author of this code it was released under the same terms at https://github.com/yt-dlp.

What is the purpose of your pull request?

Bug fix
Improvement
New extractor
New feature

Description of your pull request and other information

This PR switches VOD extraction to new APIs.

The original proposal added an extractor TVappIE. After alignment with the yt-dlp extractor from yt-dlp/yt-dlp#6989, there are now two new extractors, TVPStreamIE and TVPVODVideoIE, plus TVPVODSeriesIE replacing TVPWebsiteIE: thanks @selfisekai.

These changes have been applied over the aligned code:

use traverse_obj() for safer extraction
fix tests that are not blocked from UK.

Compared with the yt-dlp extractor,

episode number 0 in site metadata is treated as missing (None), since it seems to apply to one-off shows
as the yt-dl download test harness now supports test cases with variable ids, a real test has been added for TVPStreamIE.

dirkf

Thanks for your work !

~~I've made a few suggestions and will enable the CI test once you've had time to address them.~~

youtube_dl/extractor/extractors.py

youtube_dl/extractor/tvp.py

selfisekai · 2023-05-04T21:20:15Z

what we do in yt-dlp is, we take the externalUid and use it with the embed extractor, which has much less strict geo-blocking.
https://github.com/yt-dlp/yt-dlp/blob/master/yt_dlp/extractor/tvp.py#L492-L493

dirkf · 2023-05-04T21:32:55Z

But see yt-dlp/yt-dlp#6987, where the external ID just gives (in Polish) "payment required".

@bibiak1, I should also have asked you to complete the "Explanation", especially since there's no related issue. Is this a new scheme or an alternative as suggested above? If the latter, it should be built into the existing TVPIE.

bibiak1 · 2023-05-06T22:30:04Z

It's a news scheme. I have checked unpayed content only. I can try to implement log-in as I was not focusing on that

dirkf · 2023-05-07T13:42:41Z

Let's back-port the yt-dlp version that was just committed using the new scheme. I'd like try to keep the two as similar as possible. If you fancy doing that please go ahead.

The diff from yt-dlp should apply straightforwardly to the base yt-dl code; then any Python3-only syntax (yield from ... -> for from_ in ...:; yield from_, f'formatted text with {value}' ->, f'formatted text with {0}'.format(value), str -> compat_str, etc) in the new code has to be tweaked; then see if there are any features of the existing PR that should be added.

[skip ci]

* pull changes from yt-dlp/yt-dlp#6989, thanks selfisekai * use `traverse_obj()` for safer extraction * fix tests that are not blocked from UK Co-authored-by: selfisekai

youtube_dl/extractor/tvp.py

Add `txt_or_none()` shim

youtube_dl/extractor/tvp.py

dirkf · 2023-05-09T16:35:42Z

@bibiak1, I've merged the yt-dlp changes. If you're able to check (and fix if necessary) the geo-blocked tests, that would be great.

Let me know if you want to add login code; or you could make a separate PR?

selfisekai · 2023-05-09T17:32:43Z

hmmmm. tested from NL. turns out https://vod.tvp.pl/filmy-dokumentalne,163/krzysztof-krawczyk--cale-moje-zycie,332512 plays in browser if logged in to a free account. the geoblock error message I'm getting when not logged in is this:

Z powodów licencyjnych materiał jest niedostępny w Twoim kraju.

Przepisy UE pozwalają abonentom, którzy w kraju zamieszkania (w tym przypadku w PL) zakupili pakiet TVP + utrzymać dostęp do materiałów wchodzących w skład pakietu, w czasie wizyty w innym kraju UE.

so I guess they wanted to implement this but someone made an assumption that an account logged in means SVOD and not logged in means AVOD? login is by cookies so --cookies-from-browser firefox worked for me on yt-dlp master

dirkf · 2023-05-09T18:42:46Z

From UK, the extractor just sees the truthy .isGeoBlocked member of the API result. From experience, if that is ignored, the stream URLs give the special video that says "Video not available in your region".

selfisekai · 2023-05-09T20:19:22Z

that's on embed (tvplayer2 api), this workaround is specific to VOD playlist api

dirkf · 2023-05-10T03:16:26Z

When I run that URL in the current PR, it's handled by TVPVODVideoIE and then handed off to TVPEmbedIE using the externalUid, where the geo-block is detected.

The data from the VOD API includes these values:

...
'countries': [{'id': 92, 'key': 'polska', 'name': 'Polska'}],
...
'geoipManualLock': False,
...
'loginRequired': False,
...

The test case https://vod.tvp.pl/website/krzysztof-krawczyk-cale-moje-zycie,51374466 is redirected via TVPIE to 332512 with the same result.

youtube_dl/extractor/tvp.py

dirkf · 2023-05-10T22:19:40Z

youtube_dl/extractor/tvp.py

+            elif fatal:
+                raise RegexNotFoundError('Unable to extract %s' % _name)
+            else:
+                self._downloader.report_warning('unable to extract %s' % _name + bug_reports_message())


Suggested change

self._downloader.report_warning('unable to extract %s' % _name + bug_reports_message())

self.report_warning('unable to extract %s' % _name + bug_reports_message())

bibiak1 · 2023-05-15T14:37:28Z

It works from PL without logging in. Most likely it's geo location oriented. If you are not from PL IP address then you need to login.

I read all comments and do not know what i suppose to do. tvp.py code starting line 255 (class TVPappIE) is mine. I've just added support for new schema/player and didn't want to change anything that exist before.

Co-authored-by: dirkf <fieldhouse@gmx.net>

bibiak1

Workflow tested. All OK

bibiak1 · 2023-06-19T19:40:44Z

@dirkf can you merge this pull req?

pawisoon · 2023-08-10T22:02:45Z

@dirkf can this be reviewed and merged in? 🙏

bibiak1

Please proceed with merge

Marcin Biczan and others added 6 commits April 5, 2023 21:54

TVPapp extractor added :: init

953fce8

json uneeded

b08dc56

added support for full offer

4b6bef4

Merge branch 'ytdl-org:master' into master

16cb050

Merge branch 'ytdl-org:master' into master

37ff4c9

Merge branch 'ytdl-org:master' into master

93e0d82

dirkf requested changes May 3, 2023

View reviewed changes

dirkf mentioned this pull request May 4, 2023

[tvp] "Video requires payment and log-in, but log-in is not implemented" yt-dlp/yt-dlp#6987

Closed

11 tasks

dirkf reviewed May 4, 2023

View reviewed changes

youtube_dl/extractor/tvp.py Outdated Show resolved Hide resolved

This comment was marked as outdated.

Sign in to view

dirkf added 2 commits May 9, 2023 16:34

Update extractors.py

efedc80

[skip ci]

Update tvp.py from yt-dlp

6e82711

* pull changes from yt-dlp/yt-dlp#6989, thanks selfisekai * use `traverse_obj()` for safer extraction * fix tests that are not blocked from UK Co-authored-by: selfisekai

dirkf reviewed May 9, 2023

View reviewed changes

youtube_dl/extractor/tvp.py Show resolved Hide resolved

Update youtube_dl/extractor/tvp.py

4891480

Add `txt_or_none()` shim

dirkf reviewed May 9, 2023

View reviewed changes

youtube_dl/extractor/tvp.py Outdated Show resolved Hide resolved

Update youtube_dl/extractor/tvp.py

237c59f

dirkf mentioned this pull request May 10, 2023

[extractor/tvp] better hints for geoblock yt-dlp/yt-dlp#7023

Closed

10 tasks

dirkf reviewed May 10, 2023

View reviewed changes

bibiak1 and others added 2 commits May 22, 2023 20:53

Merge branch 'ytdl-org:master' into master

1d148eb

Update youtube_dl/extractor/tvp.py

56c0723

Co-authored-by: dirkf <fieldhouse@gmx.net>

bibiak1 and others added 6 commits May 22, 2023 21:21

Update youtube_dl/extractor/tvp.py

283b6b3

Co-authored-by: dirkf <fieldhouse@gmx.net>

Update youtube_dl/extractor/tvp.py

578c533

Co-authored-by: dirkf <fieldhouse@gmx.net>

Update youtube_dl/extractor/tvp.py

da19699

Co-authored-by: dirkf <fieldhouse@gmx.net>

Update youtube_dl/extractor/tvp.py

79b0cde

Co-authored-by: dirkf <fieldhouse@gmx.net>

Update youtube_dl/extractor/tvp.py

e7c4239

Co-authored-by: dirkf <fieldhouse@gmx.net>

too many except

8a2249e

bibiak1 commented May 22, 2023

View reviewed changes

bibiak1 requested a review from dirkf May 22, 2023 19:37

bibiak1 added 2 commits June 13, 2023 07:10

Merge branch 'ytdl-org:master' into master

7caa31f

Merge branch 'ytdl-org:master' into master

9f3bbdd

Merge branch 'ytdl-org:master' into master

a922336

selfisekai mentioned this pull request Aug 10, 2023

pilot.wp.pl support #26267

Open

5 tasks

pawisoon approved these changes Aug 10, 2023

View reviewed changes

bibiak1 added 2 commits August 31, 2023 15:14

Merge branch 'ytdl-org:master' into master

0c53d42

Merge branch 'ytdl-org:master' into master

8aac6a6

bibiak1 commented Oct 2, 2023

View reviewed changes

bibiak1 and others added 3 commits October 2, 2023 15:51

Merge branch 'ytdl-org:master' into master

efee229

moved txt_or_none outside if statement

f5dd875

Merge branch 'ytdl-org:master' into master

8e6e47d

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

TVP extractor redacted for new url schema and extract method #32028

TVP extractor redacted for new url schema and extract method #32028

bibiak1 commented Apr 11, 2023 •

edited by dirkf

Loading

dirkf left a comment •

edited

Loading

selfisekai commented May 4, 2023

dirkf commented May 4, 2023

bibiak1 commented May 6, 2023 •

edited

Loading

dirkf commented May 7, 2023

This comment was marked as outdated.

dirkf commented May 9, 2023

selfisekai commented May 9, 2023

dirkf commented May 9, 2023

selfisekai commented May 9, 2023

dirkf commented May 10, 2023

dirkf May 10, 2023

bibiak1 commented May 15, 2023 •

edited

Loading

bibiak1 left a comment

bibiak1 commented Jun 19, 2023

pawisoon commented Aug 10, 2023

bibiak1 left a comment

	self._downloader.report_warning('unable to extract %s' % _name + bug_reports_message())
	self.report_warning('unable to extract %s' % _name + bug_reports_message())

TVP extractor redacted for new url schema and extract method #32028

Are you sure you want to change the base?

TVP extractor redacted for new url schema and extract method #32028

Conversation

bibiak1 commented Apr 11, 2023 • edited by dirkf Loading

Please follow the guide below

Before submitting a pull request make sure you have:

In order to be accepted and merged into youtube-dl each piece of code must be in public domain or released under Unlicense. Check one of the following options:

What is the purpose of your pull request?

Description of your pull request and other information

dirkf left a comment • edited Loading

Choose a reason for hiding this comment

selfisekai commented May 4, 2023

dirkf commented May 4, 2023

bibiak1 commented May 6, 2023 • edited Loading

dirkf commented May 7, 2023

This comment was marked as outdated.

dirkf commented May 9, 2023

selfisekai commented May 9, 2023

dirkf commented May 9, 2023

selfisekai commented May 9, 2023

dirkf commented May 10, 2023

dirkf May 10, 2023

Choose a reason for hiding this comment

bibiak1 commented May 15, 2023 • edited Loading

bibiak1 left a comment

Choose a reason for hiding this comment

bibiak1 commented Jun 19, 2023

pawisoon commented Aug 10, 2023

bibiak1 left a comment

Choose a reason for hiding this comment

bibiak1 commented Apr 11, 2023 •

edited by dirkf

Loading

dirkf left a comment •

edited

Loading

bibiak1 commented May 6, 2023 •

edited

Loading

bibiak1 commented May 15, 2023 •

edited

Loading