disablePictureInPicture interoperability

[mounirlamouri]

We should start looking at what are the issues we need to resolve in order to publish our CR. @beaufortfrancois could you triage the issues that are vNext (thinking of element-based Picture-in-Picture), and start gathering folks around the ones that need to be solved for CR?

[mounirlamouri]

FWIW, I think we should resolve the naming for the pseudo-class but on top of my head, this may be the only critical change we would have to do.

[mounirlamouri]

Is there any issue/feature that the group would like to see added/fixed in order to go to CR?

[kened-ferreira]

Are you planning to allow multiple picture-in-picture?

[mounirlamouri]

I don't expect the specification to allow that given that every system I'm aware of with native picture-in-picture do not allow multiple picture-in-picture.

[kened-ferreira]

It would be really useful. I have to use multiple browsers simultaneously to have multiple pips.

[kened-ferreira]

I don't expect the specification to allow that given that every system I'm aware of with native picture-in-picture do not allow multiple picture-in-picture.

The limitation you mentioned refers to each instance of the app. I can run, for example, two instances of vlc (or bsplayer), so I can have two pip. What if each browser tab is treated as an instance? I think the limitation imposed by the OS would be overcome.

[tidoust]

Are you planning to allow multiple picture-in-picture?

For reference, I note that @kened-ferreira raised this in #183.

[beaufortfrancois]

@mounirlamouri I've triaged all issues and PRs for Picture-in-Picture.

Have a look at what's left at https://github.com/w3c/picture-in-picture/issues?q=-label%3Av2+is%3Aopen+

[beaufortfrancois]

@jernoble I've noticed some web platform tests for Picture-in-Picture are failing:
https://wpt.fyi/results/picture-in-picture?label=master&label=experimental&product=safari&q=status%3A%21pass%20and%20status%3A%21ok

Here are my notes:

• disablePictureInPicture is present in the IDL but not actually implemented. It should be pretty easy to add.
• A video whose src is the result of canvas.captureStream() does not update its ready_state, causing some mediastream related wpt tests to fail.
• Requesting Picture-in-Picture should consumes user gesture, it does not. Intended behaviour?

[jernoble]

• disablePictureInPicture is present in the IDL but not actually implemented. It should be pretty easy to add.

This is intentional. As I noted when this property was proposed, WebKit‘s position is that sites should not be allowed to block the user from entering picture-in-picture if the user so chooses. We don’t plan on implementing this.

• A video whose src is the result of canvas.captureStream() does not update its ready_state, causing some mediastream related wpt tests to fail.

Ok, we’ll look into this.

• Requesting Picture-in-Picture should consumes user gesture, it does not. Intended behaviour?

WebKit hasn’t implemented the User Activation API, where “consuming a user gesture” is defined. If there’s a failing test here, it should probably be a part of the activation test suite, and not PiP specifically.

[mounirlamouri]

• disablePictureInPicture is present in the IDL but not actually implemented. It should be pretty easy to add.

This is intentional. As I noted when this property was proposed, WebKit‘s position is that sites should not be allowed to block the user from entering picture-in-picture if the user so chooses. We don’t plan on implementing this.

@tidoust what are the options in case like this where we have two browsers incompatibilities? AFAIK, we could mark this feature as OPTIONAL, but the IDL will still be there so not having the property would be incorrect. The alternative would be to drop the feature from the spec entirely. Is that correct?

• Requesting Picture-in-Picture should consumes user gesture, it does not. Intended behaviour?

WebKit hasn’t implemented the User Activation API, where “consuming a user gesture” is defined. If there’s a failing test here, it should probably be a part of the activation test suite, and not PiP specifically.

https://html.spec.whatwg.org/multipage/interaction.html#consume-user-activation

[tidoust]

@tidoust what are the options in case like this where we have two browsers incompatibilities? AFAIK, we could mark this feature as OPTIONAL, but the IDL will still be there so not having the property would be incorrect. The alternative would be to drop the feature from the spec entirely. Is that correct?

That is correct. Just to clarify, with OPTIONAL, you mean making the effect of the IDL attribute optional, not making the IDL attribute itself optional, right?

(Making the IDL attribute optional seems a bad idea in any case, as that would require web applications to check on the attribute's presence before they set it.)

Setting the disablePictureInPicture attribute may already have no effect given that the spec uses a SHOULD. If the specification gets completed to note that a user agent may choose not to honor the attribute's value at all, would Webkit be amenable to expose the (essentially inert) IDL attribute, @jernoble ?

In any case, from a process perspective, it may be wise to mark the feature "at risk" (see CR section in Process document) if it remains in the spec. If WebKit exposes the attribute without supporting its optional behavior, I believe that we would still need two independent implementations that actually disable PiP when the attribute is set.

[marcoscaceres]

That is correct. Just to clarify, with OPTIONAL, you mean making the effect of the IDL attribute optional, not making the IDL attribute itself optional, right?

Making the IDL attribute is not ideal. We should leave it (expecting that it's a hint), and expect it to reflect the HTML attribute.

Setting the disablePictureInPicture attribute may already have no effect given that the spec uses a SHOULD. If the specification gets completed to note that a user agent may choose not to honor the attribute's value at all, would Webkit be amenable to expose the (essentially inert) IDL attribute, @jernoble ?

Not @jernoble, but from the Gecko side, I think we would be happy for the attribute to go away entirely. If it remains, I imagine Firefox would still allow users to bring up a context menu, and select "PiP it!" on any video. I feel bad that we will be lying to developers (or that developers may start doing annoying things, like putting divs on-top of videos to prevent the right context menu from appearing), but users should be able to access this feature irrespective of what the site wants.

In any case, from a process perspective, it may be wise to mark the feature "at risk" (see CR section in Process document) if it remains in the spec. If WebKit exposes the attribute without supporting its optional behavior, I believe that we would still need two independent implementations that actually disable PiP when the attribute is set.

Agree. I think it might be good to just mark this "at risk" for now.

[jernoble]

I feel bad that we will be lying to developers (or that developers may start doing annoying things, like putting divs on-top of videos to prevent the right context menu from appearing), but users should be able to access this feature irrespective of what the site wants.

I strongly agree. Sites who implement custom playback controls can (and do) already make it difficult to enter PiP. Enforcing this user-hostile behavior at the User-Agent level should be a non-goal, IMO.

[mounirlamouri]

Dropping the agenda label as this was discussed at the previous meeting. If you all think it should be discussed again, feel free to add the label back.

[mounirlamouri]

I strongly agree. Sites who implement custom playback controls can (and do) already make it difficult to enter PiP. Enforcing this user-hostile behavior at the User-Agent level should be a non-goal, IMO.

FWIW, we've added that because we received feedback from developers that they wanted to prevent some videos to be played in Picture-in-Picture. Because the API would allow a website to detect that a video has entered PIP and the website can then call exitPictureInPicture(), it seemed that we would either have to play a cat and mouse game that would hurt the platform in some ways like not exposing that the video is in PIP or hurt the user by offering PIP when the website would leave PIP immediately. Chrome offers Picture-in-Picture in 3 different places in the UI, we want to make sure that if the user triggers PIP, it works.

[marcoscaceres]

Because the API would allow a website to detect that a video has entered PIP and the website can then call exitPictureInPicture()

I wonder then if "enterpictureinpicture" should only fire as a result of .requestPictureInPicture()?

Then .exitPictureInPicture() should only work on site initiated .requestPictureInPicture(), instead of user initiated ones. The sites shouldn't know when the user puts a video in PiP mode.

[mounirlamouri]

I do not think we should keep the websites in the dark about what's happening. This would be harmful to the platform because it would have a lot of side effects. One on top of my head: we are exposing the size of the PIP window so websites can do ABR and now I website may send a full HD video to a tiny PIP video, wasting resources on the client and server side. I'm sure we could find others.

[jernoble]

Then .exitPictureInPicture() should only work on site initiated .requestPictureInPicture(), instead of user initiated ones. The sites shouldn't know when the user puts a video in PiP mode.

An alternative would be to require exitPictureInPicture() to be called from a user gesture, which ensures that "exit PiP" is an action performed by the user.

[mounirlamouri]

That would have a lower cost but would have still a cost and possible backward compatible issues. To be clear, these are the type of things I wanted to avoid by simply letting websites have control. If Firefox and Safari see problems with websites actively blocking PIP sessions triggered by the UA, maybe we should start looking at solutions at that point. There will be no good solutions but only bandaid that will make the problem less painful and here taking a move to block abuse is important and preempting moves may lead to a worse result faster. For example, exitPictureInPicture() requiring a user gesture has a cost in backward compat, probably a cost for some reasonable usage but most importantly, I'm sure Chrome isn't the only UA to auto-exit Picture-in-Picture in some cases. A website could find one of these cases to exit Picture-in-Picture to workaround the limitation (in the case, of Chrome scr='' exits PIP, I believe Safari exits PIP when the element is removed from the DOM).

[eric-carlson]

There will be no good solutions but only bandaid that will make the problem less painful and here taking a move to block abuse is important and preempting moves may lead to a worse result faster. For example, exitPictureInPicture() requiring a user gesture has a cost in backward compat, probably a cost for some reasonable usage but most importantly, I'm sure Chrome isn't the only UA to auto-exit Picture-in-Picture in some cases. A website could find one of these cases to exit Picture-in-Picture to workaround the limitation (in the case, of Chrome scr='' exits PIP, I believe Safari exits PIP when the element is removed from the DOM).

While we can't prevent sites from being user hostile, but we should not make it easier.

[marcoscaceres]

Requiring a gesture to exit seems like a reasonable compromise. If sites do things like setting src="" or removing the video from the DOM, they may end up compromising their own user experience (but that's kinda on them). So I agree with @eric-carlson, we shouldn't make it easier and should encourage sites to the right thing by designing the API in a way that enforces best practice.

[marcoscaceres]

Just to follow up, we discussed this internally in Mozilla a little bit... if we end up implementing, this is what we will likely do: if the user initiated PiP, the site won't be notified at all that it's in PiP mode. However, if the user enables PiP through the site's UI, then events will fire etc. Requiring a gesture to also exit still seems like a good idea.

[beaufortfrancois]

Any updates on this @jernoble?

• A video whose src is the result of canvas.captureStream() does not update its ready_state, causing some mediastream related wpt tests to fail.

Ok, we’ll look into this.

[pliu6]

We are working on it and will fix it in a future release.

By the way, I think the issue is not directly related to the picture-in-picture implementation, and we can fix (work around) the test failure by changing the test.

Any updates on this @jernoble?

• A video whose src is the result of canvas.captureStream() does not update its ready_state, causing some mediastream related wpt tests to fail.

Ok, we’ll look into this.

[beaufortfrancois]

By the way, I think the issue is not directly related to the picture-in-picture implementation, and we can fix (work around) the test failure by changing the test.

Feel free to update tests at https://github.com/web-platform-tests/wpt/tree/master/picture-in-picture. I'll be happy to review them.

[beaufortfrancois]

This is intentional. As I noted when this property was proposed, WebKit‘s position is that sites should not be allowed to block the user from entering picture-in-picture if the user so chooses. We don’t plan on implementing this.

For info, this causes some real UX issues as websites stop Picture-in-Picture after entering it to disallow it. See https://twitter.com/firt/status/1307083085666750468 and https://www.theverge.com/2020/9/18/21445912/youtube-videos-website-picture-in-picture-ios-14-google-apple-premium

cc @jernoble

[jernoble]

And that UX issue is entirely the fault of the web site, and not the user or the user agent. Blocking Picture-in-Picture is no less user hostile than breaking Picture-in-Picture; it just makes the user agent culpable in that hostility.

[eric-carlson]

For info, this causes some real UX issues as websites stop Picture-in-Picture after entering it to disallow it. See https://twitter.com/firt/status/1307083085666750468 and https://www.theverge.com/2020/9/18/21445912/youtube-videos-website-picture-in-picture-ios-14-google-apple-premium

"Websites do terrible things, so lets add API to make it easier"?

[beaufortfrancois]

1. The disablepictureinpicture HTMLVideoElement attribute is a hint for the user agent that the website communicates to basically say this video is not meant to go into Picture-in-Picture. 360 videos for instance are one example where it's not suitable for the user to see the "raw" video.

2. The disableremoteplayback HTMLMediaElement attribute in the Remote Playback API is implemented in WebKit and is quite similar.

[edwardchanjw]

Do we wanna to have alternative API, Is the PIP is enable/disable on the page etc.
For example:
isPictureInPictureActive

Potentially for YouTube platform, I am notice YouTube might wanna to ask User after less than hour of listening, Would hope if any customized good video playing app (PIP type) potentially created for multi-"tasking entertainment", would allow the user to have more inactive hours.

Video paused. Continue watching
Could be an API for check above and One PIP Windows only?

[chrisn]

Revisiting this issue, as I'd like us to move the spec forward to CR if we can. Reading through the discussion here, there isn't consensus on disablePictureInPicture, but we haven't come to a resolution. Options discussed:

• Look further at whether disablePictureInPicture is a good idea
• Spec disablePictureInPicture as OPTIONAL (agreed to be undesirable)
• Spec disablePictureInPicture as a SHOULD (what we have now)
• Mark disablePictureInPicture as at-risk, then eventually remove from the spec if we don't have multi-implementer support

Has anything changed since this was last discussed?

I recommend doing another triage for newer issues raised also contacting Mozilla for update on their position.

cc @beaufortfrancois @jernoble

[chrisn]

As the discussion has focused on disablePictureInPicture, I'll rename this issue and raise a new one for the steps to publish CR.

[chrisn]

There's a spec inconsistency:
https://w3c.github.io/picture-in-picture/#request-pip says OPTIONALLY (which we should rephrase as MAY), whereas https://w3c.github.io/picture-in-picture/#disable-pip says if the attribute is added to the video element, the user agent SHOULD ..., which could be a MAY.

With these changes, we don't think we need to mark this feature as at-risk.