Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Set starting send bitrate to 1000kbps to reduce time of low resolution at start of connection #2987

Merged
merged 2 commits into from
Nov 6, 2024
Merged

Set starting send bitrate to 1000kbps to reduce time of low resolution at start of connection #2987

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
37d539c
Set starting send bitrate to 1000kbps to reduce time of low resolutio…
hensmi-amazon Nov 5, 2024
eddc8c4
Merge branch 'main' into start-br
hensmi-amazon Nov 5, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
16 changes: 3 additions & 13 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,32 +5,22 @@ All notable changes to this project will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

## [Unreleased]
## [3.26.0] - 2024-10-07
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should still be Unreleased?


### Added

### Removed

### Changed

- Set starting send bitrate to 1000kbps to reduce time of low resolution at start of connection
- Add per-frame encoding time in video encoding CPU connection health policy

### Fixed

- Prevent DataMessage callback errors from killing a meeting

## [3.26.0] - 2024-10-07

### Added

### Removed

### Changed

### Fixed

- Avoid breaking audio input when external devices are disconnected on iOS browsers when using Web Audio by suspending and resuming the audio context in that case.
- Fixed incoming audio loss calculation when server side network adaption and redundant audio features are running together.
- Prevent DataMessage callback errors from killing a meeting

## [3.25.0] - 2024-09-10

Expand Down
8 changes: 4 additions & 4 deletions docs/classes/setlocaldescriptiontask.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -123,7 +123,7 @@ <h3>constructor</h3>
<aside class="tsd-sources">
<p>Overrides <a href="basetask.html">BaseTask</a>.<a href="basetask.html#constructor">constructor</a></p>
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L16">src/task/SetLocalDescriptionTask.ts:16</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L15">src/task/SetLocalDescriptionTask.ts:15</a></li>
</ul>
</aside>
<h4 class="tsd-parameters-title">Parameters</h4>
Expand Down Expand Up @@ -154,7 +154,7 @@ <h3><span class="tsd-flag ts-flagProtected">Protected</span> task<wbr>Name</h3>
<aside class="tsd-sources">
<p>Overrides <a href="basetask.html">BaseTask</a>.<a href="basetask.html#taskname">taskName</a></p>
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L14">src/task/SetLocalDescriptionTask.ts:14</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L13">src/task/SetLocalDescriptionTask.ts:13</a></li>
</ul>
</aside>
</section>
Expand All @@ -172,7 +172,7 @@ <h3>cancel</h3>
<aside class="tsd-sources">
<p>Overrides <a href="basetask.html">BaseTask</a>.<a href="basetask.html#cancel">cancel</a></p>
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L22">src/task/SetLocalDescriptionTask.ts:22</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L21">src/task/SetLocalDescriptionTask.ts:21</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
Expand Down Expand Up @@ -288,7 +288,7 @@ <h3>run</h3>
<aside class="tsd-sources">
<p>Overrides <a href="basetask.html">BaseTask</a>.<a href="basetask.html#run">run</a></p>
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L32">src/task/SetLocalDescriptionTask.ts:32</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/task/SetLocalDescriptionTask.ts#L31">src/task/SetLocalDescriptionTask.ts:31</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
Expand Down
29 changes: 15 additions & 14 deletions docs/classes/videocodeccapability.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -215,14 +215,15 @@ <h3>fmtp<wbr>Line<wbr>Matches</h3>
<li class="tsd-description">
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L37">src/sdp/VideoCodecCapability.ts:37</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L50">src/sdp/VideoCodecCapability.ts:50</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
<div class="lead">
<p>Returns whether the codec capability fmtp line matches. This will not
attempt to match H.264 profile levels (e.g. 5.2, 3.1), see internal comments for
more detailed information.</p>
more detailed information. This function takes care of checking for an
expected payload type as well.</p>
</div>
<p>This function is meant to only be used internally.</p>
</div>
Expand All @@ -249,7 +250,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> av1</h3>
<li class="tsd-description">
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L194">src/sdp/VideoCodecCapability.ts:194</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L221">src/sdp/VideoCodecCapability.ts:221</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
Expand All @@ -271,7 +272,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> av1<wbr>Main</h3>
<li class="tsd-description">
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L184">src/sdp/VideoCodecCapability.ts:184</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L211">src/sdp/VideoCodecCapability.ts:211</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
Expand All @@ -293,7 +294,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> from<wbr>Signaled</h3>
<li class="tsd-description">
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L201">src/sdp/VideoCodecCapability.ts:201</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L228">src/sdp/VideoCodecCapability.ts:228</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
Expand Down Expand Up @@ -321,7 +322,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> h264</h3>
<li class="tsd-description">
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L158">src/sdp/VideoCodecCapability.ts:158</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L185">src/sdp/VideoCodecCapability.ts:185</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
Expand All @@ -343,7 +344,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> h264<wbr>Baseline<wbr>Pro
<li class="tsd-description">
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L81">src/sdp/VideoCodecCapability.ts:81</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L108">src/sdp/VideoCodecCapability.ts:108</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
Expand All @@ -367,7 +368,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> h264<wbr>Constrained<wbr>
<li class="tsd-description">
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L95">src/sdp/VideoCodecCapability.ts:95</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L122">src/sdp/VideoCodecCapability.ts:122</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
Expand All @@ -391,7 +392,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> h264<wbr>Constrained<wbr>
<li class="tsd-description">
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L147">src/sdp/VideoCodecCapability.ts:147</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L174">src/sdp/VideoCodecCapability.ts:174</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
Expand All @@ -415,7 +416,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> h264<wbr>High<wbr>Profile
<li class="tsd-description">
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L133">src/sdp/VideoCodecCapability.ts:133</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L160">src/sdp/VideoCodecCapability.ts:160</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
Expand Down Expand Up @@ -446,7 +447,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> h264<wbr>Main<wbr>Profile
<li class="tsd-description">
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L110">src/sdp/VideoCodecCapability.ts:110</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L137">src/sdp/VideoCodecCapability.ts:137</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
Expand All @@ -471,7 +472,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> vp8</h3>
<li class="tsd-description">
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L68">src/sdp/VideoCodecCapability.ts:68</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L95">src/sdp/VideoCodecCapability.ts:95</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
Expand All @@ -493,7 +494,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> vp9</h3>
<li class="tsd-description">
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L177">src/sdp/VideoCodecCapability.ts:177</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L204">src/sdp/VideoCodecCapability.ts:204</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
Expand All @@ -515,7 +516,7 @@ <h3><span class="tsd-flag ts-flagStatic">Static</span> vp9<wbr>Profile0</h3>
<li class="tsd-description">
<aside class="tsd-sources">
<ul>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L166">src/sdp/VideoCodecCapability.ts:166</a></li>
<li>Defined in <a href="https://github.com/aws/amazon-chime-sdk-js/blob/main/src/sdp/VideoCodecCapability.ts#L193">src/sdp/VideoCodecCapability.ts:193</a></li>
</ul>
</aside>
<div class="tsd-comment tsd-typography">
Expand Down
5 changes: 3 additions & 2 deletions src/sdp/SDP.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -927,8 +927,9 @@ export default class SDP {
// `x-google-start-bitrate` is an unofficial flag that has existed in libwebrtc since its release and is unlikely
// to be removed without notification.
//
// libwebrtc reference: https://webrtc.googlesource.com/src/+/b6ef1a736ee94d97cc28f3bd59b826c716a3278f/media/base/media_constants.cc#97
const newLine = line + `;x-google-start-bitrate=${bitrateKbps * 1000}`;
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

oopsie daisy

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wow, wonder how this would have worked so far with * 1000 already added.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It just uses the max bitrate configured, so nothing terrible.

// libwebrtc constant: https://webrtc.googlesource.com/src/+/b6ef1a736ee94d97cc28f3bd59b826c716a3278f/media/base/media_constants.cc#97
// libwebrtc parsing: https://webrtc.googlesource.com/src/+/61c5e86dca59b0ac8240444e7df5c31da27ee40f/media/engine/webrtc_media_engine.cc#172
const newLine = line + `;x-google-start-bitrate=${bitrateKbps}`;
dstLines.push(newLine);
} else {
dstLines.push(line);
Expand Down
65 changes: 46 additions & 19 deletions src/sdp/VideoCodecCapability.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,19 +18,32 @@ export default class VideoCodecCapability implements Eq {
) {}

equals(other: this): boolean {
return (
other !== undefined &&
this.codecName === other.codecName &&
this.codecCapability.mimeType === other.codecCapability.mimeType &&
this.codecCapability.clockRate === other.codecCapability.clockRate &&
this.codecCapability.sdpFmtpLine === other.codecCapability.sdpFmtpLine
);
if (other === undefined) {
return false;
}

if (
this.codecName !== other.codecName ||
this.codecCapability.mimeType !== other.codecCapability.mimeType ||
this.codecCapability.clockRate !== other.codecCapability.clockRate
) {
return false;
}

const thisFmtpLine = this.codecCapability.sdpFmtpLine
? this.cleanupFmtpLine(this.codecCapability.sdpFmtpLine, this.codecName)
: '';
const otherFmtpLine = other.codecCapability.sdpFmtpLine
? this.cleanupFmtpLine(other.codecCapability.sdpFmtpLine, other.codecName)
: '';
return thisFmtpLine === otherFmtpLine;
}

/**
* Returns whether the codec capability fmtp line matches. This will not
* attempt to match H.264 profile levels (e.g. 5.2, 3.1), see internal comments for
* more detailed information.
* more detailed information. This function takes care of checking for an
* expected payload type as well.
*
* This function is meant to only be used internally.
*/
Expand All @@ -40,7 +53,28 @@ export default class VideoCodecCapability implements Eq {
return false;
}

if (this.codecName === 'H264') {
const prefix = `a=fmtp:${expectedPayloadType} `;
if (!line.startsWith(prefix)) {
return false;
}
const fmtpLineFromSdp = line.substring(prefix.length);

return (
this.cleanupFmtpLine(fmtpLineFromSdp, this.codecName) ===
this.cleanupFmtpLine(codecCapabilityFmtpLine, this.codecName)
);
}

/**
* Helper function to clean up the FMTP line by removing unnecessary parameters
* and normalizing certain codec-specific attributes.
*/
private cleanupFmtpLine(fmtpLine: string, codecName: string): string {
// Remove starting bitrate parameter which isn't relevant to matching.
const bitrateRegex = /;x-google-start-bitrate=\d+/g;
let cleanedLine = fmtpLine.replace(bitrateRegex, '');

if (codecName === 'H264') {
// Given that most H.264 decoders for the past decade can support level 5.2, we do not bother checking the Profile Level
// with regards to canonically defined, meeting wide, H.264 profile options (i.e. we intentionally do not split up `h264HighProfile`
// into `h264HighProfileLevel52, `h264HighProfileLevel31`, etc.).
Expand All @@ -49,17 +83,10 @@ export default class VideoCodecCapability implements Eq {
//
// If maximum compatability with H.264 is desired by a builder they should stick to H.264 Constrained Baseline Profile.
const profileLevelIdRegex = /profile-level-id=([0-9a-f]{4})[0-9a-f]{2}/i;
const modifiedCodecCapabilityFmtpLine = codecCapabilityFmtpLine.replace(
profileLevelIdRegex,
'profile-level-id=$1'
);
const modifiedLine = line.replace(profileLevelIdRegex, 'profile-level-id=$1');

return modifiedLine.startsWith(
`a=fmtp:${expectedPayloadType} ${modifiedCodecCapabilityFmtpLine}`
);
cleanedLine = cleanedLine.replace(profileLevelIdRegex, 'profile-level-id=$1');
}
return line.startsWith(`a=fmtp:${expectedPayloadType} ${codecCapabilityFmtpLine}`);

return cleanedLine;
}

/**
Expand Down
21 changes: 12 additions & 9 deletions src/task/SetLocalDescriptionTask.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,6 @@

import AudioVideoControllerState from '../audiovideocontroller/AudioVideoControllerState';
import DefaultBrowserBehavior from '../browserbehavior/DefaultBrowserBehavior';
import DefaultModality from '../modality/DefaultModality';
import SDP from '../sdp/SDP';
import BaseTask from './BaseTask';

Expand Down Expand Up @@ -56,16 +55,20 @@ export default class SetLocalDescriptionTask extends BaseTask {
sdp = new SDP(sdp).removeH264SupportFromSendSection().sdp;
}

// We set content hint to `motion` as a workaround for the issue Chrome cannot enable temporal
// The default start bitrate is 300kbps. While this is a reasonable starting point with respect
// to network adaptation, the encoding bitrate does not adjust as quickly as the estimate, which
// can quickly max out on a clear network after a few seconds. Thus it will often take 10-25 seconds
// for the encode bitrate/resolution to reach the intended value. The network adaptation detects issues quick enough
// that raising it to 1000kbps is not much concern, on most networks the congestion controller
// will react to network restrictions before the first keyframe is even generated!
//
// Note that this is also important with regards to content share with SVC enabled,
// as we set content hint to `motion` as a workaround for the issue that Chrome cannot enable temporal
// scalability for screen share https://bugs.chromium.org/p/chromium/issues/detail?id=1433486
// As a side effect, content share may start at a low resolution and take a long time to adapt,
// especially when there is limited motion on screen. To mitigate the problem, we set a starting
// bitrate of 100 kbps for content share with SVC enabled.
const attendeeId = this.context.audioVideoController.configuration.credentials.attendeeId;
const isContent = new DefaultModality(attendeeId).hasModality(DefaultModality.MODALITY_CONTENT);
if (isContent && this.context.audioVideoController.configuration.enableSVC) {
sdp = new SDP(sdp).withStartingVideoSendBitrate(100).sdp;
}
// especially when there is limited motion on screen, if we do not set this slightly higher starting
// bitrate.
sdp = new SDP(sdp).withStartingVideoSendBitrate(1000).sdp;

if (
this.context.videoSendCodecPreferences !== undefined &&
Expand Down
Loading
Loading