-
Notifications
You must be signed in to change notification settings - Fork 17
Audio Video Editing
This commentary relates to TNO-758 - MP4 clips are not viewable in the browser. Jeremy F had issues playing some MP4 clips that were extracted from captured streams on Raspberry Pi devices in the server room. When played in the MMIA Editor interface, the audio worked but the video did not. The captured files were in mpg format and the extracted clips were mp4 files. The goal of this ticket was to explore different ffmpeg configurations to ensure that both the audio and video components of a clip worked.
This ticket should involve few, if any, code changes as most of the issues can be addressed by changing the ingests' configuration, such as modifying the extension of the capture file name or customizing the copy arguments for the clip ingest. As such this is largely a research task rather than a coding one and will not require a pull request.
Jeremy and I determined that the most efficient approach would be to address the issue for captured streamed internet video (CBC Newsworld). This would allow me to work independently and not be constrained by the need for access to the server room or any of its contents. While this might not address all the issues with the Raspberry Pi based content, the findings would be applicable to captured content of all types.
The HTML5 video tag supports three formats:
- MP4
- WebM
- Ogg
MP4 and WebM are container formats which can include audio, video and subtitle streams in a single file. These components are kept in sync with each other using timestamps. For our purposes, we will be using the H.264 codec for video compression and the vorbis format for audio.
Although the formally supported formats are limited to those listed above, the browser manufacturer can support others. Also of note, a WebM file need not have the extension webm. The WebM container format is based on the Matroska standard and WebM files typically have the extension mkv (Matroska Video). These files will play successfully in the latest versions of Microsoft Edge, Google Chrome and Firefox.
Our initial solution for video capture involved writing to .mp4 files. It became clear that this would prevent us from viewing or clipping a stream while recording was in progress. It also made files that had failed during recording unreadable. One potential solution was to write to a different container format, so experiments were undertaken using mpg, resulting in the behaviour reported in TNO-758. I chose the Matroska format as a third option. Ffmpeg automatically determines the output format from the extension of the output file provided, so writing to Capture.mkv ensures that the resulting file uses the Matroska format.
I used the capture service to record the CBC Newsworld stream to the local file 14-27-40-CBC News - 01.mkv in the CBC capture directory for that day and determined the following:
- The file was captured without error.
- Accessing the file from the storage menu of the editor app allowed me to play it in the browser.
- The file played successfully while it was being recorded.
- The file still played successfully after the capture process was terminated prematurely.
I did find one limitation with this approach. If a file is played while recording is in progress the progress bar does not provide much control over playback. Normally the current position of the playback head is displayed in minutes and seconds and the total length of the video is also displayed on the timeline. In this case, because the length of the video is increasing as playback progresses, the position of the playback head is indeterminate. Also, only the portion of the video that was recorded prior to the file being accessed will play, e.g., if the recording started 1 minute 35 seconds prior to viewing, only the first 1 minute and 35 seconds of video will play. As our approach to playing captured streams has not been fleshed out, I'm not sure if this will be an issue.
The next step was to determine if I could create a clip from a captured file.
I configured the Clip service to extract a five-minute video from an mkv file while the Capture service was writing to it. I started the Clip service locally and it successfully extracted the clip, showing that clips can also be extracted from in-progress mkv files. (In a later conversation with Jeremy it became clear that the clip service allows clip extraction by copying the in-progress video to a second file - see "Obtaining the duration of an in-progress mkv capture" below.) The audio and video components of the clip extracted in this way were out of sync.
This is due to the nature of asymmetric video compression where only the changes from the previous frame are recorded between keyframes. A keyframe is a full-frame snapshot from which the changes are calculated for the following frames until a new keyframe is provided. If a requested clip must start exactly ten minutes into the video, it is possible that the frame at that timestamp is a partial one. In that case, the clip process will start extracting video from the following keyframe. This can result in lip-sync issues.
The syncronization issue relates to the copy settings for the ffmpeg clip command. The default setting in the editor app is:
-c:v copy -c:a copy
which copies the audio and video between the source and destination files without re-encoding. The issue was resolved by re-encoding the video using the H.264 codec library libx264 as follows:
-c:v libx264 -c:a copy
These results should be verified using video captured in the server room when the resources are accessible.
In researching the mkv format I successfully extracted clips from an in-progress recording on my development system using ffmpeg, without the involvement of the clip service. Unfortunately, it is not possible to extract the duration of such a stream from the file's metadata because it is unavailable. The clip service relies on the availability of this information to determine whether the entire timespan of the clip exists within the captured file. The clip service currently tries to retrieve the video duration using ffprobe as follows:
ffprobe -i '{inputFile}' -show_format -v quiet | sed -n 's/duration=//p'
These in-progress mkv recordings can be opened and viewed on Windows using the VLC media player. In such a case the duration field of the video metadata is blank, when viewed in the the player's media information dialog, but the length of the video stream is shown at the end of the timeline beneath the video frame. This means this information is available in the file in some form. I was able to retrieve the current duration of an in-progress mkv recording using the following command:
ffprobe -v 0 -show_entries packet=pts -of compact=p=0:nk=1 -read_intervals 999999 -select_streams v:0 live.mkv | tail -1
This command relies on the hierarchical structure of the file to retrieve the Picture Timestamp of the final packet in the container's first video stream. The mkv file's hierarchy can be defined as follows: Container>Stream>Packet>Frame. Each packet of frames contains a picture timestamp (pts) indicating the offset of that packet, in milliseconds, from the start of the stream. For the last packet in the stream, this value indicates the duration of the stream at the time it was sampled. This number is quite accurate as multiple packets are appended to the file every second.
The following table summarizes the arguments used in the ffprobe command above:
| Argument | Description |
|---|---|
| -v 0 | Limit output to the fields requested, but report errors. |
| -show_entries packet=pts | Select the Picture TimeStamp value from all selected packets and output on separate lines. |
| -of | Output format. |
| compact=p=0:nk=1 | Use the compact output format and suppress all descriptive text |
| -read_intervals 999999 | Seek to the position in the video stream indicated by the offset in seconds and return data relating to the previous Group Of Pictures (GOP). In order to get the last group of pictures in the stream an offset is required that is greater than or equal to the length of the stream. In our case, this is unlikely to exceed 24 hours (86,400 seconds) but using a larger value increases the utility of the command. A GOP, in the context of this command, is the set of packets written to the file since the previous keyframe. No more than a few dozen of these will be returned. |
| -select_streams v:0 | Select the first video stream. |
| live.mkv | The name of the file to probe. |
| tail -1 | Only return the last picture time stamp in the list. |
In testing, this command returned the value 11215367 indicating a stream duration of 187 minutes.
These are potential audio/video editors for Windows for use by MMIA staff on the new Government PC platform. The switch from Mac to Windows requires a replacement for Apple's Quicktime editor as this has not been supported on Windows since 2009. The ticket associated with this task (TNO-371) required the compilation of "a List of Viable Audio/Video Editing Software". A somewhat loose definition of "Viable" was taken in order to make this list as comprehensive as possible and to allow the editors to assess the viability of a product based on their own experience with it. Some packages were ruled out based on the following:
- The lowest pricing option available was deemed too high. This was hard to assess without any guidelines, but if the cost involved payments of hundreds of dollars per year or a perennial license that costs more than $300 the package was not assessed.
- The software was written with a feature set fitting the Hollywood movie industry rather than the application at hand.
All the packages assessed are available for immediate download online and provide free trial plans.
For each package, the following steps were followed:
- Install the package.
- Run it.
- Load two 320x240 mp4 videos retrieved from TNO. These videos, created by the TNO editors, were used as the raw clips were not available.
- Trim/split both videos and insert a portion of one between two portions of the other.
- Export the video as an mp4 file and ensure it plays.
- To save time, audio editing was not assessed, but each package allows the editing of uncompressed audio files.
The product owner specified the following requirement for the exported video format: H.264 > MP4 > 352 x 288 > 25 fps. I have noted when a package is capable of meeting this requirement. One thing to note is that the files posted by the editors have a resolution of 320x240.
The most intuitive packages allow you to drag and drop multiple media files into a media library. Media can be dragged from the library to a timeline in which they are cut into sections. The best packages support multiple timelines (one for each media file) that can be edited independently. Individual clips can be dragged between timelines and arranged in the correct order using the mouse.
The more expensive solutions provide features that are not required by TNO editors. These include transitions, effects, colour management and rich media libraries that include music and images. The free solutions tend not to have these features, but as TNO editors do not use them this is not important.
| Package | Assessment | Price | Website |
|---|---|---|---|
| Clipchamp | Intuitive and easy to use. Limited export options (480p, 720p and 1080p). Requires a Microsoft 365 subscription to run the free version. An "Essentials" version is available for a fee. Editing can be performed online or by installing a desktop app. | Free/CAD$13 per month for Essentials. | https://app.clipchamp.com/ |
| Microsoft Video Editor | Uses a storyboard instead of a timeline for editing making the task disjointed and time-consuming. Limited export options (540p, 720p and 1080p.) | Free (included with Windows, in the Photos app and standalone) | Pre-installed |
| Wondershare Filmora | Excellent interface and functionality. Supports many mp4 export resolutions including 320x240 and 352x288. Also supports multiple frame rates including 25 fps. Can export using 15 different audio and video formats. | Currently US$79.99 for a perpetual license (down from US$146.95) | https://filmora.wondershare.net/filmora-video-editor.html |
| VideoPad | Good interface with all the features of Filmora, but a little more difficult to use. Exports in 320x240 resolution, but not 352x288. Frame rates include 25 fps. | Free | https://www.nchsoftware.com/videopad |
| Shotcut | Not a very intuitive interface. As there is only one timeline, the second video must be placed after the initial one. They can then both be cut into segments and the segments can be moved around. This is the typical flow for editors with only one timeline. Export options allow full customization of resolution and frame rate. Had trouble exporting the edited video and only completed the first 20%. | Free | https://shotcut.org |
| Movavi Video Editor | Like Shotcut, Movavi uses a single timeline. I was able to split/splice the video clips successfully. Supports many export resolutions, frame rates and video formats including 320x240, 352x288 and 25 fps mp4. | CAD$54.95 per year. | https://www.movavi.com/video-editor-plus/ |
| PowerDirector 365 | Intuitive interface with multiple timelines. Files can be split/spliced by dragging and dropping between timelines. This package asks if you want to leave a gap when deleting unwanted sections, which takes a bit of time to respond to. Allows the definition of export profiles that support 320x240 resolution, but not 352x288. Good choice of frame rates including 25 fps. | CAD$5.58 per month | https://www.cyberlink.com/index_en_CA.html |
| WeVideo | The interface and editing experience is intuitive, smooth and fast. Unfortunately, this is an online-only solution which means the videos have to upload before you can edit them. This is very time-consuming. This also means that you are competing for server resources with other users every time you export a video, making it a slow process. The package also has limited export options - the lowest export resolution is 640x480. | US$4.99 per month | https://www.wevideo.com/ |
| VideoProc Vlogger | Intuitive interface with multiple timelines. Clips can be split/spliced across timelines. Export options are flexible but the lowest resolution available is 480x480. | Free | https://www.videoproc.com/video-editing-software/ |
| MAGIX Movie Studios 2023 | The installer would not download. | US$35.88 per year. | https://www.magix.com/us/video-editor/movie-studio/ |
| EaseUS | Intuitive editor with multiple timelines. The timeline display is a little confusing as there is a primary video timeline and multiple Picture In Picture timelines that can be edited separately, but it works well. Export options are flexible, but the lowest resolution is 640x350. | US$17.97 per month | https://multimedia.easeus.com/video-editor |
| MiniTool MovieMaker | Uses a single timeline, but clips can be placed side by side, split into segments and the segments can be moved into the desired order. Output options are limited, with no control over frame rate and the lowest resolution is 1280x720. | CAD$19.03 per month | https://moviemaker.minitool.com/ |
| OpenShot | Intuitive interface with multiple timelines. Videos can be split/spliced by dragging segments across timelines. Output options are rich with full control over frame rate. Supports 320x240 resolution, but not 352x288. | Free | https://www.openshot.org |
| Vegas Pro 20 | This product was difficult to use, but did support editing on multiple timelines. It uses an older-style Windows interface with tiny icons. It is very technical and feature-rich. Instead of "Exporting" a video you "Render" it, which was confusing. Supports many export formats including 25 fps and 320x240, but not 352x288. | US$8.63 per month | https://www.vegascreativesoftware.com/us/vegas-pro/ |