Understanding the Screen Share Protocol

The screen share protocol in Ulterius works by looking for difference in individual frames of the desktop. These changes are then sent to the client as patches which can be stitched into the canvas.

Screen capture data is packed into the following binary format

-----Body-----
Guid (byte[16])
Bounds X (int32)
Bounds Y (int32)
Bounds Top (int32)
Bounds Bottom (int32)
Bounds Left (int32)
Bounds Right (int32)
Jpeg Patch (byte[<remaining>])
----------

For example, the web client uses this parse function (using the jDataView library):

export function unpackFrameData(data: Uint8Array) {
    let fv = new jDataView(data, 0, data.length, true)
    let uid = fv.getString(16) // unused
    let x = fv.getInt32()
    let y = fv.getInt32()
    let top = fv.getInt32()
    let bottom = fv.getInt32()
    let left = fv.getInt32()
    let right = fv.getInt32()
    let image = decompressData(data.subarray(16 + (4*6)))
    return {
        x, y, top, bottom, left, right, image
    }
}

When starting an Ulterius screen share stream these patches will be sent to you in real time.

You can also request a patch consisting of the entire screen (full frame) by using the fullframe endpoint. The full frame response actually comes in a json format instead:

declare interface FrameData {
    screenBounds: {
        top: number,
        bottom: number,
        left: number,
        right: number,
        x: number,
        y: number,
        height: number,
        isEmpty: boolean,
    },
    frameData: number[] //byte array; the actual image
}