Skip to content

Data

Jump to bottom Edit New page
Thowaah edited this page Jul 9, 2024 · 66 revisions

The Augmenta tracking system supports a few output protocols to accommodate for a wide range of client applications.

Augmenta OSC Protocol v2.0

This protocol is the current default one. It is implemented in Augmenta Fusion, in Augmenta Simulator and is the most widely used in the current plugins and integrations. This protocol has been designed to support all the features that the Augmenta system is capable of.

Note : All coordinates are (otherwise specified) normalized [0-1] with a top left origin.

When an object enters, is updated or leaves the scene

/object/enter   arg0 arg1 ... argN
/object/update  arg0 arg1 ... argN
/object/leave   arg0 arg1 ... argN

where args are :

    0: frame (int)                         // Scene frame number
    1: id (int)                            // id ex : 42th object to enter stage has id=42
    2: oid (int)                           // Ordered id ex : if 3 objects on stage, 43th object can have oid=2 
    3: age (float)                         // Alive time (in s)
    4: centroid.x (float 0:1)              // Position projected to the ground (normalized)
    5: centroid.y (float 0:1)               
    6: velocity.x (float)                  // Speed and direction vector (in unit.s-1) (normalized)
    7: velocity.y (float)
    8: orientation (float 0:360)           // With respect to horizontal axis right (0° = (1,0)), rotate counter-clockwise
    		                  Estimation of the object orientation from its rotation and velocity
    9: boundingRect.x (float 0:1)          // Bounding box centre coord (normalized)	
    10: boundingRect.y (float 0:1)       
    11: boundingRect.width (float 0:1)     // Bounding box width (normalized)
    12: boundingRect.height (float 0:1)
    13: boundingRect.rotation (float 0:360)// With respect to horizontal axis right counter-clockwise
    14: height (float)                     // Height of the object (in m) (absolute)

/object/enter/extra   arg0 arg1 ... argN
/object/update/extra  arg0 arg1 ... argN
/object/leave/extra   arg0 arg1 ... argN

    0: frame (int)                         // Scene frame number        
    1: id (int)                            // id ex : 42th object to enter scene has pid=42
    2: oid (int)                           // Ordered id ex : if 3 object on stage, 43th object might have oid=2
    3: highest.x (float 0:1)               // Highest point placement (normalized)
    4: highest.y (float 0:1)
    5: distance (float)                    // Sensor distance (in m)
    6: reflectivity (float)                // Reflection value from Lidar sensor

Please note: The velocity is also normalized as the other position-dependant parameters (centroid, bounds, etc...). This was done for uniformity sake regarding the other arguments and is also the case in TUIO. This means that for non-square scenes, you will have a speed value scaled differently for x and y. If you want to do meaningful operations based on speed, you will have to multiply the x value by scene.width and the y value by scene.height.

Scene information

/scene   arg0 arg1 ... argN

    0: frame (int)                         // Scene frame number
    1: objectCount (int)                   // Number of objects
    2: scene.width (float)                 // Size (in m)
    3: scene.height (float)

Fusion software specific information

/fusion   arg0 arg1 ... argN

    0: videoOut.offset.x (float)           // Offset from scene top left (in m)
    1: videoOut.offset.y (float)
    2: videoOut.width (float)              // Size (in m)
    3: videoOut.height (float)
    4: videoOut.widthInPixels (int)        // Resolution (in pixels)
    5: videoOut.heightInPixels (int)

Legacy OSC protocols can be found here

Augmenta TUIO Protocol

We also support TUIO to be interoperable with a wide range of applications that already support it. For software for which we or the community did not already create a plugin (or cannot) it can be a good solution.

We use the standard TUIO 1.1 protocol, with additional /scene and /fusion OSC messages from the OSC protocol. Those messages can either be sent on the same network port or on a different one depending on your needs.

TUIO Protocol does not give access to the complete data collection from the Augmenta protocol. Two points need to be highlighted:

TUIO protocol doesn't support sending both the bounding box and the centroid position and orientation at the same time. The bounding box information is only sent in Blob descriptors mode. The centroid information is sent in the cursor and object descriptors mode.

We chose to use the class ID field (symbol i) of TUIO to represent Augmenta's slotID. This means that if you are interested in using the slotID, you will have to use the Object descriptor mode.

In Augmenta Fusion, as well as in the Augmenta Simulator there is no such thing as a scene depth (on the Z axis). To match with the TUIO spec, we provide the sceneDepth parameter that defaults to a value of 10 in the TUIO output of those software. It can be adjusted in the user interface of both software. This is also the reason why the Z value (velocity on z axis) is a fixed 0, there is no such data computed for Augmenta objects.

To carry this sceneDepth parameter to the clients we add a value to the scene message ONLY when it is sent from a TUIO output. The scene message sent in parallel of the TUIO data looks like this:

/scene   arg0 arg1 ... argN

    0: frame (int)                         // Scene frame number
    1: objectCount (int)                   // Number of objects
    2: scene.width (float)                 // Size (in m)
    3: scene.height (float)
    4: sceneDepth (float)

Here is the correspondence table between Augmenta data and TUIO data:

TUIO Augmenta (cursor TUIO mode) Augmenta (object TUIO mode) Augmenta (blob TUIO mode)
s id id id
i - slotID -
x, y, z centroid.x, centroid.y, (1/2 height)/sceneDepth centroid.x, centroid.y, (1/2 height)/sceneDepth boundingRect.x, boundingRect.y, (1/2 height)/sceneDepth
a, b, c orientation, 0, 0 orientation, 0, 0 boundingRect.rotation, 0, 0
w, h, d - - boundingRect.width, boundingRect.height, height/sceneDepth
f, v - - (bR.width x bR.height), (bR.width x bR.height x height)
X, Y, Z velocity.x, velocity.y, 0 velocity.x, velocity.y, 0 velocity.x, velocity.y, 0
A, B, C 0, 0, 0 0, 0, 0 0, 0, 0
m 0 0 0
r 0 0 0

in the table above bR is a shorthand for boundingRect.

Please note: As in the TUIO standard, the velocity is normalized by the size of the scene (as all the other position-dependant parameters such as centroid, bounds, etc...). This means that for non-square scenes, you will have a speed value scaled differently for x and y. If you want to do meaningful operations based on speed, you will have to multiply the x value by scene.width and the y value by scene.height.

Reminder of TUIO 1.1 semantic types:

TUIO Data Description Type
s Session ID (temporary object ID) int32
i Class ID (e.g. marker ID) int32
x, y, z Position float32 [0,1]
a, b, c Angle float32 [0,2PI]
w, h, d Dimension float32 [0,1]
f, v Area, Volume float32 [0,1]
X, Y, Y Velocity vector (motion speed and direction) float32
A, B, C Rotation velocity vector (rotation speed and direction) float32
m Motion acceleration float32
r Rotation acceleration float32

Presets are implemented in Augmenta Fusion to help you choose the best TUIO format:

Presets Supported descriptors Supported dimensions Default Description
None Cursor, Blob, Object 2D, 2.5D, 3D - -
Minimal Cursor 2D 2D Cursor -
Best Blob 2.5D, 3D 2.5D Blob Best fit for Augmenta's data
SlotID Object 2.5D 2.5D Object Supports the SlotID attribute
Notch Cursor, Object, Blob 2D 2D Blob Supported formats in Notch
Touch Designer Cursor, Object, Blob 2D, 2.5D, 3D 3D Blob Supported formats in Touch Designer

Note : By default, OSC and TUIO are sent on the same port for Touch Designer's preset as you can receive both OSC messages and TUIO messages on the same connection. For Notch's preset they are sent on separate ports.

Augmenta JSON Protocol v2.0

This JSON protocol is used over Websocket mostly to communicate with web based apps. It is a simple wrapping of the OSC message structure in JSON format.

Note : All coordinates are (otherwise specified) normalized [0-1] with a top left origin.

When an object enters the scene

{"object": 
     { "enter": {
          "frame": frame,
          "id": id,
          "oid": oid,
          "age": age,
          "centroid": {
               "x": centroid.x,
               "y": centroid.y
          },
          "velocity": {
               "x": velocity.x,
               "y": velocity.y
          },
          "orientation": orientation,
          "boundingRect": {
               "x": boundingRect.x,
               "y": boundinRect.y,
               "width": boundingRect.width,
               "height": boundingRect.height,
               "rotation": boundingRect.rotation
          },
          "height": height,
          "extra": {
               "frame": extra.frame,
               "id": extra.id,
               "oid": extra.oid,
               "highest": {                     
                    "x": extra.highest.x,
                    "y": extra.highest.y
               },
               "distance": extra.distance,
               "reflectivity": extra.reflectivity
          }
     }
}

When an object is updated (same format than enters)

{"object": 
     { "update": {

         ...

         }
     }
}

When an object leaves the scene

{"object": 
     { "leave": {

         ...

         }
     }
}

Scene information

{"scene": {
     "frame": frame,
     "objectCount": objectCount,
     "scene": {
          "width": scene.width,
          "height": scene.height
     }
}

Fusion software specific information

{"fusion":
     "textureOffset": {
          "x": textureOffset.x,
          "y": textureOffset.y
     },
     "textureBounds": {
          "x": textureBounds.x,
          "y": textureBounds.y
     },
     "targetOutSize": {
          "x": targetOutSize.x,
          "y": targetOutSize.y
     }
}

Augmenta OSC Protocol v3.0

Coordinate system

Note : This is the internal coordinate system of Augmenta, the origin and axis are freely tunable when sending data, to match the target coordinate system.

lefthandedrot_resized2

  • Origin : Top-left of detected plane
  • Axis : left-handed, Z up
  • System : metric

Design note and improvements

This redesign has been performed with two things in mind:

  • being easier to use in node-based ("no-code") creative software and other limited environment.
  • being more robust to packet lost (no need to have a timeout to clear objects if no leave message was received)

Other improvements:

  • better human readability

Limitations

There is one drawback compared to OSCv2:

By splitting the data over more messages and using more human-readable paths means that this new protocol has more overhead, thus using more bandwidth. We added options to mitigate this but keep in mind that this protocol is not aimed at delivering the maximum amount of performance.

Protocol

The Augmenta system will send 2 types of bundles to clients:

  • object updates
  • scene infos

The object udpates bundle(s) is sent at each frame. It contains information on all the objects in the scene associated with a frame number.

The scene infos bundle contains information about the scene (its size, its position etc.), it is sent at a fixed frequency (default to 10Hz in Augmenta Fusion, can be adjusted).

Object updates bundle

Object updates are packed into bundles. Each bundle includes a frame id header as well as the number and list of active objects for this frame at the end.

Bundle splitting

You can choose to receive split object update bundles with a maximum size of 1432 bytes. This can be useful if you encounter network issues due to packets being too big. In this case, multiple object update bundles belonging to the same frame will have:

  • the same frame id header
  • the same number of objects and list of objects present in the scene at the end of each bundle

Values configuration

You can choose between relative or normalized arguments. The relative arguments are in meters, relative to the scene origin. The normalized values are always between 0 and 1, we get them by dividing the positions/dimensions by the scene size.

You can also toggle an option to have split coordinates. If it is enabled, you received one message for x, one message for y and on message for z. Otherwise all 3 arguments will be sent in one message.

Types notation

i -> int, f -> float, s -> string, if not specified, type is float

# Objects update header 
/au/frame/id i                    // frame id (starts at 0, resets at MAX_INT32)

# Objects data
## Mandatory args

### When split is unchecked
/au/N/uid i                       // unique ID ex : starts at 0, e.g. 42nd object to enter scene has uid=41 (resets at MAX_INT32)
/au/N/pos x y z                   // centroid position 3d coordinates

### if split is checked
/au/N/uid i                       // unique ID ex : starts at 0, e.g. 42nd object to enter scene has uid=41 (resets at MAX_INT32)
/au/N/pos/x x                     // centroid X coordinate
/au/N/pos/y y                     // centroid Y coordinate
/au/N/pos/z z                     // centroid Z coordinate

## Optional args

/au/N/speed/val f                 // magnitude of the velocity vector in m/s, always an absolute value
/au/N/presence f                  // 0 means the point just arrived or left, 1 means the point is fully in the scene. It can also have any intermediate value depending on the state of the object.
/au/N/state s                     // state of the object ("in" -> object just entered, "out" -> object is leaving, "up" -> object is still in the scene)
/au/N/age i                       // time alive in frame
/au/N/box/rot r                   // bounding box rotation angle in degrees (counter-clockwise around normal axis from floor plane, 0° aligned with Augmenta's X axis)

### If split is unchecked
/au/N/box/pos x y z                // x y z : bounding box centre coordinates
/au/N/box/size x y z               // x y z : bounding box size alongside x y z axis
/au/N/highest/pos x y z            // position of the highest point
/au/N/speed/dir x y z              // normalized velocity vector

### if split is checked
/au/N/box/pos/x x
/au/N/box/pos/y y
/au/N/box/pos/z z
/au/N/box/size/x x
/au/N/box/size/y y
/au/N/box/size/z z
/au/N/highest/pos/x x
/au/N/highest/pos/y y
/au/N/highest/pos/z z
/au/N/speed/dir/x x
/au/N/speed/dir/y y
/au/N/speed/dir/z z

## Bundle end
/au/frame/numObj i                // number of objects
/au/frame/objList i...            // list of objects slotID in the scene

where N is the object id (also called slot id), starting at index 1.

More about the slot ID algorithm

The SlotID algorithm puts new incoming objects in the first free "slot" of a an array. When a registered object leaves, it frees the slot. The SlotID algorithm essentially gives an ID for the whole lifetime of an object in a known range. The maximum number of slots can be set by the user.

This is useful in 2 ways:

  • we can limit the number of slots to accommodate for limited processing environments (that can only handle a fixed number of objects)
  • we can be used to assign a "role" to each object that will not change during its lifetime

Below is an example of the SlotID algorithm in action.

We can take this scenario:

  1. We start with an empty scene
  2. A new object with UID 5 enters
  3. A new object with UID 6 enters
  4. A new object with UID 7 enters
  5. The object with UID 6 leaves
  6. A new object with UID 8 enters
  7. The object with UID 5 leaves
  8. The object with UID 8 leaves
  9. The object with UID 7 leaves

The result of the SlotID algorithm for the scenario above is displayed in the table below:

Each row is the state of the slots in each frame.

SlotID: 1 2 3 4 5
frame 0: - - - - -
frame 1: 5 - - - -
frame 2: 5 6 - - -
frame 3: 5 6 7 - -
frame 4: 5 - 7 - -
frame 5: 5 8 7 - -
frame 6: - 8 7 - -
frame 7: - - 7 - -
frame 8: - - - - -

Scene information bundle

# Sent at a fixed frequency
/au/scene/maxNumObj i             // 0 for unlimited, >1 for the number of slots available (the max number of slots)
/au/scene/size w h d              // scene size

# Optional scene infos
/au/scene/video/res w h           // resolution in pixels

// Scene transformation
/au/scene/pos x y z               // scene center 3d coordinate
/au/scene/rot rx ry rz            // Euler rotation of the scene

If provided, you can use the scene position and rotation to convert the relative coordinates of the objects to their absolute position in the Augmenta system. This way, changing the scene transform in the Augmenta configuration will also move it in the content.

Example sequence

In this example we use normalized data.

# server sends a clearing pattern when (re)connecting
{
/au/scene/frame  0
/au/scene/numObj 0
/au/scene/objList
}

// scene infos bundle sent when (re)connecting
{ 
/au/scene/pos     0 0 0           //scene info
/au/scene/size    10 15 10         //scene info
}

...   // a lot of frames are received

# objects update bundle
{
/au/frame/id    21375

/au/1/uid         13
/au/1/pos         0.21 0.08 0.1
/au/1/highest/pos 0.23 0.09 0.178  // normalized with 10 in sceneDepth
/au/1/speed/dir   -0.74 0.67 0.0
/au/1/speed/val   1.114
/au/1/box/pos     0.21 0.08 0.089
/au/1/box/rot     50.3
/au/1/presence    0.8
/au/1/state       up

/au/2/uid         34
/au/2/pos         0.95 0.2 0.08
/au/2/highest/pos 0.94 0.21 0.16
/au/2/speed/dir   0.52 -0.4 0.0
/au/2/speed/val   0.656
/au/2/box/pos     0.95 0.19 0.08
/au/2/box/rot     110
/au/2/presence    0.0
/au/2/state       out

/au/4/uid         512
/au/4/pos         0.53 0.1 0.4  
... // other infos
/au/4/presence    0.1
/au/4/state       in

/au/frame/numObj  4
/au/frame/objList 13 34 512 147 // objects list, so we can clear whatever is not in this list
}

# second bundle for this update
{
/au/frame/id   21375

/au/5/uid      147
/au/5/pos      0.5 0.5 0.5
...
/au/5/presence 1.0
/au/5/state    up

/au/frame/numObj   4
/au/frame/objList  13 34 512 147
}

Notes for client applications developers

If you can't script object addition or deletion, you can use the presence value as an alpha value of an asset that would represent the user (1.0 when an object is correctly detected, 0.0 when it disappears, alpha value can vary [0,1] depending on the detection quality).

When we can script/program, we can use the /frame/objList message to remove all objects that are not part of this list. This is a global state of all currently active objects. This information is transmitted to each frame, so if we lose one of them we can have the information in the next frame.

Other server side options

It is possible to change the sending addresses. For example, you can change /au/N/pos to /au/N/position.

Add a custom footer
Add a custom sidebar
Clone this wiki locally