Skip to content

Commit

Permalink
Proper image support in new pipeline/data API + Context/Metrics/Profi…
Browse files Browse the repository at this point in the history
…ler API (placeholder) (#308)

* API change: add Context to execution API with profiler() and metrics() methods + placeholder APIs

Signed-off-by: Alex Black <blacka101@gmail.com>

* Image JSON, PNG file support, class relocations, fixes

Signed-off-by: Alex Black <blacka101@gmail.com>

* BufferedImage support

Signed-off-by: Alex Black <blacka101@gmail.com>

* Add konduit-serving-javacv

Signed-off-by: Alex Black <blacka101@gmail.com>

* Add 2-step image conversion: X -> PNG -> Y when no direct X -> Y converter exists

Signed-off-by: Alex Black <blacka101@gmail.com>

* 2-step X -> SerializedNDArray -> Y conversion for NDArrays; fix image type conversion in konduit-serving-camera

Signed-off-by: Alex Black <blacka101@gmail.com>

* Update readme

Signed-off-by: Alex Black <blacka101@gmail.com>
  • Loading branch information
AlexDBlack committed May 7, 2020
1 parent 67673c7 commit 65c2372
Show file tree
Hide file tree
Showing 83 changed files with 2,271 additions and 230 deletions.
47 changes: 43 additions & 4 deletions dev_docs/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,7 +29,8 @@ The list below briefly describes those implemented so far (as of 05/05/2020)- th
* konduit-serving-models: Parent module for each of the model types
* konduit-serving-deeplearning4j: Deeplearning4j models.
* konduit-serving-data: Parent module for data and datatypes
* konduit-serving-nd4j: Mainly NDArray integration/functionality for ND4J
* konduit-serving-nd4j: Mainly NDArray integration/functionality for ND4J
* konduit-serving-javacv: Image conversion functionality for JavaCV
* konduit-serving-io: Parent module for I/O functionality - sensors, cameras, etc - and maybe later things like HDFS, S3, etc
* konduit-serving-camera: Steps related to capturing data from device-connected cameras (WIP)

Expand Down Expand Up @@ -248,6 +249,20 @@ ByteBuffer bb = sArr.getBuffer();
All modules that define an NDArray type should implement conversion to SerializedNDArray, so we can do JSON and Protobuf
serialization from any format - in a standardized form.

As of 07/05/2020 supported formats for images include:
* Png
* BufferedImage
* JavaCV Mat (konduit-serving-javacv)
* JavaCV Frame (konduit-serving-javacv)
* OpenCV Mat (konduit-serving-javacv)
More formats will be added in the future.

As of 07/05/2020 supported NDArray formats include:
* SerializedNDArray (Konduit Serving serialization/interchange format)
* float[], float[][], float[][][], float[][][][], float[][][][][]
* INDArray (konduit-serving-nd4j)
The full set of Java primitive array types (1d to 5d int[], double[], byte[] etc is planned to be added)


## JSON Serialization / Deserialization

Expand Down Expand Up @@ -314,6 +329,13 @@ As for the actual JSON format - this is basically as follows (taken from DataJso
"myKey" : {
"@BytesBase64" : "AAEC"
}
}
----- IMAGE -----
{
"myKey" : {
"@ImageFormat" : "PNG",
"@ImageData" : "iVBORw0KGgoAAAANSUhEUgAAACAAAAAgCAYAAABzenr0AAAAAXNSR0IArs4c6QAAAARnQU1BAACxjwv8YQUAAAAJcEhZcwAADsQAAA7EAZUrDhsAAAPcSURBVFhH7VfZK3VRFF/XNRSZyTyHyAOSUqZHpRCRPCjlzZM/QH3/AU+G4oHyJK9KmQohJSKZZxkjmcf1rbWce13Hce5xP3Vfvl/92ufss4ffXnuttfcxAQASnYKkpCRwUZ6dhv8CnC7gn5zQZDIB4nv39PR08PHxAX9/fzg+PoalpSUoKyt7dzQXF5ienoaZmRk4Pz+X9gz+ZkgAD8ATeXt7ywReXl7CqKgoeHl5gcfHR5ksODgYQkNDYXFxEQYHB6G5uRliY2NljJaWFmhra4O1tTV5Z9gVwCt0dXWFyspKmYgHS0tLg7i4OAgLC4OEhASl5QeOjo5gYWEBrq+vIScnR6yxvr4Ora2tMD8/Dzc3N0pLAwI8PDwgIiICNjc3lZrPeHt7E7KFGFx2dnZCf3+/iL+/vxdBt7e3cHBwIG1swQIYLECTNKCUr6+vtAMfsH2nVeHw8LDyhtjQ0PBlnO9IAtBMD3+ImqDxpFxZWZFV9PX1wcPDAyQnJ0vJe52RkQE9PT3Q0dEhfjE3Nwerq6vSzx4CAwOl/KJMTV9fX0xMTMSAgACsqqrCra0t3N7extraWmsbs9mM0dHR0ta2rx7ZAobywNXVlfjBxcUFnJ6eioP5+flBamqq0gKAtgX29vak7U9gOBGxszGWl5clxHh76urqrGZ0FLo+oAX2BV5lSEgIZGdnW8OKTC8Wsgg1AsM+oCZHR2FhIVKMIzkjjoyMYFdXF1I2RApdpBDU7Kcm+wCV2h/t0dPTE0tLSyX0LOju7sbc3FwRodVHzX8SYCGlZ5yYmFAkIFIWRPINaw7R468IYLKI4uJi3NnZERGHh4fY3t6u2daWvybAQjrxRABjf38fKUw121loOA/ogT25uroaGhsbgRKR1N3d3cHu7q6ErD04dB/ggyYyMlJOu7y8PMjKyoKgoCCIiYkBNzc3SVRDQ0NQXl6u9NAGWUDKL6bRI+93QUEBNjU14eTkJD49PSlGR6QjGzc2NpBOQ0OHEm+BIQtQnpejmW89vOKKigrIzMyUegYnJ07Ds7OzMDAwAJQX4OzsTL7pwa4FOJTItJiSkoJFRUVIg+Pz87OyXkS6CSFlQhwdHcWamhrNMfRoNwro1oNjY2N4eXmpTPkBFjI1NYX5+fmafY1QUwBPyolkfHwcyZuV6T6DLpxYX1//qZ8j1BRQUlIiq7YF7Sf29vZKvueJw8PDJRWr+/6Umk5Ig4tzcElmlpDjsKLsJifdycmJIQczAp7HoTzwW2ABTv0zEgtziHH2UoM/quv534D/E9T4rp6SFLi7uytv7yCXkusbb218fDz8Bd+qYeQWjEzlAAAAAElFTkSuQmCC"
}
}
----- DOUBLE -----
{
Expand All @@ -330,8 +352,8 @@ As for the actual JSON format - this is basically as follows (taken from DataJso
----- DATA -----
{
"myKey" : {
"myInnerKey" : "myInnerValue"
}
"myInnerKey" : "myInnerValue"
}
}
```

Expand All @@ -342,6 +364,8 @@ Bytes, is a special case - it's an object with a special protected key: `@BytesB
will encode `byte[]` objects as base64 format for space efficiency; later we will allow "array style" byte[] encoding
(i.e., `"@BytesArray" : [1,2,3]`). Users are not allowed to add anything to a Data instance with these protected keys.

Image data is stored by default in PNG format, base64 encoded (this will be configurable eventually).

NDArray is a special case also: it in a JSON object with type/shape/data keys. Currently data is base64 encoded, but
we may allow a "1d buffer array" format in the future also.

Expand All @@ -353,7 +377,6 @@ The full set of protected keys can be found on the Data interface. They include:
* @NDArrayShape
* @NDArrayType
* @NDArrayDataBase64
* @NDArrayDataBase64
* @Metadata

In practice, Data JSON serialization/deserialization is implemneted in the DataJsonSerializer and DataJsonDeserializer classes.
Expand Down Expand Up @@ -390,6 +413,13 @@ See for example: konduit-serving-deeplearning4j
- For NDArray: the main (strictly required) one is conversion to/from SerializedNDArray - this is necessary for JSON
and protobuf serialization/deserialization, and will be used as the intermediate format for conversion between arbitrary
types that don't have direct (1 step) conversion enabled (i.e., X -> SerializedNDArray -> Y for any X and Y).
- For Image: the main (strictly required) one is conversion to/from Png (i.e., `ai.konduit.serving.pipeline.impl.data.image.Png`)
as this is used as the default format for both JSON and Protobuf serialization - and also used as the intermediate
format for conversion between arbitrary image formats that don't have direct (1 step) conversion available (i.e., X
-> Png -> Y for any X and Y).
The reason PNG was chosen: PNG is a compressed lossless image format, unlike some alternatives such as jpeg. It does
have a size overhead for natural images vs. jpeg, but in practice for deep learning we typically don't have very large
input/output images so this is a secondary concern.
- Add an NDArrayFactory / ImageFactory (used within NDArray.create(Object) / Image.create(Object))
- Add a `resources/META-INF/services/ai.konduit.serving.pipeline.api.format.NDArrayConverter` (or `.ImageConverter`) file
listing the fully-qualified class name of all of the new NDArrayConverter/ImageConverter implementations you added
Expand All @@ -399,6 +429,15 @@ See for example: konduit-serving-deeplearning4j

See for example: konduit-serving-nd4j


A note on naming packages for new modules: The packages (that directly contain classe) should be unique, and are based on
the module name.
For example, the kondit-serving-nd4j module (under konduit-serving-data) has classes in `ai.konduit.serving.data.nd4j`.
Similarly, konduit-serving-deeplearning4j (under konduit-serving-models) has classes in `ai.konduit.serving.models.deeplearning4j`.
It's fine to have sub-packages (i.e., any X in `ai.konduit.serving.data.nd4j.X` is fine) but we cannot have classes
in the same package - i.e., modules X and Y can't both define classes directly in the same namespace.
The reason is to avoid split packages issues for OSGi and Java 9 Modules. We will likely use OSGi extensively in the future.




Expand Down
68 changes: 68 additions & 0 deletions konduit-serving-data/konduit-serving-javacv/pom.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,68 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ /* ******************************************************************************
~ * Copyright (c) 2020 Konduit K.K.
~ *
~ * This program and the accompanying materials are made available under the
~ * terms of the Apache License, Version 2.0 which is available at
~ * https://www.apache.org/licenses/LICENSE-2.0.
~ *
~ * Unless required by applicable law or agreed to in writing, software
~ * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
~ * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
~ * License for the specific language governing permissions and limitations
~ * under the License.
~ *
~ * SPDX-License-Identifier: Apache-2.0
~ ******************************************************************************/
-->

<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>konduit-serving-data</artifactId>
<groupId>ai.konduit.serving</groupId>
<version>0.1.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>

<artifactId>konduit-serving-javacv</artifactId>

<dependencies>
<dependency>
<groupId>ai.konduit.serving</groupId>
<artifactId>konduit-serving-pipeline</artifactId>
<version>${project.version}</version>
</dependency>

<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>

<dependency>
<groupId>org.bytedeco</groupId>
<artifactId>javacv-platform</artifactId>
<version>${javacpp.version}</version>
</dependency>


<!-- Test dependencies -->
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>


<dependency>
<groupId>ch.qos.logback</groupId>
<artifactId>logback-classic</artifactId>
<version>${logback.version}</version>
<scope>test</scope>
</dependency>

</dependencies>

</project>
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
/*
* ******************************************************************************
* * Copyright (c) 2020 Konduit K.K.
* *
* * This program and the accompanying materials are made available under the
* * terms of the Apache License, Version 2.0 which is available at
* * https://www.apache.org/licenses/LICENSE-2.0.
* *
* * Unless required by applicable law or agreed to in writing, software
* * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
* * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
* * License for the specific language governing permissions and limitations
* * under the License.
* *
* * SPDX-License-Identifier: Apache-2.0
* *****************************************************************************
*/

package ai.konduit.serving.data.javacv.data;

import ai.konduit.serving.pipeline.impl.data.image.BaseImage;
import org.bytedeco.javacv.Frame;

public class FrameImage extends BaseImage<Frame> {
public FrameImage(Frame image) {
super(image);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,28 @@
/*
* ******************************************************************************
* * Copyright (c) 2020 Konduit K.K.
* *
* * This program and the accompanying materials are made available under the
* * terms of the Apache License, Version 2.0 which is available at
* * https://www.apache.org/licenses/LICENSE-2.0.
* *
* * Unless required by applicable law or agreed to in writing, software
* * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
* * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
* * License for the specific language governing permissions and limitations
* * under the License.
* *
* * SPDX-License-Identifier: Apache-2.0
* *****************************************************************************
*/

package ai.konduit.serving.data.javacv.data;

import ai.konduit.serving.pipeline.impl.data.image.BaseImage;
import org.bytedeco.opencv.opencv_core.Mat;

public class MatImage extends BaseImage<Mat> {
public MatImage(Mat image) {
super(image);
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,34 @@
/*
* ******************************************************************************
* * Copyright (c) 2020 Konduit K.K.
* *
* * This program and the accompanying materials are made available under the
* * terms of the Apache License, Version 2.0 which is available at
* * https://www.apache.org/licenses/LICENSE-2.0.
* *
* * Unless required by applicable law or agreed to in writing, software
* * distributed under the License is distributed on an "AS IS" BASIS, WITHOUT
* * WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the
* * License for the specific language governing permissions and limitations
* * under the License.
* *
* * SPDX-License-Identifier: Apache-2.0
* *****************************************************************************
*/

package ai.konduit.serving.data.javacv.data;

import ai.konduit.serving.data.javacv.util.OpenCVUtil;
import ai.konduit.serving.pipeline.impl.data.image.BaseImage;
import org.opencv.core.Mat;

public class OpenCVMatImage extends BaseImage<Mat> {

static {
OpenCVUtil.ensureOpenCVLoaded();
}

public OpenCVMatImage(Mat image) {
super(image);
}
}
Loading

0 comments on commit 65c2372

Please sign in to comment.