Update docs

README.md

@@ -75,12 +75,12 @@ Vidformer is a highly modular suite of tools that work together; these are detai
 However, vidformer is highly complementary to each of these.
 If you're working on any of the later four, vidformer may be for you.
 
-**License:** Vidformer is open source under [Apache-2.0](./LICENSE).
-Contributions welcome.
-
 **File Layout**:
 - [*./vidformer*](./vidformer/): The core transformation library
 - [*./vidformer-py*](./vidformer-py/): A Python video editing client
 - [*./vidformer-cli*](./vidformer-cli/): A command-line interface for vidformer servers
 - [*./snake-pit*](./snake-pit/): The main vidformer test suite
 - [*./docs*](./docs/): The [vidformer website](https://ixlab.github.io/vidformer/)
+
+**License:** Vidformer is open source under [Apache-2.0](./LICENSE).
+Contributions welcome.

docs/src/SUMMARY.md

@@ -2,10 +2,16 @@
 
 - [Introduction](./introduction.md)
 - [Getting Started](./getting-started.md)
+    - [Install](./install.md)
+    - [Getting Started - cv2](./getting-started-cv2.md)
+    - [Getting Started - DSL](./getting-started-dsl.md)
 - [Concepts & Data Model](./concepts.md)
 - [Tools](./tools.md)
     - [vidformer-py](./vidformer-py.md)
     - [libvidformer](./libvidformer.md)
-- [Built-in Filters](./builtin-filters.md)
-- [User-Defined Filters](./udf-filters.md)
-- [FAQ](./faq.md)
+- [Filters](./filters.md)
+    - [Built-in Filters](./builtin-filters.md)
+    - [OpenCV/cv2](./opencv-filters.md)
+    - [User-Defined Filters](./udf-filters.md)
+- [Roadmap](./roadmap.md)
+- [FAQ](./faq.md)

docs/src/filters.md

@@ -0,0 +1,5 @@
+# Filters
+
+* [For the vidformer builtin filters](./builtin-filters.md)
+* [For the OpenCV/cv2 filters](./opencv-filters.md)
+* [For using User Defined Filters (UDFs)](./udf-filters.md)

docs/src/getting-started-cv2.md

@@ -0,0 +1,81 @@
+# Getting Started - cv2
+
+This is a walkthrough of getting started with vidformer OpenCV `cv2` compatability layer.
+
+> ⚠️ Adding `cv2` functions is a work in progress. See the [cv2 filters page](./opencv-filters.md) for which functions have been implemented.
+
+## Installation
+
+See [Installation guide](./install.md)
+
+Or you can [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/ixlab/vidformer/blob/main/misc/Colab_Vidformer.ipynb).
+> ⚠️ Due to how Colab networking works, vidformer can't stream/play results in Colab, only save them to disk. `cv2.vidplay()` will not work!
+
+## Hello, world!
+
+Copy in your video, or use ours:
+
+```bash
+curl -O https://f.dominik.win/data/dve2/tos_720p.mp4
+```
+
+Then just replace `import cv2` with `import vidformer.cv2 as cv2`.
+Here's our example script:
+
+```python
+import vidformer.cv2 as cv2
+
+cap = cv2.VideoCapture("tos_720p.mp4")
+fps = cap.get(cv2.CAP_PROP_FPS)
+width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
+height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
+
+out = cv2.VideoWriter("output.mp4", cv2.VideoWriter_fourcc(*"mp4v"),
+                        fps, (width, height))
+while True:
+    ret, frame = cap.read()
+    if not ret:
+      break
+
+    cv2.putText(frame, "Hello, World!", (100, 100), cv2.FONT_HERSHEY_SIMPLEX,
+                1, (255, 0, 0), 1)
+    out.write(frame)
+
+cap.release()
+out.release()
+```
+
+### Stream the Results
+
+Saving videos to disk works, but we can also display them in the notebook.
+Since we stream the results and only render them on demand this can start practically instantly!
+
+First, replace `"output.mp4"` with `None` to skip writing the video to disk.
+Then you can use `cv2.vidplay()` to play the video!
+
+```python
+import vidformer.cv2 as cv2
+
+cap = cv2.VideoCapture("tos_720p.mp4")
+fps = cap.get(cv2.CAP_PROP_FPS)
+width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
+height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
+
+out = cv2.VideoWriter(None, cv2.VideoWriter_fourcc(*"mp4v"),
+                        fps, (width, height))
+while True:
+    ret, frame = cap.read()
+    if not ret:
+      break
+
+    cv2.putText(frame, "Hello, World!", (100, 100), cv2.FONT_HERSHEY_SIMPLEX,
+                1, (255, 0, 0), 1)
+    out.write(frame)
+
+cap.release()
+out.release()
+
+cv2.vidplay(out)
+```
+
+> ⚠️ By default `cv2.vidplay()` will return a video which plays in a Jupyter Notebook. If running outside a jupyter notebook you can pass `method="link"` to return a link instead.

docs/src/getting-started-dsl.md

@@ -0,0 +1,154 @@
+# Getting Started - DSL
+
+This is a walkthrough of getting started with `vidformer-py` core DSL.
+
+## Installation
+
+See [Installation guide](./install.md)
+
+## Hello, world!
+
+> ⚠️ We assume this is in a Jupyter notebook. If not then [`.play()`](https://ixlab.github.io/vidformer/vidformer-py/vidformer.html#Spec.play) won't work and you have to use [`.save()`](https://ixlab.github.io/vidformer/vidformer-py/vidformer.html#Spec.save) instead.
+
+We start by connecting to a server and registering a source:
+```python
+import vidformer as vf
+from fractions import Fraction
+
+server = vf.YrdenServer(domain='localhost', port=8000)
+
+tos = vf.Source(
+    server,
+    "tos_720p",     # name (for pretty printing)
+    "https://f.dominik.win/data/dve2/tos_720p.mp4",
+    stream=0,       # index of the video stream we want to use
+)
+
+print(tos.ts())
+print(tos.fmt())
+```
+
+This will print the timestamps of all the frames in the video, and then format information:
+This may take a few seconds the first time, but frame times are cached afterwords.
+
+```
+> [Fraction(0, 1), Fraction(1, 24), Fraction(1, 12), Fraction(1, 8), ...]
+> {'width': 1280, 'height': 720, 'pix_fmt': 'yuv420p'}
+```
+
+Now lets create a 30 second clip starting at the 5 minute mark.
+The source video is at at a constant 24 FPS, so lets create a 24 FPS output as well:
+
+```python
+domain = [Fraction(i, 24) for i in range(24 * 30)]
+```
+
+Now we need to render each of these frames, so we define a render function.
+```python
+def render(t: Fraction, i: int):
+    clip_start_point = Fraction(5 * 60, 1) # start at 5 * 60 seconds
+    return tos[t + clip_start_point]
+```
+We used timestamp-based indexing here, but you can also use integer indexing (`tos.iloc[i + 5 * 60 * 24]`).
+
+Now we can create a spec and play it in the browser.
+We create a spec from the resulting video's frame timestamps (`domain`), a function to construct each output frame (`render`), and the output videos format (matching `tos.fmt()`).
+```python
+spec = vf.Spec(domain, render, tos.fmt())
+spec.play(server)
+```
+
+This plays this result:
+<video controls width="100%">
+  <source src="https://f.dominik.win/data/dve2/quickstart-hello-world.mp4" type="video/mp4" />
+</video>
+
+> Some Jupyter environments are weird (i.e., VS Code), so  `.play()` might not work. Using `.play(..., method="iframe")` may help.
+
+It's worth noting that we are playing frames in order here and outputing video at the same framerate we recieved, but that doesn't need to be the case.
+Here are some things other things you can now try:
+
+* Reversing the video
+* Double the speed of the video
+    * Either double the framerate or sample every other frame
+* Shuffle the frames into a random order
+* Combining frames from multiple videos
+* Create a variable frame rate video
+    * Note: `.play()` will not work with VFR, but `.save()` will.
+
+## Bounding Boxes
+
+Now let's overlay some bouding boxes over the entire clip:
+
+```python
+# Load some data
+import urllib.request, json 
+with urllib.request.urlopen("https://f.dominik.win/data/dve2/tos_720p-objects.json") as r:
+    detections_per_frame = json.load(r)
+
+bbox = vf.Filter("BoundingBox") # load the built-in BoundingBox filter
+
+domain = tos.ts() # output should have same frame timestamps as our example clip
+
+def render(t, i):
+    return bbox(
+        tos[t],
+        bounds=detections_per_frame[i])
+
+spec = vf.Spec(domain, render, tos.fmt())
+spec.play(server)
+```
+
+This plays this result (video is just a sample clip):
+<video controls width="100%">
+  <source src="https://f.dominik.win/data/dve2/quickstart-bounding-box.mp4" type="video/mp4" />
+</video>
+
+## Composition
+
+We can place frames next to each other with the `HStack` and `VStack` filters.
+For example, `HStack(left_frame, middle_frame, right_frame, width=1280, height=720, format="yuv420p")` will place three frames side-by-side.
+
+As a larger example, we can view a window function over frames as a 5x5 grid:
+
+```python
+hstack = vf.Filter("HStack")
+vstack = vf.Filter("VStack")
+
+w, h = 1920, 1080
+
+def create_grid(tos, i, N, width, height, fmt="yuv420p"):
+    grid = []
+    for row in range(N):
+        columns = []
+        for col in range(N):
+            index = row * N + col
+            columns.append(tos.iloc[i + index])
+        grid.append(hstack(*columns, width=width, height=height//N, format=fmt))
+    final_grid = vstack(*grid, width=width, height=height, format=fmt)
+    return final_grid
+
+domain = [Fraction(i, 24) for i in range(0, 5000)]
+
+def render(t, i):
+    return create_grid(tos, i, 5, w, h)
+
+fmt = {'width': w, 'height': h, 'pix_fmt': 'yuv420p'}
+
+spec = vf.Spec(domain, render, fmt)
+spec.play(server)
+```
+
+This plays this result (video is just a sample clip):
+<video controls width="100%">
+  <source src="https://f.dominik.win/data/dve2/quickstart-composition.mp4" type="video/mp4" />
+</video>
+
+## Viewing Telemetry (and User-Defined Filters)
+
+This [notebook](https://github.com/ixlab/vidformer/blob/main/misc/UDFs-demo.ipynb) shows how to build custom filters to overlay data.
+
+This plays this result (video is just a sample clip):
+<video controls width="100%">
+  <source src="https://f.dominik.win/data/dve2/quickstart-telemetry.mp4" type="video/mp4" />
+</video>