Ocr showcase #866

hepengfe · 2022-07-08T00:50:40Z

This PR fixes #867

Description of changes

Implement a notebook tutorial for OCR to show forte's capability on managing related data such as images containing text and recognized text, texts' bounding boxes and their relations..

Possible influences of this PR.

Describe what are the possible side-effects of the code change.

Test Conducted

Describe what test cases are included for the PR.

…y.py

hunterhector

Let's always put the images to the asset branch, besides the size problem, there are two more reasons: 1. git is also not good at managing binary data 2. the asset branch won't subject to frequent change like master (so the url will be more stable)
I have attempted to clean up the payload structure in another branch. It is not finished and I deliberately make a wrong test case, but some changes there may help you.
It feels like there are a lot of payload basic not ready to finish up this OCR showcase.

hunterhector · 2022-07-08T23:10:52Z

forte/data/data_pack.py

@@ -47,13 +46,13 @@
 from forte.data.ontology.core import EntryType
 from forte.data.ontology.top import (
    Annotation,
+    Grids,


Grids is still an entry? I thought we discuss that it should be a data structure. Or is that change not merged?

It's not merged yet and it's in #827

hunterhector · 2022-07-08T23:12:07Z

forte/data/data_pack.py

@@ -244,7 +243,7 @@ def text(self) -> str:
    @property
    def audio(self) -> Optional[np.ndarray]:
        r"""Return the audio of the data pack"""
-        return self.get_payload_data_at(Modality.Audio, 0)
+        return cast(np.ndarray, self.get_payload_data_at(Modality.Audio, 0))


I think this cast is not safe, not everything can be cast this way

There was a typing issue mypy forte, and I haven't find a better way to solve it.

Actually, I don't think we should assume audio to be ndarray

hunterhector · 2022-07-08T23:15:50Z

forte/data/data_pack.py

@@ -569,7 +565,7 @@ def set_text(
        # temporary solution for backward compatibility
        # past API use this method to add a single text in the datapack
        if len(self.text_payloads) == 0 and text_payload_index == 0:
-            from ft.onto.base_ontology import (  # pylint: disable=import-outside-toplevel
+            from ft.onto.payload_ontology import (  # pylint: disable=import-outside-toplevel


why are we importing locally here, these should be imported top level

hunterhector · 2022-07-08T23:16:27Z

forte/ontology_specs/payload_ontology.json

+        "description": "A payload that caches image data",
+        "attributes":[]
+      },
+        {


let's make the indent aligned.

hunterhector · 2022-07-08T23:17:22Z

forte/ontology_specs/payload_ontology.json

+              "type": "str"
+            },
+              {"name": "mime",
+              "type": "str"},


just use a json formatter

hunterhector · 2022-07-08T23:33:39Z

ft/onto/base_ontology.py

@@ -435,7 +463,12 @@ class CrossDocEventRelation(MultiPackLink):
    ParentType = EventMention
    ChildType = EventMention

-    def __init__(self, pack: MultiPack, parent: Optional[Entry] = None, child: Optional[Entry] = None):
+    def __init__(


Is this code generated? I feel like there are some manual or tool modifications to this file.

hunterhector · 2022-07-08T23:35:13Z

tests/forte/utils/payload_factory_test.py

+        img_meta = JpegMeta(datapack)
+        img_meta.source_type = "local"
+
+        self.f.register(img_meta)


The purpose of a register is to associate something together. For example, here we should associate the meta data info with the function to load the meta data, and the function to serialize the meta data. But if you only provide one value to register, it cannot achieve what we want.

hunterhector · 2022-07-08T23:37:30Z

ocr.ipynb

@@ -0,0 +1,181 @@
+{
+ "cells": [


Can you link the notebook to the tutorial section next time? So that I can also review what it looks like in the tutorial section

hunterhector · 2022-07-08T23:45:32Z

ocr.ipynb

+    "        pack.set_text(ocr_text)\n",
+    "        pack.pack_name = data_source\n",
+    "        \n",
+    "        yield pack\n",


The loading part is not what we desired, for example, it requires one to explicitly set the cache. And I don't even see metadata being used here.

pack: DataPack = DataPack() jpegMeta = JpegMeta() ImagePayload(pack, url="some_url", meta=jpegMeta) # payload_index default to 0

Now the user's job is done, the rest should all be handled by Forte (well, the user could also provide a method to load jpegMeta)

This is what we call lazy_loading, it doesn't happen anywhere near the reader, but behind the scene.

When the Payload is used, it should load the data to cache based on URL and jpegMeta, there doesn't need to be a step where the user set the cache.

hunterhector · 2022-07-08T23:45:53Z

ocr.ipynb

+    "        yield pack\n",
+    "\n",
+    "\n",
+    "# TODO: split ocr part into a processor"


yeah please do

hepengfe · 2022-07-12T17:52:46Z

#875

hepengfe added 28 commits July 6, 2022 09:17

payload factory and its test cases

2e156e2

add audio and image payload meta ontology

b94097b

add meta ontology

4f156ab

Merge branch 'master' into lazy_loading

793c35c

Merge branch 'master' into lazy_loading

aa7427d

reconstruct the payload ontology

6f76e2f

add some docstring

054b4af

correct importing path

514c17c

move payload to a separate ontology file

969c4b4

Merge branch 'lazy_loading' into ocr_showcase

313e570

move Modality Payload: base_ontology -> payload_ontology

1f7211a

move Modality Payload: base_ontology -> payload_ontology

d06d21d

move Modality Payload: base_ontology -> payload_ontology

ccbe2e6

payload ontology

9a0f0ad

add DataPack.grids back as it's used in some test cases

ba2e9c3

correct modality error message

91a2bda

change to a hashable meta info for registering in payload factory

55a4e89

payload ontology: ft/onto/base_ontology.py -> ft/onto/payload_ontolog…

5589d63

…y.py

remove used import

f00cafb

correct import path

6c2d598

correct import path

9794cfd

add audio encoding

1386f8d

rm pdb

7725fec

pylint

5cd139d

pylint

a67baa2

Merge branch 'lazy_loading' into ocr_showcase

f85b456

recover init methods to debug ontology generation

eafb17c

Merge branch 'lazy_loading' into ocr_showcase

045392d

hepengfe added topic: examples Issue about examples topic:cv labels Jul 8, 2022

hepengfe self-assigned this Jul 8, 2022

hepengfe added 4 commits July 7, 2022 17:51

ocr example file

add6582

minor changes

e0ea6fa

Merge branch 'master' into lazy_loading

3f4d32a

Merge branch 'lazy_loading' into ocr_showcase

388da05

hepengfe requested review from hunterhector and mylibrar July 8, 2022 21:53

hunterhector reviewed Jul 8, 2022

View reviewed changes

Merge branch 'asyml:master' into ocr_showcase

8eddea4

hepengfe closed this Jul 12, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Ocr showcase #866

Ocr showcase #866

hepengfe commented Jul 8, 2022 •

edited

Loading

hunterhector left a comment

hunterhector Jul 8, 2022

hepengfe Jul 8, 2022

hunterhector Jul 8, 2022

hepengfe Jul 8, 2022

hunterhector Jul 9, 2022

hunterhector Jul 8, 2022

hunterhector Jul 8, 2022

hunterhector Jul 8, 2022

hunterhector Jul 8, 2022

hunterhector Jul 8, 2022

hunterhector Jul 8, 2022

hunterhector Jul 8, 2022 •

edited

Loading

hunterhector Jul 8, 2022

hepengfe commented Jul 12, 2022

Ocr showcase #866

Ocr showcase #866

Conversation

hepengfe commented Jul 8, 2022 • edited Loading

Description of changes

Possible influences of this PR.

Test Conducted

hunterhector left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hunterhector Jul 8, 2022 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

hepengfe commented Jul 12, 2022

hepengfe commented Jul 8, 2022 •

edited

Loading

hunterhector Jul 8, 2022 •

edited

Loading