JsonSchema support

[harshach]

There are few requests from community to add support JsonSchema. Based on discussions so far , the idea here is to associate a JsonSchema document to a service or a NoSql store like MongoDb. Starting a thread to iron out the details and what would be the best way to add this feature in OpenMetadata.

[ivbeg]

LinkedIn DataHub uses SQLAlchemy to get metadata from MongoDB. It's one of the ways. Probably not the best one.

But I think that more important is OpenMetadata spec update, to support NoSQL.

• rename sqlQuery to query (NoSQL engines have other query types)
• add "nested" value to property "TableType" in Table to define tables with nested objects with jsonSchema required.
• add property "schema" to Table. Yes, jsonSchema already exists in column type, but I think that it's important to define it as part of Table definition, since it will simplify table validation against jsonSchema

[mahajpuneet]

In the case of MongoDB, we could also ideally use the GraphQL api to pull the records out. Problem is the lack of structure or any sense of schema. Perhaps we could profile the data and show the distribution of the dominant json schema present in a table. Just spit balling

[heroWang]

Or. It's something that I recently thought of.
May be we can solve this problem using some reflection tools at the code level. Some utils like https://github.com/FasterXML/jackson-module-jsonSchema https://github.com/invopop/jsonschema could transform classes to json schema, which make these no structure data in MongoDB have structured schema.