Incident Management

[pmbrull]

WDYT about adding an “incidents” property to the entities? That could be specially interesting for tables so that we can easily see something like “We lost some data on YYYYMMDD”, which would help on future debugging. That could then be also linked to onduty systems such as PagerDuty to have realtime info on outages.

Ideas:

• relationship like asset (table, pipeline) --> has --> incident and incident being an entity on his own?
• an incident can be started from any entity such as table, pipeline and dashboard. But should be able to capture the root cause most probably a pipeline failure and what are all related entities that are affected can be extracted from lineage.

Happy to discuss!

[s-sanjay]

This is a good idea. Will also be good to integrate with Jira so incidents can document proper post mortem and follow ups. I am wondering if we can directly integrate Jira with entities and use that for incident management instead of creating an entity called incident but there could be other Jiras ( that are not incidents ) listed with the table as well.

[pmbrull]

That would be really interesting! We could start by preparing properties such as issueUrl and postMortemUrl and build (somehow) integrations on top of this

[gavin-denyse]

Having the ability to link a Jira ticket to an entity would help with change management and who is actively working on the entities in a more automated way.

[mithmatt]

I foresee this to be the sequence of events

1. Some pipeline P that ingests to data to table T failed. Table T usually gets 100k rows added per day. However it got 1k rows since the pipeline P failed midway.
2. Periodic data quality test on table T noticed that the number of rows inserted that day is abnormally low compared to expected 100k rows
3. Data Quality test reports for table T ingested into OpenMetadata
4. OpenMetadata sees that the Data Quality SLA was breached (a change event)
5. Bot(s) watches the change event on table T data quality report and sends notifications to Table Owner and/or Follower of the table (and join tables). (PS: Data Quality system may have directly notified the table owner)
6. Table owner chooses to register an incident entity that references multiple entities (table T, pipeline P)
7. When the incident is resolved the incident entity is marked as resolved and is available in the incident history of table T and pipeline P

[pmbrull]

Thanks for your input @mithmatt! Indeed having some automatic reports on this would be awesome.

I believe that these SLAs would be interesting to annotate not only on tables but on any entity that has any property that we can measure. For example, we should be able to check the % of failed runs in the last X days for a pipeline, or if a pipeline is taking too long to finish, or if a model performance is starting to degrade, etc.

Being able to define these kinds of automatic rules and binding them up with incident management would be a great addition IMO.

WDYT?

[mithmatt]

Yes, the idea of having SLA on entities is great.

There are more hidden capabilities that will unravel itself when we embark on the implementation for this.

If SLA of one pipeline or table is affected by an incident, it becomes easier to assess potential impact to downstream tables by leveraging lineage and ETL pipeline schedule(@sureshms and @harshach may already be familiar with this concept).

There is potential here to build a rich experience in surfacing up potential impact to downstream owners and consumers data assets.

[pmbrull]

Indeed! Moreover, while we currently have the Tiering of the assets which already give insights on what is more important/impactful for the business, being able to tie that with a set of rules could be interesting.

IMO this has two major benefits:

• Being able to create a set of automatic rules for the different assets (the pipeline should be finished every day at 10, or a pipeline should run successfully for the past 10 consecutive days...) would help in having incidents being risen without the need of engineering/support/users.
• This could improve the trust over an asset, as we would be transparent about how we expect it to behave and the SLA evolution.

This means that apart from the asset importance with Tiering, we would have an asset behavior evolution and automatic alerting.

Being able to then handle the communications and updates directly on the catalogue would centralise the lifecycle of the incidents, as developers would not need to constantly ping in Slack etc.