Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add debug logging when ingestion table lineage from Databricks UC #19458

Open
laufermi opened this issue Jan 21, 2025 · 0 comments
Open

Add debug logging when ingestion table lineage from Databricks UC #19458

laufermi opened this issue Jan 21, 2025 · 0 comments
Labels
enhancement New feature or request

Comments

@laufermi
Copy link

Is your feature request related to a problem? Please describe.
When the hereditary extraction pipeline is executed, it is only partially successful. However, the execution process does not indicate which table failed.

I executed the pipeline for 8 hours and it was partially successful. However, the problem was caused by a lack of permissions on one of the tables. The current log only shows the following information, which is not enough to help me find the table name of the error from thousands of tables

| Source | Unhandled     | Encountered exception running step [<metadata.ingestion.source.database.unitycatalog.lineage.UnitycatalogLineageSource object at 0x7395cf06f1f0>]: [Table Name should be informed, but got table=`None`] | Traceback (most recent call last):      

Describe the solution you'd like
Add debug log in UnityCatalog Client to print the name of the table being processed

Describe alternatives you've considered
Enhance the error message, add query conditions provided by pre-step when query fails

Additional context
Add any other context or screenshots about the feature request here.

@laufermi laufermi added the enhancement New feature or request label Jan 21, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant