[Unity Catalog] - Change the import library process

[thesqlpro]

Libraries are being imported into DBFS - as of now with Unity Catalog enabled these files go into the legacy Hive metastore. We would like to upload them into Unity Catalog. Additionally, need to back port the work to ADF as it will fail with the new way of storing the libraries. #896 depends on this task as the upgrade to the cluster will cause problems for ADF. Task #896 is ready, but this needs to be completed first.

DoD

• Library files are stored in the Unity Catalog metastore and no longer in the legacy Hive Metastore
• Research -Back port of ADF pipelines to use new file paths from metastore

Task for #765

[ydaponte]

@thesqlpro, please do add a DoD and fill in the metadata.

[thesqlpro]

DoD and meta updated

[thesqlpro]

Adding some notes here so we can discuss at the next design meeting. Cluster configured to 15.4LTS with Unity Catalog. Libraries are still being mounted properly in DBFS notebooks function properly as well.

[thesqlpro]

ADF executes properly until SQLDW load (investigating).

[thesqlpro]

Some more findings. The ADF Linked Service for Databricks uses an older version of spark that's not in our current configuration.