Big Query Profiler Ingestion Problem

[mchristianto]

Why can i ingest sample data from partitioned table Big Query?

The profiler ingestion including sample data generation works on regular table and view, but when i tried on partitioned table the job finished successfully with no error and the data profiler ingested too, but no sample data generated.

This is the result of ingestion:

this is the logs:
'records': ['ProfilerSourceAndEntity [dim_branch]'],
'source_start_time': 1721968846.5706282,
'updated_records': [],
'warnings': []}�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - �[1mProfiler Status:�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - {'failures': [], 'filtered': [], 'records': ['Table [dim_branch]'], 'source_start_time': 1721968846.612994, 'updated_records': [], 'warnings': []}�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - �[1mPII Processor Status:�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - {'failures': [], 'filtered': [], 'records': ['Table [dim_branch]'], 'source_start_time': 1721968846.6157877, 'updated_records': [], 'warnings': []}�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - �[1mOpenMetadata Status:�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - {'failures': [], 'filtered': [], 'records': ['Table [dim_branch]'], 'source_start_time': 1721968846.6467316, 'updated_records': [], 'warnings': []}�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - �[1mExecution Time Summary�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO -
�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - �[1mWorkflow OpenMetadata Service Summary:�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Processed records: 1�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Updated records: 0�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Warnings: 0�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Filtered: 339�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Errors: 0�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - �[1mWorkflow Profiler Summary:�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Processed records: 1�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Updated records: 0�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Warnings: 0�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Errors: 0�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - �[1mWorkflow PII Processor Summary:�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Processed records: 1�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Updated records: 0�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Warnings: 0�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Errors: 0�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - �[1mWorkflow OpenMetadata Summary:�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Processed records: 1�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Updated records: 0�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Warnings: 0�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - Errors: 0�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - �[1m�[36;1mSuccess %: 100.0�[0m
[2024-07-26, 04:41:19 UTC] {logger.py:184} INFO - �[1m�[36;1mWorkflow finished in time: 32.83s�[0m
[2024-07-26, 04:41:19 UTC] {python.py:194} INFO - Done. Returned value was: None
[2024-07-26, 04:41:19 UTC] {taskinstance.py:1400} INFO - Marking task as SUCCESS. dag_id=80ac9fa6-d88c-4a0b-82e7-8426d9fbe6eb, task_id=profiler_task, execution_date=20240726T044044, start_date=20240726T044046, end_date=20240726T044119
[2024-07-26, 04:41:19 UTC] {local_task_job_runner.py:228} INFO - Task exited with return code 0
[2024-07-26, 04:41:19 UTC] {taskinstance.py:2778} INFO - 0 downstream tasks scheduled from follow-on schedule check

[Prajwal214]

The issue with ingesting sample data from partitioned tables in BigQuery is likely due to the way OpenMetadata handles partitioned tables. According to the OpenMetadata documentation, you need to configure the profiler settings specifically for partitioned tables.
Here are the steps to ensure sample data is generated for partitioned tables:

1. Enable Partitioning in Profiler Settings: Go to the table's profiler settings and enable partitioning. Specify the partition column and the interval type (e.g., TIME-UNIT, INGESTION-TIME, COLUMN-VALUE, INTEGER-RANGE).
2. Set Interval and Unit: Define the interval and interval unit for the partitioning logic. For example, if your table is partitioned by date, set the interval type to TIME-UNIT and the interval unit to DAY.
3. Re-run the Profiler Workflow: After updating the profiler settings, re-run the profiler workflow to generate sample data for the partitioned table.