4.3.10

Key Highlights

• Added a new property spark.cdm.trackRun.runId to support a custom unique identifier for the current run. This can be used by wrapper scripts to pass a known runId and then use it to query the cdm_run_info and cdm_run_details tables.

Jar file can also be found in the packages section here.

4.3.9

Key Highlights

• Added new status value of DIFF_CORRECTED on cdm_run_details table to specifically mark partitions that were corrected during the CDM validation run.
• Upgraded Validation job to skip partitions with DIFF_CORRECTED status on rerun with a previous runId.

Jar file can also be found in the packages section here.

4.3.8

Key Highlights

• Upgraded spark.cdm.trackRun feature to include status on cdm_run_info table.
• Code fixes to handle rerun of previous run which may have exited before being correctly initialized.

Jar file can also be found in the packages section here.

4.3.7

Key Highlights

• Added property spark.cdm.transform.custom.ttl to allow a custom constant value to be set for TTL instead of using the values from origin rows.
• Repo wide code formating & imports organization

Jar file can also be found in the packages section here.

4.3.6

Key Highlights

• Added overwrite option to conditionally check or skip Validation when it has a non-null value in target for the spark.cdm.feature.extractJson feature.

Jar file can also be found in the packages section here.

4.3.5

Key Highlights

• Added feature spark.cdm.feature.extractJson which allows you to extract a json value from a column with json content in an Origin table and map it to a column in the Target table.
• Upgraded to use Spark 3.5.2.

Jar file can also be found in the packages section here.

4.3.4

Key Highlights

• Minor bug fix (Use origin keyspace/table value when target value is empty/missing) to the regression introduced in 4.3.3.

Jar file can also be found in the packages section here.

4.3.3

Key Highlights

• Removed partitions file feature (which was deprecated in the prior versions).
• Minor bug fix (Enable tracking when only previousRunId provided, but trackRun not set to true).
• Upgraded Spark Cassandra Connector (SCC) version to 3.5.1.

Jar file can also be found in the packages section here.

4.3.2

• Removed deprecated functionality related to retry

Jar file can also be found in the packages section here.

4.3.1

• Fixed a validation run [bug] (#266) that sometimes did not report a failed token-range
• Removed deprecated MigrateRowsFromFile job

Jar file can also be found in the packages section here.