[BUG][SPARK] Compile performed for files that have been changed being discarded

[Kishan-Kulkarni]

Bug

Which Delta project/connector is this regarding?

• [ x] Spark
• Standalone
• Flink
• Kernel
• Other (fill in here)

Describe the problem

When the working of methods within a class are modified, these classes are not included in the build/sbt compile step, these classes are shown as discarded.Any help on this would be heavily appreciated

Steps to reproduce

1.Clone or download the repo and perform extraction if need be.
2.In terminal navigate to the project directory.
3.Perform build/sbt compile.
4.Observe logs saying those classes have been discarded.

Observed results

Discarding class: io/delta/tables/DeltaTable.class
Discarding class: io/delta/tables/DeltaMergeMatchedActionBuilder.class
Discarding class: io/delta/tables/DeltaMergeMatchedActionBuilder$.class
Discarding class: io/delta/tables/DeltaOptimizeBuilder.class
Discarding class: io/delta/tables/DeltaOptimizeBuilder$.class
Discarding class: io/delta/tables/DeltaColumnBuilder.class

I am currently tweaking DeltaTable.class, the code has no errors and these are not the only classes being discarded and the corresponding delta-spark jar is not produced. The jar however is produced on running build/sbt package but does not contain the changes made.

Environment information

• Delta Lake version:3.3.0
• Spark version:3.5.0
• Scala version:3.4.2

Willingness to contribute

The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?

• Yes. I can contribute a fix for this bug independently.
• [ x] Yes. I would be willing to contribute a fix for this bug with guidance from the Delta Lake community.
• No. I cannot contribute a bug fix at this time.

[Kishan-Kulkarni]

Anyone has any idea on this, any help on this is going to be very helpful.

[felipepessoto]

The discard messages you are seeing are from the delta-iceberg build, to avoid the classes from delta-spark being duplicated in delta-iceberg, because of the shade.

delta/build.sbt

Line 656 in 6495cba

assembly / assemblyMergeStrategy := {

These classes are still compiled as part of delta-spark