Skip to content

Releases: palantir/spark

3.0.0-palantir.70

07 May 10:41
e4f64d3
Compare
Choose a tag to compare
  • Infer Pandas string columns in Arrow conversion on Python 2 (#679)

3.0.0-palantir.69

01 May 10:36
da8a7ff
Compare
Choose a tag to compare
  • Change AQE default minPartitionNum to defaultParallelism (#674)
  • Upgrade Jackson to 2.11 (#675)
  • Support java-library dependency declaration in SparkDockerPlugin (#630)

3.0.0-palantir.68

28 Apr 11:01
7068b6c
Compare
Choose a tag to compare
  • Upgrade parquet to 1.12.0-palantir.7 (#666)

3.0.0-palantir.67

07 Apr 15:18
73e6e6f
Compare
Choose a tag to compare
  • Support refreshing CondaEnvironment auth in specfile mode (#669)

3.0.0-palantir.66

10 Mar 21:25
9903e96
Compare
Choose a tag to compare
  • [SPARK-26837][SQL] Pruning nested fields from object serializers (#663)
  • [SPARK-26865][SQL] DataSourceV2Strategy should push normalized filters (#648)
  • [SPARK-25299] Add an implementation of the SPARK-25299 shuffle storage plugin that asynchronously backs up shuffle data to remote storage (#646)

3.0.0-palantir.65

13 Feb 21:54
b5c101e
Compare
Choose a tag to compare
  • Split image name into snapshot and release registries, with a separate path parameter (#645)

3.0.0-palantir.64

07 Feb 11:35
92fdc23
Compare
Choose a tag to compare
Not print conda environment  when executing conda process (#644)

Not print conda environment as spark doesn't have arg logging and the env can therefore contain unsafe information.

3.0.0-palantir.63

04 Feb 23:04
55a26e9
Compare
Choose a tag to compare
  • [SPARK-28699][SQL] Disable using radix sort for ShuffleExchangeExec in repartition case (#640)

3.0.0-palantir.62

30 Jan 22:20
1501ead
Compare
Choose a tag to compare
  • Rename the configuration to set the shuffle storage plugin class (#637)

3.0.0-palantir.61

29 Jan 19:17
1642de6
Compare
Choose a tag to compare
  • [SPARK-27083][SQL] Add a new conf to control subqueryReuse
  • [SPARK-27393][SQL] Show ReusedSubquery in the plan when the subquery is reused
  • [SPARK-27747][SQL] add a logical plan link in the physical plan
  • [SPARK-27816][SQL] make TreeNode tag type safe
  • [SPARK-28057][SQL] Add method clone in catalyst TreeNode
  • [SPARK-20774][SPARK-27036][SQL] Cancel the running broadcast execution on BroadcastTimeout
  • [SPARK-23128][SQL] A new approach to do adaptive execution in Spark SQL
  • [SPARK-28177][SQL] Adjust post shuffle partition number in adaptive execution
  • [SPARK-28339][SQL] Rename Spark SQL adaptive execution configuration name
  • [SPARK-28356][SQL] Do not reduce the number of partitions for repartition in adaptive execution
  • [SPARK-28356][SHUFFLE][FOLLOWUP] Fix case with different pre-shuffle partition numbers
  • [SPARK-28237][SQL] Enforce Idempotence for Once batches in RuleExecutor
  • [SPARK-28583][SQL] Subqueries should not call onUpdatePlan in Adaptive Query Execution
  • [SPARK-28753][SQL] Dynamically reuse subqueries in AQE
  • [SPARK-28739][SQL] Add a simple cost check for Adaptive Query Execution
  • [SPARK-26840][SQL] Avoid cost-based join reorder in presence of join hints
  • [SPARK-27225][SQL] Implement join strategy hints
  • [SPARK-27783][SQL] Add customizable hint error handler
  • [SPARK-29002][SQL] Avoid changing SMJ to BHJ if the build side has a has a high ratio of empty partitions
  • [SPARK-29060][SQL] Add tree traversal helper for adaptive spark plans
  • [SPARK-27236][TEST] Refactor log-appender pattern in tests
  • [SPARK-28532][SQL] Make optimizer batch "subquery" FixedPoint(1)
  • [SPARK-28530][SQL] Cost-based join reorder optimizer batch should be FixedPoint(1)
  • [SPARK-28532][SPARK-28530][SQL][FOLLOWUP] Inline doc for FixedPoint(1) batches "Subquery" and "Join Reorder"
  • [SPARK-29906][SQL] AQE should not introduce extra shuffle for outermost limit
  • [SPARK-27824][SQL] Make rule EliminateResolvedHint idempotent
  • [SPARK-28306][SQL] Make NormalizeFloatingNumbers rule idempotent
  • [SPARK-28306][SQL][FOLLOWUP] Fix NormalizeFloatingNumbers rule idempotence for equi-join with <=> predicates