Add ColumnStatistics::Sum

[gatesn]

Which issue does this PR close?

This PR adds a sum statistic to DataFusion.

Future use will include optimizing aggregation functions (sum, avg, count), see https://github.com/apache/datafusion/pull/13736/files for examples.

Are there any user-facing changes?

The ColumnStatistics struct has an extra public sum_value field.

[alamb]

Thank you @gatesn -- I think this is a nice addition.

It looks like the cargo fmt test is failing

Ideally we would add unit test coverage for Precision::multiply Precision::sub and Precision::cast_to before we merged.

Thanks again -- excited to see this working

FYI @suremarc @berkaysynnada / @ozankabak as this changes statistics and I think you are already working on things related to that:

• Introduce a way to represent constrained statistics / bounds on values in Statistics #8078 (comment)

[alamb]

As I think we mentioned in #13736 my only real concern with this addition is that it will make ColumnStatistics even bigger (each ScalarValue is quite large already and ColumnStatistics are copied a bunch

However, I think the "right" fix for that is to move to using a different statistics representation (e.g. Arc::ColumnStatistics) so I don't see this as a blocker

[gatesn]

@alamb one question I have is whether this should return a Result, or we should assume that a failed cast implies overflow and therefore return Precision::Absent?

The caller (currently in cross-join) unwraps to Absent, I just didn't know whether to internalize that here.

Edit: I decided it was better to propagate the error and allow the caller to decide. It was more useful in a couple of places.

[gatesn]

Added some tests and hopefully appeases the linter!

[berkaysynnada]

FYI @suremarc @berkaysynnada / @ozankabak as this changes statistics and I think you are already working on things related to that:

We've started to refactor. The design is complete, and the implementation is in progress.

I’ve taken a look at this and have some questions. For example, are we planning to add many types of functions to statistics, or is there a defined list of statistics that can be inferred from the sources or have meaningful applications in optimizer rules? If we agree that these kinds of extensions to column statistics are indeed useful and obtainable, then we can proceed with merging this. We would also ensure it is included in the new setup.

FYI @ozankabak

[alamb]

We've started to refactor. The design is complete, and the implementation is in progress.

Thanks! Is there anywhere I can follow along @berkaysynnada (I am particularly interested in what the final API / representation looks like)

[berkaysynnada]

We've started to refactor. The design is complete, and the implementation is in progress.

Thanks! Is there anywhere I can follow along @berkaysynnada (I am particularly interested in what the final API / representation looks like)

I've reached you via discord

[alamb]

We've started to refactor. The design is complete, and the implementation is in progress.

Thanks! Is there anywhere I can follow along @berkaysynnada (I am particularly interested in what the final API / representation looks like)

I've reached you via discord

For anyone else who is interested, the draft PR in the synnada fork is here:

• StatisticsV2: statistics framework initial redesign for Datafusion synnada-ai/datafusion-upstream#57

[gatesn]

Looks like I got hit by some new ColumnStatistics tests on main. Should be fixed now 🤞

@berkaysynnada can you expand on the rationale for the V2 stats? I understand that it's more expressive, but I can't see in the PR or Notion how those distributions might actually be used? Is this for join planning?

My understanding is I would no longer define a "min" or a "max" for a column. But there doesn't seem to be a place to define null count or sum?

[berkaysynnada]

Looks like I got hit by some new ColumnStatistics tests on main. Should be fixed now 🤞

@berkaysynnada can you expand on the rationale for the V2 stats? I understand that it's more expressive, but I can't see in the PR or Notion how those distributions might actually be used? Is this for join planning?

My understanding is I would no longer define a "min" or a "max" for a column. But there doesn't seem to be a place to define null count or sum?

You can still define min or max. We are not replacing Statistics with Statistics_v2; it is actually replacing the Precision and Interval objects. We plan to rename the API of the execution plan from fn statistics(&self) -> Statistics to fn statistics(&self) -> TableStatistics, which is still structured as:

pub struct TableStatistics {
    pub num_rows: Statistics,
    pub total_byte_size: Statistics,
    pub column_statistics: Vec<ColumnStatistics>,
}

and

pub struct ColumnStatistics {
    pub null_count: Statistics,
    pub max_value: Statistics,
    pub min_value: Statistics,
    pub distinct_count: Statistics,
}

What we are trying to address is how the current way of indeterminate quantities are handled in a target-dependent way. For example, if there’s a possibility of indeterminate statistics, it is stored as the mean value when the caller requires an estimate. However, if bounds are required, that indeterminism is stored as an interval.

Our goal is to consolidate all forms of indeterminism and structure them with a strong mathematical foundation. This way, every user can utilize the statistics in their intended way. We aim to preserve and sustain all possible helpful quantities wherever feasible.

We are also constructing a robust evaluation and back-propagation mechanism (similar to interval arithmetic, evaluate_bounds, and propagate_constraints). With this mechanism, any kind of expression—whether projection-based (evaluation only) or filter-based (evaluation followed by propagation)—will automatically resolve using the new statistics.