fix(iceberg): Date partition value parse issue #12126

nmahadevuni · 2025-01-20T19:22:02Z

Iceberg partition values are already in daysSinceEpoch, but in velox we assume its in date form and try to convert as with Hive. Fixed this.

netlify · 2025-01-20T19:22:31Z

✅ Deploy Preview for meta-velox canceled.

Name	Link
🔨 Latest commit	`67b025c`
🔍 Latest deploy log	https://app.netlify.com/sites/meta-velox/deploys/67972e8d0283d70008b14117

czentgr · 2025-01-21T14:46:52Z

velox/connectors/hive/HiveConnectorUtil.cpp

-    return applyFilter(*filter, result.value());
+    int32_t result = 0;
+    if (tableFormat == SplitReader::TableFormat::kIceberg) {
+      result = boost::lexical_cast<int32_t>(partitionValue.c_str());


Can we not use std::stoi here instead of including a boost header?
In other comments i see that this function is slow. This could be a problem?

Also we wouldn't need a new include and add a new dependency here.

std::stoi is converting a string like "2022-04-05" to int value 2022, so its not safe and may lead to wrong results.

std::stoi should work here. https://www.geeksforgeeks.org/convert-string-to-int-in-cpp/

std::stoi will work for converting int string to int. But it also doesn't throw error if we input a string like "2022-04-05" as it just converts it to 2022.

Well, 2022-04-05 is parsed by boost into what number? It is not a valid integer in the first place. So either function will not work.

Also you are forgetting that the string is actually the days since epoch like you have in your description. Which means it is not an actual date formatted string. If this was the case you need a date parser here and not a string to int parse function in the first place.

changed to use std::stoi

aditi-pandit

@nmahadevuni : Thanks for this fix. Have bunch of review comments.

aditi-pandit · 2025-01-21T20:21:03Z

velox/connectors/hive/HiveConnectorUtil.cpp

-    VELOX_CHECK(!result.hasError());
-    return applyFilter(*filter, result.value());
+    int32_t result = 0;
+    if (tableFormat == SplitReader::TableFormat::kIceberg) {


Please add a comment here about this behavior.

Also please add to the PR description.

aditi-pandit · 2025-01-22T03:13:01Z

velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp

+          partitionKeys = {},
+      const std::vector<std::string> filters = {},
+      const std::string duckDbSql = "",
+      const int32_t numPrefetchSplits = 0) {


Why do you have this parameter to the function ? Since it seems like we never test its use really.

Removed this.

aditi-pandit · 2025-01-22T03:15:28Z

velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp

+  partitionKeys["ds"] = "17627";
+
+  std::vector<RowVectorPtr> dataVectors;
+  VectorPtr c0 = vectorMaker_.flatVector<int64_t>({1});


vectorMaker_ is deprecated. Use makeFlatVector API instead.

aditi-pandit · 2025-01-22T03:18:22Z

velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp

@@ -477,6 +514,15 @@ class HiveIcebergTest : public HiveConnectorTestBase {
    return PlanBuilder(pool_.get()).tableScan(rowType_).planNode();
  }

+  core::PlanNodePtr tableScanNode(


Don't think there is a need for this function as its only used in a single place and doesn't really represent anything.

aditi-pandit · 2025-01-22T03:27:32Z

velox/connectors/hive/SplitReader.cpp

  using T = typename TypeTraits<kind>::NativeType;
  if (!value.has_value()) {
    return std::make_shared<ConstantVector<T>>(pool, size, true, type, T());
  }

  if (type->isDate()) {
-    auto days = DATE()->toDays(static_cast<folly::StringPiece>(value.value()));
+    int32_t days = 0;


This seems like the wrong level of abstraction to fix this problem especially if this is a generic function.

We should add this logic in SplitReader::setPartitionValue directly.

Since all these are Hive/Iceberg util functions, this should be ok? Because this issue happens in three cases, and for only this one case we can move the logic to SplitReader::setPartitionValue and in the other cases, we cannot.

Do you anticipate this conversion for other info columns as well ?

The name of this function was too generic.. but when I look at the uses it is only for filter values. I feel we can change this function name to filterValueFromString and then its application seems more restrictive.

What are the 3 cases ?

Rechecked, its in 2 places. I see this conversion happens from SplitReader::setPartitionValue() and SplitReader::filterOnStats(). I think the current name is appropriate as its not only used for filter.

aditi-pandit · 2025-01-22T03:31:19Z

velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp

+      const std::unordered_map<std::string, std::optional<std::string>>
+          partitionKeys = {},
+      const std::vector<std::string> filters = {},
+      const std::string duckDbSql = "",


It is not reasonable to have an empty duckDbSql for this function, as that is the main sql string to verify the plan results with. Since we know the plan we are generating, it might be better to build the duckDBSql in this function itself.

For this test case, we don't know how to generate the duckDbSql, when we add more test cases, we can enhance this function.

I added an empty string check.

In this code a very specific plan with a TableScanNode and partitionfilters, columnfilters is setup. This maps to a quite precise Sql. We could just build it in the logic.

But I'm also fine with the sql passed as a parameter.

We should remove the default "" value for this parameter.

In this test case, we generate int vector for date values and create the data file which is ok for velox, but cannot create duckdb table with the same vectors, so using a sql statement to verify.

aditi-pandit · 2025-01-22T04:37:34Z

velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp

+    auto scanNodeId = plan->id();
+    auto it = planStats.find(scanNodeId);
+    ASSERT_TRUE(it != planStats.end());
+    ASSERT_TRUE(it->second.peakMemoryBytes > 0);


Why are you testing this ? Isn't matching results sufficient ?

not required, removed it.

aditi-pandit · 2025-01-22T06:49:24Z

velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp

@@ -225,6 +226,41 @@ class HiveIcebergTest : public HiveConnectorTestBase {
    ASSERT_TRUE(it->second.peakMemoryBytes > 0);
  }

+  void assertQuery(


Maybe rename this function to assertPartitionKey.

Want to keep this name generic, since it could be used to test any case.

This method is building a very specific plan with TableScanNode and IcebergSplits... The assertQuery function name is being used in all the TestBase classes for very generic usage.

majetideepak · 2025-01-23T00:58:25Z

velox/connectors/hive/HiveConnectorUtil.cpp

+    if (tableFormat == SplitReader::TableFormat::kIceberg) {
+      result = boost::lexical_cast<int32_t>(partitionValue.c_str());
+    } else {
+      result = DATE()->toDays((folly::StringPiece)partitionValue);


please fix this to use C++ cast since we are touching this code.

Sorry, didn't get it. toDays converts date into daysSinceEpoch, where to use C++ cast?

The C++ style cast:

static_cast<folly::StringPiece>(partitionValue)

See the example in a line you actually removed (below in the review).

Added C++ cast in both places.

majetideepak · 2025-01-23T00:59:49Z

velox/connectors/hive/SplitReader.h

@@ -154,6 +157,7 @@ class SplitReader {
      std::shared_ptr<HiveColumnHandle>>* const partitionKeys_;
  const ConnectorQueryCtx* connectorQueryCtx_;
  const std::shared_ptr<const HiveConfig> hiveConfig_;
+  const TableFormat tableFormat_;


don't need a const for enums and scalars.

I made it a const reference.

Are we missing something? This is just a plain const and not a const reference. Also for enums you don't need a reference. And the argument to the constructor is not a const & either.

Sorry, I was looking at a different place altogether. You are right. I removed all const references.

aditi-pandit · 2025-01-23T05:35:41Z

velox/connectors/hive/HiveConnectorUtil.cpp

@@ -634,12 +636,16 @@ namespace {
 bool applyPartitionFilter(
    const TypePtr& type,
    const std::string& partitionValue,
-    common::Filter* filter) {
+    common::Filter* filter,
+    const SplitReader::TableFormat& tableFormat) {


Its better to move the read-only const& parameters before the writable * ones. So lets move tableFormat to the first parameter.

aditi-pandit · 2025-01-23T05:47:53Z

velox/connectors/hive/SplitReader.cpp

  using T = typename TypeTraits<kind>::NativeType;
  if (!value.has_value()) {
    return std::make_shared<ConstantVector<T>>(pool, size, true, type, T());
  }

  if (type->isDate()) {
-    auto days = DATE()->toDays(static_cast<folly::StringPiece>(value.value()));
+    int32_t days = 0;


Do you anticipate this conversion for other info columns as well ?

The name of this function was too generic.. but when I look at the uses it is only for filter values. I feel we can change this function name to filterValueFromString and then its application seems more restrictive.

What are the 3 cases ?

aditi-pandit · 2025-01-23T05:51:47Z

velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp

+      const std::unordered_map<std::string, std::optional<std::string>>
+          partitionKeys = {},
+      const std::vector<std::string> filters = {},
+      const std::string duckDbSql = "",


In this code a very specific plan with a TableScanNode and partitionfilters, columnfilters is setup. This maps to a quite precise Sql. We could just build it in the logic.

But I'm also fine with the sql passed as a parameter.

aditi-pandit · 2025-01-23T05:52:19Z

velox/connectors/hive/iceberg/tests/IcebergReadTest.cpp

+      const std::unordered_map<std::string, std::optional<std::string>>
+          partitionKeys = {},
+      const std::vector<std::string> filters = {},
+      const std::string duckDbSql = "",


We should remove the default "" value for this parameter.

nmahadevuni · 2025-01-24T06:55:45Z

Thank you @aditi-pandit @majetideepak @czentgr . I have addressed your comments. Please review.

nmahadevuni requested a review from majetideepak as a code owner January 20, 2025 19:22

facebook-github-bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Jan 20, 2025

nmahadevuni requested review from aditi-pandit and czentgr January 21, 2025 10:26

czentgr reviewed Jan 21, 2025

View reviewed changes

aditi-pandit reviewed Jan 22, 2025

View reviewed changes

majetideepak reviewed Jan 23, 2025

View reviewed changes

aditi-pandit reviewed Jan 23, 2025

View reviewed changes

nmahadevuni force-pushed the fix_ice_date_partition_value_parse branch 2 times, most recently from d3564ed to c7843f4 Compare January 24, 2025 06:55

fix(iceberg): Date partition value parse issue

67b025c

nmahadevuni force-pushed the fix_ice_date_partition_value_parse branch from c7843f4 to 67b025c Compare January 27, 2025 06:58

fix(iceberg): Date partition value parse issue #12126

Are you sure you want to change the base?

fix(iceberg): Date partition value parse issue #12126

Conversation

nmahadevuni commented Jan 20, 2025 • edited Loading

netlify bot commented Jan 20, 2025 • edited Loading

✅ Deploy Preview for meta-velox canceled.

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aditi-pandit left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nmahadevuni commented Jan 24, 2025

nmahadevuni commented Jan 20, 2025 •

edited

Loading

netlify bot commented Jan 20, 2025 •

edited

Loading