[native] Add row expression optimizer #22927

pramodsatya · 2024-06-05T15:59:54Z

Description

Introduces capability to optimize and constant fold row expressions in the Presto native sidecar.

Motivation and Context

Please refer to #24126 for full context of the changes, this is as described in RFC-0006.

Test Plan

Unit tests for simple cases are added in RowExpressionOptimizerTest.cpp. End to end tests will be added as in the PR #24126.

Release Notes

== NO RELEASE NOTE ==

aditi-pandit

@pramodsatya : Have done a first round of comments. Will read your tests more closely once you address the comments here.

presto-native-execution/presto_cpp/main/PrestoServer.cpp

presto-native-execution/presto_cpp/main/expression/RowExpressionEvaluator.h

presto-native-execution/presto_cpp/main/expression/RowExpressionEvaluator.cpp

presto-native-execution/presto_cpp/main/expression/RowExpressionEvaluator.h

presto-native-execution/presto_cpp/main/types/PrestoToVeloxExpr.h

presto-native-execution/presto_cpp/main/expression/RowExpressionEvaluator.cpp

presto-native-execution/presto_cpp/main/expression/tests/RowExpressionEvaluatorTest.cpp

pramodsatya

Thanks for the feedback @aditi-pandit, addressed the comments. Could you please take another look?

presto-native-execution/presto_cpp/main/expression/RowExpressionEvaluator.cpp

pramodsatya · 2024-09-17T02:01:22Z

presto-native-execution/presto_cpp/main/expression/RowExpressionEvaluator.cpp

+void RowExpressionEvaluator::evaluateExpression(
+    const std::vector<std::unique_ptr<folly::IOBuf>>& body,
+    proxygen::ResponseHandler* downstream) {
+  try {


Thanks for the suggestion, added helper functions and updated accordingly. I could not find a similar utility function in the codebase. Could you please take another look?

presto-native-execution/presto_cpp/main/expression/RowExpressionEvaluator.cpp

presto-native-execution/presto_cpp/main/expression/tests/RowExpressionEvaluatorTest.cpp

Summary: prestodb/presto#23331 adds a native expression optimizer to delegate expression evaluation to the native sidecar. This is used to constant fold expressions on the presto native sidecar, instead of on the presto java coordinator (which is the current behavior). prestodb/presto#22927 implements a proxygen endpoint to accept `RowExpression`s from `NativeSidecarExpressionInterpreter`, optimize them if possible (rewrite special form expressions), and compile the `RowExpression` to a velox expression with constant folding enabled. This velox expression is then converted back to a `RowExpression` and returned by the sidecar to the coordinator. When the constant folded velox expression is of type `velox::exec::ConstantExpr`, we need to return a `RowExpression` of type `ConstantExpression`. This requires us to serialize the constant value from `velox::exec::ConstantExpr` into `protocol::ConstantExpression::valueBlock`. This can be done by serializing the constant value vector to presto SerializedPage::column format, followed by base 64 encoding the result (reverse engineering the logic from `Base64Util.cpp::readBlock`). This PR adds a new function, `serializeSingleColumn`, to `PrestoVectorSerde`. This can be used to serialize input data from vectors containing a single element into a single PrestoPage column format (without the PrestoPage header). This function is not added to `PrestoBatchVectorSerializer` alongside the existing `serialize` function since that would require adding it as a virtual function in `BatchVectorSerializer` as well, and this is not desirable since the `PrestoPage` format is not relevant in this base class. There is an existing function `deserializeSingleColumn` in `PrestoVectorSerde` which is used to deserialize data from a single column, since `serializeSingleColumn` performs the inverse operation to this function, it is added alongside it in `PrestoVectorSerde`. Pull Request resolved: #10657 Reviewed By: amitkdutta Differential Revision: D66044754 Pulled By: pedroerp fbshipit-source-id: e509605067920f8207e5a3fa67552badc2ce0eba

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.h

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

aditi-pandit · 2024-12-02T16:24:45Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

+  json res;
+  res["@type"] = "call";
+  protocol::Signature signature;
+  std::string exprName = expr->name();


Should a dynamic_pointer_cast to a CallExpr to ensure it is a correct Velox CallExpression be done ?

The expression here can be either a special form or a call expression so a dynamic_pointer_cast will likely fail here. Also, core::CallExpr derives from core::IExpr and not exec::Expr, so I'm not sure if a dynamic_pointer_cast could be done here. Please let me know if I am missing something.

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

pramodsatya

Thanks for the feedback @aditi-pandit, addressed the review comments. Could you please take another look?

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

pramodsatya · 2024-12-10T05:33:09Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

+  json res;
+  res["@type"] = "call";
+  protocol::Signature signature;
+  std::string exprName = expr->name();


The expression here can be either a special form or a call expression so a dynamic_pointer_cast will likely fail here. Also, core::CallExpr derives from core::IExpr and not exec::Expr, so I'm not sure if a dynamic_pointer_cast could be done here. Please let me know if I am missing something.

aditi-pandit

Thanks @pramodsatya. Have reviewed the RowExpressionConverter class so far.

aditi-pandit · 2025-01-03T18:26:01Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

+  } else {
+    std::vector<TypePtr> childTypes;
+    if (type->isRow()) {
+      typeSignature = "row(";


Would be a bit cleaner to abstract a std::string variable called "complexTypeString" say, and only set it to "row", "array", "map" in the if condition.

Then post the if condition, when assembling typeSignature for the complex type with children, you can start with

typeSignature = complexTypeString + "("; and then add the typeSignatures for children and then closing parentheses.

aditi-pandit · 2025-01-03T18:31:02Z

presto-native-execution/presto_cpp/main/expression/CMakeLists.txt

Would be sufficient to add this code in presto_cpp/main/types along with the other code like VeloxPlanConversion and FunctionMetadata that is used by the native engine. If you prefer, you can add expression as a separate folder under presto_cpp/main/types

aditi-pandit · 2025-01-03T18:32:00Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

+}
+
+json toVariableReferenceExpression(
+    const std::shared_ptr<const exec::FieldReference>& fieldReference) {


Nit rename fieldReference to field

aditi-pandit · 2025-01-03T18:33:49Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

+
+json::array_t getInputExpressions(
+    const std::vector<std::unique_ptr<folly::IOBuf>>& body) {
+  std::ostringstream oss;


There is a function in presto_cpp/main/common/Utils.h called extractMessageBody to get a std::string from the body variable. Please use it here.

aditi-pandit · 2025-01-03T18:36:50Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

Move RowExpressionConverter and RowExpressionOptimizer to separate files as this file has become very big.

Would be also good to write RowExpressionConverterTest separate from RowExpressionOptimizerTest so that their functionality is clear.

aditi-pandit · 2025-01-03T19:08:44Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

+  } else {
+    json::array_t inputArguments = input["arguments"];
+    const auto numInputs = exprInputs.size();
+    VELOX_USER_CHECK_LE(numInputs, inputArguments.size());


Why is this LE and not EQ ?

For coalesce expressions containing NULLs such as:

coalesce(2 * 3 * unbound_long, 1 - 1, null)

The velox expression does not have NULLs whereas the Presto expression has NULL inputs as well:
Velox expression:

coalesce(presto.default.multiply(6:BIGINT, unbound_long), 0:BIGINT)

Presto expression:

{"@type":"special","arguments":[{"@type":"call","arguments":[{"@type":"call","arguments":[{"@type":"call","arguments":[{"@type":"constant","type":"integer","valueBlock":"CQAAAElOVF9BUlJBWQEAAAAAAgAAAA=="},{"@type":"constant","type":"integer","valueBlock":"CQAAAElOVF9BUlJBWQEAAAAAAwAAAA=="}],"displayName":"MULTIPLY","functionHandle":{"@type":"$static","signature":{"argumentTypes":["integer","integer"],"kind":"SCALAR","longVariableConstraints":[],"name":"presto.default.$operator$multiply","returnType":"integer","typeVariableConstraints":[],"variableArity":false}},"returnType":"integer"}],"displayName":"CAST","functionHandle":{"@type":"$static","signature":{"argumentTypes":["integer"],"kind":"SCALAR","longVariableConstraints":[],"name":"presto.default.$operator$cast","returnType":"bigint","typeVariableConstraints":[],"variableArity":false}},"returnType":"bigint"},{"@type":"variable","name":"unbound_long","type":"bigint"}],"displayName":"MULTIPLY","functionHandle":{"@type":"$static","signature":{"argumentTypes":["bigint","bigint"],"kind":"SCALAR","longVariableConstraints":[],"name":"presto.default.$operator$multiply","returnType":"bigint","typeVariableConstraints":[],"variableArity":false}},"returnType":"bigint","sourceLocation":{"column":18,"line":1}},{"@type":"call","arguments":[{"@type":"call","arguments":[{"@type":"constant","type":"integer","valueBlock":"CQAAAElOVF9BUlJBWQEAAAAAAQAAAA=="},{"@type":"constant","type":"integer","valueBlock":"CQAAAElOVF9BUlJBWQEAAAAAAQAAAA=="}],"displayName":"SUBTRACT","functionHandle":{"@type":"$static","signature":{"argumentTypes":["integer","integer"],"kind":"SCALAR","longVariableConstraints":[],"name":"presto.default.$operator$subtract","returnType":"integer","typeVariableConstraints":[],"variableArity":false}},"returnType":"integer"}],"displayName":"CAST","functionHandle":{"@type":"$static","signature":{"argumentTypes":["integer"],"kind":"SCALAR","longVariableConstraints":[],"name":"presto.default.$operator$cast","returnType":"bigint","typeVariableConstraints":[],"variableArity":false}},"returnType":"bigint"},{"@type":"call","arguments":[{"@type":"constant","sourceLocation":{"column":39,"line":1},"type":"unknown","valueBlock":"AwAAAFJMRQEAAAAKAAAAQllURV9BUlJBWQEAAAABgA=="}],"displayName":"CAST","functionHandle":{"@type":"$static","signature":{"argumentTypes":["unknown"],"kind":"SCALAR","longVariableConstraints":[],"name":"presto.default.$operator$cast","returnType":"bigint","typeVariableConstraints":[],"variableArity":false}},"returnType":"bigint","sourceLocation":{"column":39,"line":1}}],"form":"COALESCE","returnType":"bigint","sourceLocation":{"column":18,"line":1}}

LE was used here to cover this case. Changed it to EQ check for non-coalesce expressions.

aditi-pandit · 2025-01-03T19:09:55Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

+  // Presto requires the field form to be in upper case.
+  std::transform(form.begin(), form.end(), form.begin(), ::toupper);
+  res["form"] = form;
+  auto exprInputs = expr->inputs();


Move this variable closer to its use.

aditi-pandit · 2025-01-03T19:10:38Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

+  std::transform(form.begin(), form.end(), form.begin(), ::toupper);
+  res["form"] = form;
+  auto exprInputs = expr->inputs();
+  res["arguments"] = json::array();


Move this line after the comment and just above the if condition that involves setting it.

Moved it inside the else clause closer to it's usage, please let me know if that is fine.

aditi-pandit · 2025-01-03T19:11:30Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

+  return std::make_shared<protocol::ConstantExpression>(cexpr);
+}
+
+// When the second value in the returned pair is true, the arguments for switch


Can you give examples of this ?

I have updated this function to return a struct SwitchFormArguments with distinct fields for these cases, and added examples in comments. Could you PTAL?

aditi-pandit · 2025-01-03T19:13:25Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

+// special form are returned. Otherwise, the switch expression has been
+// simplified and the first value corresponding to the switch case that always
+// evaluates to true is returned.
+std::pair<json::array_t, bool> RowExpressionConverter::getSwitchSpecialFormArgs(


This function is too long. Please break this into smaller more readable pieces.

pramodsatya

Thanks for the suggestions @aditi-pandit. Refactored RowExpressionConverter and RowExpressionOptimizer into separate files in presto_cpp/main/types, and addressed the remaining comments. Could you please take another look?

pramodsatya · 2025-01-06T01:37:47Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

+  std::transform(form.begin(), form.end(), form.begin(), ::toupper);
+  res["form"] = form;
+  auto exprInputs = expr->inputs();
+  res["arguments"] = json::array();


Moved it inside the else clause closer to it's usage, please let me know if that is fine.

pramodsatya · 2025-01-06T18:14:38Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

+  return std::make_shared<protocol::ConstantExpression>(cexpr);
+}
+
+// When the second value in the returned pair is true, the arguments for switch


I have updated this function to return a struct SwitchFormArguments with distinct fields for these cases, and added examples in comments. Could you PTAL?

pramodsatya · 2025-01-06T19:14:14Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

+  } else {
+    json::array_t inputArguments = input["arguments"];
+    const auto numInputs = exprInputs.size();
+    VELOX_USER_CHECK_LE(numInputs, inputArguments.size());


For coalesce expressions containing NULLs such as:

coalesce(2 * 3 * unbound_long, 1 - 1, null)

The velox expression does not have NULLs whereas the Presto expression has NULL inputs as well:
Velox expression:

coalesce(presto.default.multiply(6:BIGINT, unbound_long), 0:BIGINT)

Presto expression:

{"@type":"special","arguments":[{"@type":"call","arguments":[{"@type":"call","arguments":[{"@type":"call","arguments":[{"@type":"constant","type":"integer","valueBlock":"CQAAAElOVF9BUlJBWQEAAAAAAgAAAA=="},{"@type":"constant","type":"integer","valueBlock":"CQAAAElOVF9BUlJBWQEAAAAAAwAAAA=="}],"displayName":"MULTIPLY","functionHandle":{"@type":"$static","signature":{"argumentTypes":["integer","integer"],"kind":"SCALAR","longVariableConstraints":[],"name":"presto.default.$operator$multiply","returnType":"integer","typeVariableConstraints":[],"variableArity":false}},"returnType":"integer"}],"displayName":"CAST","functionHandle":{"@type":"$static","signature":{"argumentTypes":["integer"],"kind":"SCALAR","longVariableConstraints":[],"name":"presto.default.$operator$cast","returnType":"bigint","typeVariableConstraints":[],"variableArity":false}},"returnType":"bigint"},{"@type":"variable","name":"unbound_long","type":"bigint"}],"displayName":"MULTIPLY","functionHandle":{"@type":"$static","signature":{"argumentTypes":["bigint","bigint"],"kind":"SCALAR","longVariableConstraints":[],"name":"presto.default.$operator$multiply","returnType":"bigint","typeVariableConstraints":[],"variableArity":false}},"returnType":"bigint","sourceLocation":{"column":18,"line":1}},{"@type":"call","arguments":[{"@type":"call","arguments":[{"@type":"constant","type":"integer","valueBlock":"CQAAAElOVF9BUlJBWQEAAAAAAQAAAA=="},{"@type":"constant","type":"integer","valueBlock":"CQAAAElOVF9BUlJBWQEAAAAAAQAAAA=="}],"displayName":"SUBTRACT","functionHandle":{"@type":"$static","signature":{"argumentTypes":["integer","integer"],"kind":"SCALAR","longVariableConstraints":[],"name":"presto.default.$operator$subtract","returnType":"integer","typeVariableConstraints":[],"variableArity":false}},"returnType":"integer"}],"displayName":"CAST","functionHandle":{"@type":"$static","signature":{"argumentTypes":["integer"],"kind":"SCALAR","longVariableConstraints":[],"name":"presto.default.$operator$cast","returnType":"bigint","typeVariableConstraints":[],"variableArity":false}},"returnType":"bigint"},{"@type":"call","arguments":[{"@type":"constant","sourceLocation":{"column":39,"line":1},"type":"unknown","valueBlock":"AwAAAFJMRQEAAAAKAAAAQllURV9BUlJBWQEAAAABgA=="}],"displayName":"CAST","functionHandle":{"@type":"$static","signature":{"argumentTypes":["unknown"],"kind":"SCALAR","longVariableConstraints":[],"name":"presto.default.$operator$cast","returnType":"bigint","typeVariableConstraints":[],"variableArity":false}},"returnType":"bigint","sourceLocation":{"column":39,"line":1}}],"form":"COALESCE","returnType":"bigint","sourceLocation":{"column":18,"line":1}}

LE was used here to cover this case. Changed it to EQ check for non-coalesce expressions.

pramodsatya · 2025-01-06T20:02:52Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.cpp

+    if (expr->inputs().empty()) {
+      return toConstantRowExpression(expr);
+    } else {
+      // Inputs to constant expressions are constant, eg: divide(1, 2).


This else condition was added to handle cases like 0 / 0 that would throw if evaluated in Velox. One of the testcases is:

assertOptimizedMatches("if(false, 1, 0 / 0)", "cast(fail(8, 'ignored failure message') as integer)");

Removing this condition causes the test to fail, so the input json was returned unchanged in this case:

com.facebook.presto.spi.PrestoException: / by zero at com.facebook.presto.type.IntegerOperators.divide(IntegerOperators.java:108) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627) at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:649) at com.facebook.presto.sql.InterpretedFunctionInvoker.invoke(InterpretedFunctionInvoker.java:109) at com.facebook.presto.sql.InterpretedFunctionInvoker.invoke(InterpretedFunctionInvoker.java:60) at com.facebook.presto.sql.planner.RowExpressionInterpreter$Visitor.visitCall(RowExpressionInterpreter.java:285) at com.facebook.presto.spi.relation.CallExpression.accept(CallExpression.java:131) at com.facebook.presto.sql.planner.RowExpressionInterpreter.optimize(RowExpressionInterpreter.java:189) at com.facebook.presto.sql.planner.RowExpressionInterpreter.optimize(RowExpressionInterpreter.java:180) at com.facebook.presto.sql.relational.RowExpressionOptimizer.optimize(RowExpressionOptimizer.java:52) at com.facebook.presto.sql.relational.DelegatingRowExpressionOptimizer.optimize(DelegatingRowExpressionOptimizer.java:78)

I have updated the comment with this example. Please let me know whether this check is fine or should be modified.

pramodsatya · 2025-01-06T20:06:15Z

presto-native-execution/presto_cpp/main/expression/RowExpressionOptimizer.h

+      const exec::ExprPtr& expr,
+      const json& inputRowExpr);
+
+ protected:


No this was unintentional, modified access spec to private.

pramodsatya · 2025-01-06T20:07:25Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxExpr.cpp

+  for (const auto& entry : kPrestoOperatorMap) {
+    veloxToPrestoOperatorMap[entry.second] = entry.first;
+  }
+  veloxToPrestoOperatorMap.insert({"cast", "presto.default.$operator$cast"});


This is needed for the cast e2e tests, it was added separately so the existing map kPrestoOperatorMap is not modified. It is also needed only in this function veloxToPrestoOperatorMap() which returns an inverse map, and is not needed in the original kPrestoOperatorMap.
Please let me know if this is fine or whether it should also be added in kPrestoOperatorMap.

aditi-pandit · 2025-01-07T22:25:30Z

presto-native-execution/presto_cpp/main/PrestoServer.cpp

@@ -1546,6 +1546,26 @@ void PrestoServer::registerSidecarEndpoints() {
         proxygen::ResponseHandler* downstream) {
        http::sendOkResponse(downstream, getFunctionsMetadata());
      });
+  rowExpressionOptimizer_ =


Might be better to have a single global function for RowExpressionOptimizer->optimize for registration here (It could construct a RowExpressionOptimizer internally though). Lets avoid constructing the object here.

aditi-pandit · 2025-01-07T22:28:12Z

presto-native-execution/presto_cpp/main/types/PrestoToVeloxExpr.cpp

+  for (const auto& entry : kPrestoOperatorMap) {
+    veloxToPrestoOperatorMap[entry.second] = entry.first;
+  }
+  veloxToPrestoOperatorMap.insert({"cast", "presto.default.$operator$cast"});


If this is for specfic test functions, then it might be better to make this change in the test function logic instead of here. Its not particularly server side logic then.

aditi-pandit · 2025-01-07T22:30:01Z

presto-native-execution/presto_cpp/main/types/RowExpressionConverter.h

+// case that can be simplified since 'a' is a variable here, so the WHEN clauses
+// that are required by Presto as switch expression arguments are returned in
+// the field 'arguments'.
+struct SwitchFormArguments {


This needn't be defined in the header here. It seems to have local usage only in the cpp file. Please move it there.

aditi-pandit · 2025-01-07T22:30:49Z

presto-native-execution/presto_cpp/main/types/RowExpressionConverter.h

+};
+
+// Helper class to convert Velox expressions of type exec::Expr to their
+// corresponding type of RowExpression in Presto.


These are protocol expressions right ? That is an external format defined in Presto. Please state that explicitly.

aditi-pandit · 2025-01-07T22:32:29Z

presto-native-execution/presto_cpp/main/types/RowExpressionConverter.h

+class RowExpressionConverter {
+ public:
+  explicit RowExpressionConverter(memory::MemoryPool* pool)
+      : pool_(pool), veloxToPrestoOperatorMap_(veloxToPrestoOperatorMap()) {}


Any reason you are usign veloxToPrestoOperatorMap_ as a member variable ? You can always call the veloxToPrestoOperatorMap() function directly ? Or add a local function that returns a static variable.

aditi-pandit · 2025-01-08T03:49:25Z

presto-native-execution/presto_cpp/main/types/RowExpressionOptimizer.cpp

+        resultJson = rowExpressionConverter_.veloxToPrestoRowExpression(
+            compiledExpr, input[i]);
+      } else {
+        // Velox does not evaluate expressions that are non-deterministic during


Abstract the else part in a separate function.

aditi-pandit · 2025-01-08T03:56:16Z

presto-native-execution/presto_cpp/main/types/tests/RowExpressionOptimizerTest.cpp

+  std::unique_ptr<expression::RowExpressionConverter> rowExpressionConverter_;
+};
+
+TEST_F(RowExpressionConverterTest, constant) {


Lets add more tests for constants of all simple types and few constants of complex types.

aditi-pandit · 2025-01-08T03:56:50Z

presto-native-execution/presto_cpp/main/types/tests/RowExpressionOptimizerTest.cpp

+TEST_F(RowExpressionConverterTest, variable) {
+  auto field = std::make_shared<exec::FieldReference>(
+      VARCHAR(), std::vector<exec::ExprPtr>{}, "c0");
+  auto result =


Would be great to add tests for field references for complex types as well.

aditi-pandit · 2025-01-08T03:57:40Z

presto-native-execution/presto_cpp/main/types/tests/RowExpressionOptimizerTest.cpp

+// Velox expression to Presto RowExpression conversion for different types of
+// expressions can be found in TestDelegatingExpressionOptimizer.java in
+// presto-native-sidecar-plugin.
+class RowExpressionConverterTest


Separate RowExpressionConverterTest to a separate file as well.

aditi-pandit · 2025-01-08T03:58:21Z

presto-native-execution/presto_cpp/main/types/tests/RowExpressionOptimizerTest.cpp

+  EXPECT_EQ(result, json::parse(expected));
+}
+
+TEST_F(RowExpressionConverterTest, variable) {


Add more tests for call expressions and special forms for RowExpressionConverter as well.

…bator#10657) Summary: prestodb/presto#23331 adds a native expression optimizer to delegate expression evaluation to the native sidecar. This is used to constant fold expressions on the presto native sidecar, instead of on the presto java coordinator (which is the current behavior). prestodb/presto#22927 implements a proxygen endpoint to accept `RowExpression`s from `NativeSidecarExpressionInterpreter`, optimize them if possible (rewrite special form expressions), and compile the `RowExpression` to a velox expression with constant folding enabled. This velox expression is then converted back to a `RowExpression` and returned by the sidecar to the coordinator. When the constant folded velox expression is of type `velox::exec::ConstantExpr`, we need to return a `RowExpression` of type `ConstantExpression`. This requires us to serialize the constant value from `velox::exec::ConstantExpr` into `protocol::ConstantExpression::valueBlock`. This can be done by serializing the constant value vector to presto SerializedPage::column format, followed by base 64 encoding the result (reverse engineering the logic from `Base64Util.cpp::readBlock`). This PR adds a new function, `serializeSingleColumn`, to `PrestoVectorSerde`. This can be used to serialize input data from vectors containing a single element into a single PrestoPage column format (without the PrestoPage header). This function is not added to `PrestoBatchVectorSerializer` alongside the existing `serialize` function since that would require adding it as a virtual function in `BatchVectorSerializer` as well, and this is not desirable since the `PrestoPage` format is not relevant in this base class. There is an existing function `deserializeSingleColumn` in `PrestoVectorSerde` which is used to deserialize data from a single column, since `serializeSingleColumn` performs the inverse operation to this function, it is added alongside it in `PrestoVectorSerde`. Pull Request resolved: facebookincubator#10657 Reviewed By: amitkdutta Differential Revision: D66044754 Pulled By: pedroerp fbshipit-source-id: e509605067920f8207e5a3fa67552badc2ce0eba

pramodsatya force-pushed the expr_endpt branch from e8f3c68 to 418b219 Compare July 25, 2024 21:49

pramodsatya changed the title ~~[WIP] Add proxygen endpoint for expression evaluation~~ [native] Add proxygen endpoint for expression evaluation Aug 5, 2024

pramodsatya mentioned this pull request Aug 5, 2024

Add function serializeSingleColumn to PrestoVectorSerde facebookincubator/velox#10657

Closed

pramodsatya force-pushed the expr_endpt branch from 418b219 to 50d6456 Compare August 5, 2024 08:32

pramodsatya requested review from aditi-pandit, tdcmeehan and czentgr August 5, 2024 08:33

tdcmeehan self-assigned this Aug 5, 2024

pramodsatya force-pushed the expr_endpt branch 2 times, most recently from 822f79f to 2352cd4 Compare September 11, 2024 14:52

aditi-pandit reviewed Sep 12, 2024

View reviewed changes

pramodsatya force-pushed the expr_endpt branch from 2352cd4 to 496bbc5 Compare September 17, 2024 02:41

pramodsatya commented Sep 17, 2024

View reviewed changes

pramodsatya force-pushed the expr_endpt branch from 496bbc5 to d9c3dcd Compare September 17, 2024 03:09

pramodsatya force-pushed the expr_endpt branch from d9c3dcd to e19ae51 Compare October 7, 2024 15:05

pramodsatya force-pushed the expr_endpt branch from e19ae51 to 747ef08 Compare November 4, 2024 13:40

pramodsatya force-pushed the expr_endpt branch from 747ef08 to 10bfc08 Compare December 2, 2024 02:40

pramodsatya requested a review from aditi-pandit December 2, 2024 02:40

aditi-pandit reviewed Dec 2, 2024

View reviewed changes

pramodsatya changed the title ~~[native] Add proxygen endpoint for expression evaluation~~ [native] Add row expression optimizer Dec 10, 2024

pramodsatya force-pushed the expr_endpt branch from 10bfc08 to 37ef905 Compare December 10, 2024 08:05

pramodsatya commented Dec 10, 2024

View reviewed changes

pramodsatya force-pushed the expr_endpt branch from 37ef905 to 7425558 Compare December 10, 2024 17:17

pramodsatya marked this pull request as ready for review December 10, 2024 17:49

pramodsatya requested a review from a team as a code owner December 10, 2024 17:49

prestodb-ci added the from:IBM PR from IBM label Dec 10, 2024

prestodb-ci requested review from a team and imjalpreet and removed request for a team December 10, 2024 17:49

pramodsatya force-pushed the expr_endpt branch 4 times, most recently from 416d768 to b244385 Compare December 11, 2024 17:28

pramodsatya requested a review from aditi-pandit December 17, 2024 07:38

aditi-pandit reviewed Jan 3, 2025

View reviewed changes

pramodsatya force-pushed the expr_endpt branch 2 times, most recently from 6297ece to 7cbdc9c Compare January 6, 2025 23:31

pramodsatya commented Jan 7, 2025

View reviewed changes

[native] Add row expression optimizer

ad0a9e2

pramodsatya force-pushed the expr_endpt branch from 7cbdc9c to ad0a9e2 Compare January 8, 2025 00:11

aditi-pandit reviewed Jan 8, 2025

View reviewed changes

[native] Add row expression optimizer #22927

Are you sure you want to change the base?

[native] Add row expression optimizer #22927

Conversation

pramodsatya commented Jun 5, 2024 • edited Loading

Description

Motivation and Context

Test Plan

Release Notes

aditi-pandit left a comment

Choose a reason for hiding this comment

pramodsatya left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pramodsatya left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

aditi-pandit left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pramodsatya left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pramodsatya commented Jun 5, 2024 •

edited

Loading