Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[native] Add JWT processing when using http(s) comms #20290

Merged
merged 1 commit into from
Oct 4, 2023

Conversation

czentgr
Copy link
Contributor

@czentgr czentgr commented Jul 12, 2023

Adding creation and verification of a JWT signed using HMAC SHA-256.
The JWT is used for authentication of internal communication.
On the Java side PR #19706 added authentication for internal
communication using a JWT token signed with HMAC SHA-256.

Adding this comment enables Prestissimo to to be configured
to verify internal communication requests, e.g from a Java
coordinator.

New system configuration options:

  • internal-communication.jwt.enabled=[true/false]
  • internal-communication.shared-secret=<shared-secret-value>
  • internal-communication.jwt.expiration-seconds=<value in seconds>

A new external dependency jet-cpp from the https://github.com/Thalhammer/jwt-cpp
repo is added to a new setup-adapter.sh script for Prestissimo.
In the dependency directory run
"/scripts/setup-adapter.sh jwt" to install it
The jwt-cpp project handles the creation/parsing/verification of the JWT.

A new build options are added to the build environment:
PRESTO_ENABLE_JWT - default off, if on adds jet creation and verification capability,
turned on during the pipeline build.

Test plan - (Please fill in how you tested your changes)

The PR adds a number of unit tests to the gtest HttpTest suite:

  • good case
  • case when JWT is expired
  • case when the shared secret doesn't match
  • case when the server secret was unset

Resolves issue #19861.

== RELEASE NOTES ==

Prestissimo (Native Execution) Changes
* Add support for internal authentication using JWT. It can be configured using configs "internal-communication.jwt.enabled=[true/false]", 
  "internal-communication.shared-secret=<shared-secret-value>" and "internal-communication.jwt.expiration-seconds=<value in seconds>"

@czentgr czentgr added the MVP Prestissimo MVP label Jul 12, 2023
@czentgr
Copy link
Contributor Author

czentgr commented Jul 12, 2023

TODOs left

  • cleanup more of the unit tests and add more tests (client sends no JWT but server requires one)
  • add E2E test with Java coordinator and prestissimo worker being configured to use JWT - all native E2E tests run with JWT
  • Resolve questions:
  • Replace jwt-cpp base.h with folly::base64 encode/decode implementation? - Yes.
  • Log rejected requests to the log needed? - No. Enabling the access log already shows the request result. If rejected 401 is shown. Similar to this example
I20230715 00:11:19.770259 1057195 AccessLogFilter.cpp:69] 127.0.0.1 - - [2023-07-15 00:11:19] "GET /v1/info/state HTTP/1.1" 401 8   0

@czentgr
Copy link
Contributor Author

czentgr commented Jul 12, 2023

Test results from running HttpTest gtest.

czentgr@Christians-MacBook-Pro tests % ./presto_http_test                          
[==========] Running 21 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 1 test from HttpsBasicTest
[ RUN      ] HttpsBasicTest.ssl
[       OK ] HttpsBasicTest.ssl (28 ms)
[----------] 1 test from HttpsBasicTest (28 ms total)

[----------] 20 tests from HTTPTest/HttpTestSuite
[ RUN      ] HTTPTest/HttpTestSuite.basic/0
[       OK ] HTTPTest/HttpTestSuite.basic/0 (1037 ms)
[ RUN      ] HTTPTest/HttpTestSuite.basic/1
[       OK ] HTTPTest/HttpTestSuite.basic/1 (1030 ms)
[ RUN      ] HTTPTest/HttpTestSuite.httpResponseAllocationFailure/0
E20230714 19:28:20.953740 916573 Exceptions.h:68] Line: /Users/czentgr/gitspace/presto/presto-native-execution/velox/velox/common/memory/MemoryPool.cpp:734, Function:incrementReservationThreadSafe, Expression:  
Exceeded memory pool cap of 1.00KB with max 1.00KB when requesting 1.00MB, memory manager cap is UNLIMITED
default_root_0 usage 0B peak 0B

Failed memory pool: httpResponseAllocationFailure: 0B
, Source: RUNTIME, ErrorCode: MEM_CAP_EXCEEDED
E20230714 19:28:20.955370 916495 Exceptions.h:68] Line: /Users/czentgr/gitspace/presto/presto-native-execution/./presto_cpp/main/http/HttpClient.h:75, Function:consumeBody, Expression: !hasError() , Source: RUNTIME, ErrorCode: INVALID_STATE
[       OK ] HTTPTest/HttpTestSuite.httpResponseAllocationFailure/0 (22 ms)
[ RUN      ] HTTPTest/HttpTestSuite.httpResponseAllocationFailure/1
E20230714 19:28:20.960577 916585 Exceptions.h:68] Line: /Users/czentgr/gitspace/presto/presto-native-execution/velox/velox/common/memory/MemoryPool.cpp:734, Function:incrementReservationThreadSafe, Expression:  
Exceeded memory pool cap of 1.00KB with max 1.00KB when requesting 1.00MB, memory manager cap is UNLIMITED
default_root_1 usage 0B peak 0B

Failed memory pool: httpResponseAllocationFailure: 0B
, Source: RUNTIME, ErrorCode: MEM_CAP_EXCEEDED
E20230714 19:28:20.960752 916495 Exceptions.h:68] Line: /Users/czentgr/gitspace/presto/presto-native-execution/./presto_cpp/main/http/HttpClient.h:75, Function:consumeBody, Expression: !hasError() , Source: RUNTIME, ErrorCode: INVALID_STATE
[       OK ] HTTPTest/HttpTestSuite.httpResponseAllocationFailure/1 (4 ms)
[ RUN      ] HTTPTest/HttpTestSuite.serverRestart/0
[       OK ] HTTPTest/HttpTestSuite.serverRestart/0 (17 ms)
[ RUN      ] HTTPTest/HttpTestSuite.serverRestart/1
[       OK ] HTTPTest/HttpTestSuite.serverRestart/1 (5 ms)
[ RUN      ] HTTPTest/HttpTestSuite.basicJwtTest/0
[       OK ] HTTPTest/HttpTestSuite.basicJwtTest/0 (8 ms)
[ RUN      ] HTTPTest/HttpTestSuite.basicJwtTest/1
[       OK ] HTTPTest/HttpTestSuite.basicJwtTest/1 (2 ms)
[ RUN      ] HTTPTest/HttpTestSuite.asyncRequests/0
[       OK ] HTTPTest/HttpTestSuite.asyncRequests/0 (8 ms)
[ RUN      ] HTTPTest/HttpTestSuite.asyncRequests/1
[       OK ] HTTPTest/HttpTestSuite.asyncRequests/1 (3 ms)
[ RUN      ] HTTPTest/HttpTestSuite.timedOutRequests/0
[       OK ] HTTPTest/HttpTestSuite.timedOutRequests/0 (109 ms)
[ RUN      ] HTTPTest/HttpTestSuite.timedOutRequests/1
[       OK ] HTTPTest/HttpTestSuite.timedOutRequests/1 (107 ms)
[ DISABLED ] HTTPTest/HttpTestSuite.DISABLED_outstandingRequests/0
[ DISABLED ] HTTPTest/HttpTestSuite.DISABLED_outstandingRequests/1
[ RUN      ] HTTPTest/HttpTestSuite.testReportOnBodyStatsFunc/0
[       OK ] HTTPTest/HttpTestSuite.testReportOnBodyStatsFunc/0 (13 ms)
[ RUN      ] HTTPTest/HttpTestSuite.testReportOnBodyStatsFunc/1
[       OK ] HTTPTest/HttpTestSuite.testReportOnBodyStatsFunc/1 (3 ms)
[ RUN      ] HTTPTest/HttpTestSuite.jwtSecretMismatch/0
[       OK ] HTTPTest/HttpTestSuite.jwtSecretMismatch/0 (1005 ms)
[ RUN      ] HTTPTest/HttpTestSuite.jwtSecretMismatch/1
[       OK ] HTTPTest/HttpTestSuite.jwtSecretMismatch/1 (1012 ms)
[ RUN      ] HTTPTest/HttpTestSuite.jwtExpiredToken/0
[       OK ] HTTPTest/HttpTestSuite.jwtExpiredToken/0 (1525 ms)
[ RUN      ] HTTPTest/HttpTestSuite.jwtExpiredToken/1
[       OK ] HTTPTest/HttpTestSuite.jwtExpiredToken/1 (1516 ms)
[ RUN      ] HTTPTest/HttpTestSuite.jwtServerVerificationDisabled/0
[       OK ] HTTPTest/HttpTestSuite.jwtServerVerificationDisabled/0 (526 ms)
[ RUN      ] HTTPTest/HttpTestSuite.jwtServerVerificationDisabled/1
[       OK ] HTTPTest/HttpTestSuite.jwtServerVerificationDisabled/1 (511 ms)
[----------] 20 tests from HTTPTest/HttpTestSuite (8473 ms total)

[----------] Global test environment tear-down
[==========] 21 tests from 2 test suites ran. (8502 ms total)
[  PASSED  ] 21 tests.

  YOU HAVE 2 DISABLED TESTS

@czentgr czentgr force-pushed the cz_add_jwt_usage branch 5 times, most recently from 324dfc8 to 0871a56 Compare July 17, 2023 14:43
@czentgr czentgr marked this pull request as ready for review July 17, 2023 16:23
@czentgr czentgr requested a review from a team as a code owner July 17, 2023 16:23
@czentgr czentgr changed the title [WIP] [native] Add JWT processing when using http(s) comms [native] Add JWT processing when using http(s) comms Jul 17, 2023
@czentgr
Copy link
Contributor Author

czentgr commented Jul 17, 2023

@aditi-pandit @majetideepak Please take a look when you get a chance.

@czentgr czentgr changed the title [native] Add JWT processing when using http(s) comms [WIP][native] Add JWT processing when using http(s) comms Jul 17, 2023
@czentgr czentgr force-pushed the cz_add_jwt_usage branch from 0871a56 to c774be5 Compare July 18, 2023 00:15
Copy link
Contributor

@aditi-pandit aditi-pandit left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Had a quick read and have some very high level comments.


// If the internal communication is not enabled the filter
// is not added to process incoming requests.
void JWTTokenFilter::onRequest(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would you need to add any runtime counters in this logic ?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Counter to count number of bad requests?
Not sure exactly what you mean. The Java code didn't seem to add counters so I didn't consider it.

@czentgr czentgr force-pushed the cz_add_jwt_usage branch from c774be5 to a779518 Compare July 24, 2023 22:36
@czentgr czentgr changed the title [WIP][native] Add JWT processing when using http(s) comms [native] Add JWT processing when using http(s) comms Jul 24, 2023
@czentgr
Copy link
Contributor Author

czentgr commented Jul 25, 2023

The test appears to have run into a connection issue with hive when testing insertion into partitioned tables.

2023-07-24T23:12:50.4633983Z io.prestodb.tempto.query.QueryExecutionException: java.sql.SQLException: Query failed (#20230724_231248_00401_pw3cx): File /tmp/presto-hdfs/8db65272-cdfc-4e00-a2dd-6dc8cbda05a4/returnflag=A/20230724_231248_00401_pw3cx_85c9bd23-df28-4e6e-985d-a6b489f8aa3d could only be replicated to 0 nodes instead of minReplication (=1).  There are 1 datanode(s) running and no node(s) are excluded in this operation.

Two tests failed as a result.
These failures are not related to my changes.

@czentgr czentgr force-pushed the cz_add_jwt_usage branch from a779518 to ed89762 Compare July 25, 2023 20:28
@majetideepak
Copy link
Collaborator

@czentgr I looked at the https://github.com/Thalhammer/jwt-cpp repo and it looks like there are still some changes being merged. How often do we have to update this code?
Can we make that an external dependency instead of checking in the code?
Also, any reason to pin 0.6.0?

@majetideepak
Copy link
Collaborator

Do we know the overhead of JWT?

@czentgr
Copy link
Contributor Author

czentgr commented Jul 26, 2023

Do we know the overhead of JWT?

No, probably would need to add benchmark tests to compare. Overhead is JSON parsing and generating the HS256 signature of the header/payload. It would think it is more or less negligible. The token are not that large and I assume the openssl implementation for the hashing is efficient.

@czentgr
Copy link
Contributor Author

czentgr commented Jul 26, 2023

@czentgr I looked at the https://github.com/Thalhammer/jwt-cpp repo and it looks like there are still some changes being merged. How often do we have to update this code? Can we make that an external dependency instead of checking in the code? Also, any reason to pin 0.6.0?

0.6.0 was the last release. The current one is (was) in release candidate stage (and has been for a while). Perhaps it updated now since I last checked.

I was thinking of adding it to the setup scripts (and had ti running like that) but then saw the json module was also added as code.

Edit: something like Velox does but using CMake to access github and build/install I guess would work. I tried to use that module in the prestissimo CMake and there were some issues with the macros (certain variables related to Velox need to be set).

@czentgr czentgr changed the title [native] Add JWT processing when using http(s) comms [WIP][native] Add JWT processing when using http(s) comms Jul 26, 2023
@czentgr
Copy link
Contributor Author

czentgr commented Jul 26, 2023

As discussed with @majetideepak

  • move the jwt dependency into the setup scripts and remove from the source code
  • keep base64 implementation of jwt-cpp (aka no changes to the project)
  • see how the nlohman-json can be used withiin the jwt-cpp project. It uses the nlohman-json name for the header inclusion while the actual nlohman-json source code was copied as a header and renamed in the externals folder to simply json.h
  • add CMake option (and define) to exclude building the additional code for the usage and verification - by default it is enabled but can be disabled - do some refactoring to make this easy
  • write small benchmark (with the existing benchmark tooling) to see how much overheard adding jwt usage on client and server entails

@czentgr
Copy link
Contributor Author

czentgr commented Aug 18, 2023

Trying to re-use the nlogman jsopn file is not possible. The jwt-cpp will look for

/usr/local/include/jwt-cpp/traits/nlohmann-json/traits.h:5:10: fatal error: 'nlohmann/json.hpp' file not found
#include "nlohmann/json.hpp"

the file in the system path. Without changing the code it is not possible to use the already included file that resides in external/json/json.hpp.

I guess there is a larger discussion on reuse of components and how they are accessed. Perhaps the nlohmann/json should also be a dependency. Dependencies could be installed via the setup scripts or the cmake files (that download and build the project).

@czentgr
Copy link
Contributor Author

czentgr commented Aug 21, 2023

I wrote a small benchmark and the results show that the difference is negligible (10,000 requests). I ran multiple times in my local environment. The time seemed to vary occasionally quite a bit. Here are example outputs:

czentgr@Christians-MacBook-Pro presto-native-execution % _build/release/presto_cpp/main/http/tests/presto_http_jwt_benchmark
WARNING: Benchmark running in DEBUG mode
============================================================================
[...]n/http/tests/HttpJwtBenchmarkTest.cpp     relative  time/iter   iters/s
============================================================================
nojwt                                                       15.95s    62.68m
jwt                                             98.187%     16.25s    61.54m
WARNING: Benchmark running in DEBUG mode
czentgr@Christians-MacBook-Pro presto-native-execution % _build/release/presto_cpp/main/http/tests/presto_http_jwt_benchmark
WARNING: Benchmark running in DEBUG mode
============================================================================
[...]n/http/tests/HttpJwtBenchmarkTest.cpp     relative  time/iter   iters/s
============================================================================
nojwt                                                       16.23s    61.63m
jwt                                             98.678%     16.44s    60.81m
WARNING: Benchmark running in DEBUG mode
czentgr@Christians-MacBook-Pro presto-native-execution % _build/release/presto_cpp/main/http/tests/presto_http_jwt_benchmark
WARNING: Benchmark running in DEBUG mode
============================================================================
[...]n/http/tests/HttpJwtBenchmarkTest.cpp     relative  time/iter   iters/s
============================================================================
nojwt                                                       15.82s    63.20m
jwt                                             98.752%     16.02s    62.41m
WARNING: Benchmark running in DEBUG mode
czentgr@Christians-MacBook-Pro presto-native-execution % _build/release/presto_cpp/main/http/tests/presto_http_jwt_benchmark
WARNING: Benchmark running in DEBUG mode
============================================================================
[...]n/http/tests/HttpJwtBenchmarkTest.cpp     relative  time/iter   iters/s
============================================================================
nojwt                                                       16.03s    62.37m
jwt                                             98.969%     16.20s    61.73m
WARNING: Benchmark running in DEBUG mode
czentgr@Christians-MacBook-Pro presto-native-execution % _build/release/presto_cpp/main/http/tests/presto_http_jwt_benchmark
WARNING: Benchmark running in DEBUG mode
============================================================================
[...]n/http/tests/HttpJwtBenchmarkTest.cpp     relative  time/iter   iters/s
============================================================================
nojwt                                                       15.81s    63.25m
jwt                                             94.488%     16.73s    59.77m
WARNING: Benchmark running in DEBUG mode
czentgr@Christians-MacBook-Pro presto-native-execution % _build/release/presto_cpp/main/http/tests/presto_http_jwt_benchmark
WARNING: Benchmark running in DEBUG mode
============================================================================
[...]n/http/tests/HttpJwtBenchmarkTest.cpp     relative  time/iter   iters/s
============================================================================
nojwt                                                       21.30s    46.94m
jwt                                             112.02%     19.02s    52.59m
WARNING: Benchmark running in DEBUG mode
czentgr@Christians-MacBook-Pro presto-native-execution % _build/release/presto_cpp/main/http/tests/presto_http_jwt_benchmark
WARNING: Benchmark running in DEBUG mode
============================================================================
[...]n/http/tests/HttpJwtBenchmarkTest.cpp     relative  time/iter   iters/s
============================================================================
nojwt                                                       22.34s    44.76m
jwt                                             97.487%     22.92s    43.64m
WARNING: Benchmark running in DEBUG mode

My environment is probably not the best to test at it is varying too much depending on what else is going on.

@czentgr czentgr requested a review from a team as a code owner August 21, 2023 18:13
@czentgr czentgr requested a review from presto-oss August 21, 2023 18:13
Copy link
Collaborator

@majetideepak majetideepak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@czentgr final set of comments. Looks good overall. Can you also verify on a standalone cluster?

* **Type** ``integer``
* **Default value:** ``300``

There is a time period between ceating the JWT on the client
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

typo: creating

@@ -161,6 +162,15 @@ void PrestoServer::run() {
"Https Client Certificates are not configured correctly");
}
clientCertAndKeyPath = optionalClientCertPath.value();

if (systemConfig->internalCommunicationJwtEnabled()) {
#ifdef PRESTO_JWT_ENABLED
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should be #ifndef?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, you are correct.


if(PRESTO_ENABLE_JWT)
target_include_directories(
http_filters PUBLIC ${CMAKE_SOURCE_DIR}/presto_cpp/external/json)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can this be PRIVATE?

Copy link
Contributor Author

@czentgr czentgr Sep 6, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It was private but then it needs another line in the upper cmake (for the http client). I tried to avoid that.


namespace facebook::presto::http::filters {

/// If the internal communication is not enabled the filter
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing comma after enabled If the internal communication is not enabled,

namespace facebook::presto::http::filters {

/// If the internal communication is not enabled the filter
/// is not added to process incoming requests.
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

process's

INSTANTIATE_TEST_CASE_P(
HTTPJwtTest,
HttpJwtTestSuite,
::testing::Values(true, false));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add an empty newline at the end.

@czentgr czentgr force-pushed the cz_add_jwt_usage branch 4 times, most recently from 4c4446d to f37c65e Compare September 11, 2023 16:22
Copy link
Contributor

@steveburnett steveburnett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Two small formatting suggestions for the docs page presto-docs/src/main/sphinx/develop/presto-native.rst.

Also - and there may be a good reason for this - I wanted to ask why this page is not included in the Developer index in presto-docs/src/main/sphinx/develop.rst?

--------------------------

Prestissimo supports JWT authentication for internal communication.
For details on the generally supported parameters visit :doc:`JWT</security/internal-communication>`.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
For details on the generally supported parameters visit :doc:`JWT</security/internal-communication>`.
For details on the generally supported parameters visit `JWT </security/internal-communication.html#jwt>`_.

Links directly to the JWT section.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not sure the proposed change generates the desired link (it is changed to an external link). Locally, it creates file:///security/internal-communication.html#jwt which is not correct.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Interesting: I built locally with this proposed change (yesterday, and just now to retest after your comment) and the link works for me, opening the page at the JWT target.

If it doesn't work for you, please ignore this suggestion and I'll look into this later. Thanks for checking!

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This works

`JWT <../security/internal-communication.html#jwt>`_.

(relative path needed the .. to switch dirs properly).
I will make that change.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Excellent, and thank you for letting me know what you found as this is useful elsewhere in the docs!

@czentgr
Copy link
Contributor Author

czentgr commented Sep 12, 2023

Two small formatting suggestions for the docs page presto-docs/src/main/sphinx/develop/presto-native.rst.

Also - and there may be a good reason for this - I wanted to ask why this page is not included in the Developer index in presto-docs/src/main/sphinx/develop.rst?

Thanks you @steveburnett .
I will update. Regarding the pressto-native.rst page - we had a different PR to add the link to the develop pages. See PR #20792.

@steveburnett
Copy link
Contributor

Thanks you @steveburnett . I will update. Regarding the pressto-native.rst page - we had a different PR to add the link to the develop pages. See PR #20792.

Thanks! I must not have pulled the latest merges for master when I asked that: the link on develop.rst that I wasn't seeing when I wrote my question is in place now.

@czentgr
Copy link
Contributor Author

czentgr commented Sep 18, 2023

@steveburnett would you please re-review the latest changes (based on your comments) and approve, if ok? Not sure if your requested changes block the progression. Thanks!

Copy link
Contributor

@steveburnett steveburnett left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! (docs)

@steveburnett
Copy link
Contributor

steveburnett commented Sep 18, 2023

@steveburnett would you please re-review the latest changes (based on your comments) and approve, if ok? Not sure if your requested changes block the progression. Thanks!

I don't think I block anything, honestly, but everything looks great to me now so I'm happy to approve from a docs perspective if it matters! Just did.

auto rawSystemConfig =
std::make_unique<core::MemConfigMutable>(initalSystemConfig);
auto systemConfig = SystemConfig::instance();
systemConfig->initialize(std::move(rawSystemConfig));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should create a helper SystemConfig* jwtSystemConfig(); where we set the correct config. Probably reuse this inside basicJwtTest.
We then only use the changedSystemConfig to change the JWT config.

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should probably have a separate server and client config with override.
See https://github.com/facebookincubator/velox/blob/main/velox/connectors/hive/storage_adapters/s3fs/tests/MinioServer.h#L51

});
};
}
} // namespace
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

new line here

Copy link
Collaborator

@majetideepak majetideepak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@czentgr looks good overall. Made a couple of comments. I will merge this tomorrow if you can address the comments today.

@amitkdutta Let us know if you have any comments on this.

{std::string(NodeConfig::kNodeId), std::string("testnode")}};
std::unique_ptr<Config> rawNodeConfig =
std::make_unique<core::MemConfig>(nodeConfigValues);
nodeConfig_->initialize(std::move(rawNodeConfig));
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Don't need the nodeConfig_ class variable.

return response;
}

const std::shared_ptr<MemoryPool> memoryPool_ =
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is only used once inside produceHttpResponse. Let's remove this and get the defaultMemoryManager at the call site.


NodeConfig* nodeConfig_ = NodeConfig::instance();

const std::unordered_map<std::string, std::string> defaultSystemConfig_{
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This can be part of jwtSystemConfig as well.

Adding creation and verification of a JWT signed using HMAC SHA-256.
The JWT is used for authentication of internal communication.
On the Java side PR prestodb#19706 added authentication for internal
communication using a JWT token signed with HMAC SHA-256.

Adding this comment enables Prestissimo to to be configured
to verify internal communication requests, e.g from a Java
coordinator.

New system configuration options:
 - internal-communication.jwt.enabled=[true/false]
 - internal-communication.shared-secret=<shared-secret-value>
 - internal-communication.jwt.expiration-seconds=<value in seconds>

A new external dependency jet-cpp from the https://github.com/Thalhammer/jwt-cpp
repo is added to a new setup-adapter.sh script for Prestissimo.
In the dependency directory run
"<path to presto-native-execution>/scripts/setup-adapter.sh jwt" to install it
The jwt-cpp project handles the creation/parsing/verification of the JWT.

A new build options are added to the build environment:
PRESTO_ENABLE_JWT - default off, if on adds jet creation and verification capability,
 turned on during the pipeline build.
Copy link
Collaborator

@majetideepak majetideepak left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for addressing all the comments @czentgr

@majetideepak
Copy link
Collaborator

@tdcmeehan Can we get an approval from prestodb/committers? Thanks.

@majetideepak majetideepak merged commit ed5727c into prestodb:master Oct 4, 2023
@wanglinsong wanglinsong mentioned this pull request Dec 8, 2023
26 tasks
@czentgr czentgr deleted the cz_add_jwt_usage branch July 31, 2024 17:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
MVP Prestissimo MVP
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants