Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add GCS signed URL support #5300

Merged
merged 29 commits into from
Apr 4, 2024
Merged
Changes from 1 commit
Commits
Show all changes
29 commits
Select commit Hold shift + click to select a range
15d0eda
add util function for gcp sign url
l1nxy Jan 13, 2024
18c66b3
add string to sign and other sign functions
l1nxy Jan 14, 2024
e10ba08
add GoogleCloudStorageConfig::new and config and move functions to cl…
l1nxy Jan 15, 2024
094eae3
add more code and rearrange struct
l1nxy Jan 16, 2024
6b9172e
add client_email for credential and return the signed url
l1nxy Jan 29, 2024
bb2f7aa
clean some code
l1nxy Jan 29, 2024
b9d11bd
add client email for AuthorizedUserCredentials
l1nxy Feb 2, 2024
06bf3c2
tidy some code
l1nxy Feb 2, 2024
ea84e2b
Merge branch 'apache:master' into add-gcp-sign-url-support
l1nxy Feb 2, 2024
c6bb58a
format doc
l1nxy Feb 2, 2024
c84811c
Add GcpSigningCredentialProvider for getting email
l1nxy Mar 29, 2024
00cba29
Merge remote-tracking branch 'origin/master' into add-gcp-sign-url-su…
l1nxy Mar 29, 2024
8aa9714
add test
l1nxy Mar 29, 2024
82c1401
Move some functions which shared by aws and gcp to utils.
l1nxy Mar 29, 2024
850bae0
fix some bug and make it can get proper result
l1nxy Apr 1, 2024
fbd155a
Merge remote-tracking branch 'origin/master' into add-gcp-sign-url-su…
l1nxy Apr 1, 2024
e3002fa
remoe useless code
l1nxy Apr 2, 2024
14ed5af
Merge remote-tracking branch 'origin/master' into add-gcp-sign-url-su…
l1nxy Apr 2, 2024
a4810a9
Merge branch 'add-gcp-sign-url-support' of github.com:l1nxy/arrow-rs …
l1nxy Apr 2, 2024
27dd934
tidy some code
l1nxy Apr 2, 2024
d58086e
do not export host
l1nxy Apr 2, 2024
a97a09b
add sign_by_key
l1nxy Apr 2, 2024
bfe1110
Cleanup
tustvold Apr 3, 2024
23aa8d4
Add ServiceAccountKey
tustvold Apr 3, 2024
9514df7
Further tweaks
tustvold Apr 3, 2024
6db0769
add more scope for signing.
l1nxy Apr 4, 2024
4426f8b
tidy
l1nxy Apr 4, 2024
c0b3c98
Tweak and add test
tustvold Apr 4, 2024
bf46053
Retry and handle errors for signBlob
tustvold Apr 4, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
17 changes: 11 additions & 6 deletions object_store/src/gcp/credential.rs
Original file line number Diff line number Diff line change
Expand Up @@ -44,7 +44,10 @@ use std::time::{Duration, Instant};
use tracing::info;
use url::Url;

pub const DEFAULT_SCOPE: &str = "https://www.googleapis.com/auth/devstorage.full_control";
pub const DEFAULT_SCOPE: [&str; 2] = [
"https://www.googleapis.com/auth/devstorage.full_control",
"https://www.googleapis.com/auth/cloud-platform",
l1nxy marked this conversation as resolved.
Show resolved Hide resolved
];

pub const DEFAULT_GCS_BASE_URL: &str = "https://storage.googleapis.com";

Expand Down Expand Up @@ -354,7 +357,7 @@ impl ServiceAccountCredentials {
self.private_key_id,
self.client_email,
ServiceAccountKey::from_pem(self.private_key.as_bytes())?,
DEFAULT_SCOPE.to_string(),
DEFAULT_SCOPE.join(" ").to_string(),
)?)
}

Expand Down Expand Up @@ -689,10 +692,12 @@ impl GCSAuthorizer {
.append_pair("X-Goog-SignedHeaders", &signed_headers);

let string_to_sign = self.string_to_sign(date, &method, url, &headers);
let signature = match &self.credential.private_key {
Some(key) => key.sign(&string_to_sign)?,
None => client.sign_blob(&string_to_sign, email).await?,
};
// let signature = match &self.credential.private_key {
l1nxy marked this conversation as resolved.
Show resolved Hide resolved
// Some(key) => key.sign(&string_to_sign)?,
// None => client.sign_blob(&string_to_sign, email).await?,
// };
//
let signature = client.sign_blob(&string_to_sign, email).await?;

url.query_pairs_mut()
.append_pair("X-Goog-Signature", &signature);
Expand Down
Loading