Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Azure secp bulk loading #850

Merged
merged 32 commits into from
Jul 28, 2023
Merged
Show file tree
Hide file tree
Changes from 31 commits
Commits
Show all changes
32 commits
Select commit Hold shift + click to select a range
8ed4052
Azure secp bulk loading
jframe Jul 7, 2023
c504fbf
Load keys in parallel using mapKeyProperties
jframe Jul 10, 2023
b397719
AzureKeyVault tests
jframe Jul 11, 2023
6960fff
spotless
jframe Jul 11, 2023
a32247a
remove forkjoin pool hack
jframe Jul 11, 2023
5858e75
add additional eth1 secp tests to AzureKeyVaultAcceptanceTest
jframe Jul 18, 2023
be9e45c
add support for azure bulk loading using file configuration in AT DSL
jframe Jul 19, 2023
63795da
separate tagged and non-tagged keys in azure vault AT
jframe Jul 19, 2023
cee246e
Merge remote-tracking branch 'upstream/master' into azure_secp_bulk_l…
jframe Jul 19, 2023
52f6da5
fix integration test
jframe Jul 19, 2023
ea18172
Merge remote-tracking branch 'upstream/master' into azure_secp_bulk_l…
jframe Jul 20, 2023
1d1d0ae
Remove AzureKeyVaultSignerFactory needToHash constructor as this only…
jframe Jul 20, 2023
297aa64
Set the expected number of keys to be loaded to account for multi-lin…
jframe Jul 20, 2023
a5e38ec
Merge remote-tracking branch 'upstream/master' into azure_secp_bulk_l…
jframe Jul 21, 2023
f387f00
Update AzureKeyVaultTest to use keys configured in Azure test env
jframe Jul 21, 2023
fb28ee8
Change all Azure code to use the AzureKeyVaultFactory and create an E…
jframe Jul 21, 2023
be2f509
Add shutdown hook for AzureKeyVaultFactory
jframe Jul 21, 2023
aac3092
Re-enable process runner
jframe Jul 21, 2023
b4937fb
Use separate key dirs for signers so that SlashingPruningAcceptanceTe…
jframe Jul 21, 2023
e15c332
Cleanup after review
jframe Jul 24, 2023
7125afc
Merge remote-tracking branch 'upstream/master' into azure_secp_bulk_l…
jframe Jul 24, 2023
8f7c28c
Additional tests for Azure key mapping for handling exceptions and nu…
jframe Jul 24, 2023
dba365e
AzureKeyVaultFactory tests
jframe Jul 24, 2023
17730b2
changelog
jframe Jul 24, 2023
741c7b1
changes after group review
jframe Jul 27, 2023
801e3bc
update dependencyCheck plugin
jframe Jul 27, 2023
7a9f1de
Merge branch 'master' into azure_secp_bulk_loading
jframe Jul 27, 2023
027b4f1
Revert "update dependencyCheck plugin"
jframe Jul 27, 2023
35c6dec
After PR review
jframe Jul 27, 2023
e139f26
After PR review
jframe Jul 27, 2023
5a655c4
After PR review
jframe Jul 28, 2023
8a1a964
After PR review
jframe Jul 28, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,18 @@
# Changelog

## Next release

### Breaking Changes
- Eth2 Azure command line option --azure-secrets-tags is now deprecated and is replaced with --azure-tags. The --azure-secrets-tags option will be removed in a future release.

### Features Added
- Azure bulk mode support for loading multiline (`\n` delimited, up to 200) keys per secret.
- Hashicorp connection properties can now override http protocol to HTTP/1.1 from the default of HTTP/2. [#817](https://github.com/ConsenSys/web3signer/pull/817)
- Add --key-config-path as preferred alias to --key-store-path [#826](https://github.com/Consensys/web3signer/pull/826)
- Add eth_signTransaction RPC method under the eth1 subcommand [#822](https://github.com/ConsenSys/web3signer/pull/822)
- Add eth_sendTransaction RPC method under the eth1 subcommand [#835](https://github.com/Consensys/web3signer/pull/835)
- Add EIP-1559 support for eth1 public transactions for eth_sendTransaction and eth_signTransaction [#836](https://github.com/Consensys/web3signer/pull/836)
- Add Azure bulk loading for secp256k1 keys in eth1 mode [#850](https://github.com/Consensys/web3signer/pull/850)

### Bugs fixed
- Support long name aliases in environment variables and YAML configuration [#825](https://github.com/Consensys/web3signer/pull/825)
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -115,33 +115,6 @@ public List<String> createCmdLineParams() {
if (signerConfig.getMode().equals("eth2")) {
yamlConfig.append(createEth2SlashingProtectionArgs());

if (signerConfig.getAzureKeyVaultParameters().isPresent()) {
final AzureKeyVaultParameters azureParams = signerConfig.getAzureKeyVaultParameters().get();
yamlConfig.append(
String.format(YAML_BOOLEAN_FMT, "eth2.azure-vault-enabled", Boolean.TRUE));
yamlConfig.append(
String.format(
YAML_STRING_FMT,
"eth2.azure-vault-auth-mode",
azureParams.getAuthenticationMode().name()));
yamlConfig.append(
String.format(YAML_STRING_FMT, "eth2.azure-vault-name", azureParams.getKeyVaultName()));
yamlConfig.append(
String.format(YAML_STRING_FMT, "eth2.azure-client-id", azureParams.getClientId()));
yamlConfig.append(
String.format(
YAML_STRING_FMT, "eth2.azure-client-secret", azureParams.getClientSecret()));
yamlConfig.append(
String.format(YAML_STRING_FMT, "eth2.azure-tenant-id", azureParams.getTenantId()));

azureParams
.getTags()
.forEach(
(tagName, tagValue) ->
yamlConfig.append(
String.format(
YAML_STRING_FMT, "eth2.azure-secrets-tags", tagName + "=" + tagValue)));
}
if (signerConfig.getKeystoresParameters().isPresent()) {
final KeystoresParameters keystoresParameters = signerConfig.getKeystoresParameters().get();
yamlConfig.append(
Expand Down Expand Up @@ -181,6 +154,12 @@ public List<String> createCmdLineParams() {
yamlConfig.append(createDownstreamTlsArgs());
}

signerConfig
.getAzureKeyVaultParameters()
.ifPresent(
azureParams ->
yamlConfig.append(azureBulkLoadingOptions(signerConfig.getMode(), azureParams)));

// create temporary config file
try {
final Path configFile = Files.createTempFile("web3signer_config", ".yaml");
Expand All @@ -196,6 +175,35 @@ public List<String> createCmdLineParams() {
return params;
}

private String azureBulkLoadingOptions(
jframe marked this conversation as resolved.
Show resolved Hide resolved
final String mode, final AzureKeyVaultParameters azureParams) {
final StringBuilder yamlConfig = new StringBuilder();
yamlConfig.append(String.format(YAML_BOOLEAN_FMT, mode + ".azure-vault-enabled", Boolean.TRUE));
yamlConfig.append(
String.format(
YAML_STRING_FMT,
mode + ".azure-vault-auth-mode",
azureParams.getAuthenticationMode().name()));
yamlConfig.append(
String.format(YAML_STRING_FMT, mode + ".azure-vault-name", azureParams.getKeyVaultName()));
yamlConfig.append(
String.format(YAML_STRING_FMT, mode + ".azure-client-id", azureParams.getClientId()));
yamlConfig.append(
String.format(
YAML_STRING_FMT, mode + ".azure-client-secret", azureParams.getClientSecret()));
yamlConfig.append(
String.format(YAML_STRING_FMT, mode + ".azure-tenant-id", azureParams.getTenantId()));

azureParams
.getTags()
.forEach(
(tagName, tagValue) ->
yamlConfig.append(
String.format(
YAML_STRING_FMT, mode + ".azure-tags", tagName + "=" + tagValue)));
return yamlConfig.toString();
}

private CommandArgs createSubCommandArgs() {
final List<String> params = new ArrayList<>();
final StringBuilder yamlConfig = new StringBuilder();
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -99,26 +99,7 @@ public List<String> createCmdLineParams() {
params.addAll(createEth2Args());

if (signerConfig.getAzureKeyVaultParameters().isPresent()) {
final AzureKeyVaultParameters azureParams = signerConfig.getAzureKeyVaultParameters().get();
params.add("--azure-vault-enabled=true");
params.add("--azure-vault-auth-mode");
params.add(azureParams.getAuthenticationMode().name());
params.add("--azure-vault-name");
params.add(azureParams.getKeyVaultName());
params.add("--azure-client-id");
params.add(azureParams.getClientId());
params.add("--azure-client-secret");
params.add(azureParams.getClientSecret());
params.add("--azure-tenant-id");
params.add(azureParams.getTenantId());

azureParams
.getTags()
.forEach(
(tagName, tagValue) -> {
params.add("--azure-secrets-tags");
params.add(tagName + "=" + tagValue);
});
createAzureArgs(params);
}
if (signerConfig.getKeystoresParameters().isPresent()) {
final KeystoresParameters keystoresParameters = signerConfig.getKeystoresParameters().get();
Expand All @@ -143,6 +124,10 @@ public List<String> createCmdLineParams() {
params.add("--chain-id");
params.add(Long.toString(signerConfig.getChainIdProvider().id()));
params.addAll(createDownstreamTlsArgs());

if (signerConfig.getAzureKeyVaultParameters().isPresent()) {
createAzureArgs(params);
}
}

return params;
Expand Down Expand Up @@ -331,6 +316,29 @@ private Collection<String> awsBulkLoadingOptions(
return params;
}

private void createAzureArgs(final List<String> params) {
final AzureKeyVaultParameters azureParams = signerConfig.getAzureKeyVaultParameters().get();
params.add("--azure-vault-enabled=true");
params.add("--azure-vault-auth-mode");
params.add(azureParams.getAuthenticationMode().name());
params.add("--azure-vault-name");
params.add(azureParams.getKeyVaultName());
params.add("--azure-client-id");
params.add(azureParams.getClientId());
params.add("--azure-client-secret");
params.add(azureParams.getClientSecret());
params.add("--azure-tenant-id");
params.add(azureParams.getTenantId());

azureParams
.getTags()
.forEach(
(tagName, tagValue) -> {
params.add("--azure-tags");
params.add(tagName + "=" + tagValue);
});
}

private List<String> createSubCommandArgs() {
final List<String> params = new ArrayList<>();

Expand Down
Loading