Network Tunneling in Batch SQL Connectors #2190
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The connector side of integrating the network tunnel has a bunch of boilerplate which to me doesn't seem like it needs to be copy-pasted into each connector. This commit adds the SSH tunnel config structs and a Start() helper method into the network tunnel Go package so that future integrations can be a bit more concise. This factoring-out is backwards compatible: all of the old structs and methods still exist, it's just that the package now includes the config structs and a Start() method which can be used to make the connector code cleaner. Also the package name has been changed from `network-tunnel` to `networkTunnel` because that's the name that every single import of the package assigns it.
Integrating SSH tunneling into the BigQuery connection isn't entirely obvious since we're not connecting to a specific address.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
Integrates network/SSH tunneling into the batch SQL connectors, with the exception of
source-bigquery-batchsince it's not self-evident how to apply it there and I'm just picking some extremely low-hanging fruit here.Notes for reviewers:
In order to avoid copy-pasting a bunch of unnecessary bits of boilerplate, I have factored out some of the config structs and startup logic that other connectors used into the network tunnel Go package and am using that here. I have not gone back and modified the other dozen connectors with SSH tunneling support to use the more concise integration in this PR, but it's something we might want to do at some point just for code health reasons.
This change is