Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Dataset project Id Feature implemented #242

Merged
merged 2 commits into from
Oct 17, 2023
Merged

Dataset project Id Feature implemented #242

merged 2 commits into from
Oct 17, 2023

Conversation

shivamVarCS
Copy link
Contributor

@shivamVarCS shivamVarCS commented Sep 29, 2023 •
edited
Loading

@shivamVarCS shivamVarCS force-pushed the feature/big-query-delta-plugin-project-dataset-Id branch from 15dd6f6 to bb07af0 Compare October 3, 2023 10:41
@sahusanket
Copy link
Contributor

Please add some description for context. Or attach a JIRA to this PR.

Need more context on intended behavior projectId vs datasetProjectId in order to review this.

sahusanket
sahusanket requested changes Oct 5, 2023
View reviewed changes
src/main/java/io/cdap/delta/bigquery/BigQueryTarget.java Outdated
String project = conf.getProject();

if (conf.getProject() == null) {
return;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should ideally throw a proper error like illegal argument exception in such case.

And all such checks should be inside the method getProject() to ensure the method is null safe.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated and removed

src/main/java/io/cdap/delta/bigquery/BigQueryTarget.java Outdated
}
String project = conf.getDatasetProject();
if (project == null) {
return;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again similar : adding this check throwing exception within getDatasetProject() would be better.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

src/main/java/io/cdap/delta/bigquery/BigQueryTarget.java Outdated
@@ -147,22 +159,23 @@ public void initialize(DeltaTargetContext context) throws Exception {
@Override
public EventConsumer createConsumer(DeltaTargetContext context) throws IOException {
Credentials credentials = conf.getCredentials();
String project = conf.getProject();
if (conf.getProject() == null) {
throw new RuntimeException("Project id is not Present");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar : move this inside the getProject() , will reduce code repetition.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated and Removed

src/main/java/io/cdap/delta/bigquery/BigQueryTarget.java Outdated
}
String project = conf.getDatasetProject();
if (project == null) {
throw new RuntimeException("Project Dataset id is not Present");
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar : move this inside the getDatasetProject() , will reduce code repetition.

Also it's better to throw IllegalArgumentException

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated and removed

src/main/java/io/cdap/delta/bigquery/BigQueryTarget.java Outdated
@@ -352,5 +395,22 @@ private Credentials getCredentials() throws IOException {
.createScoped(Collections.singleton("https://www.googleapis.com/auth/cloud-platform"));
}
}
public String getDatasetProject() {
// if it's "auto-detect" that means we need to detect the default project settings
// for sandbox you can use `gcloud config set project my-project-id" to set it
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please remove or move the development or debugging related comments to readme.md

Like here, details about sandbox co-relates to the whole project , not just this method.

It's usually better to comment only what code is doing.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated and Added in the .md file

src/test/java/io/cdap/delta/bigquery/BigQueryTargetTest.java Outdated
@@ -267,7 +268,7 @@ public void testCreateConsumerRetryableFailureForGcsGet() throws Exception {
}

@Test
public void testCreateConsumerNonRetryableFailure() throws Exception {
public void testCreateConsumerNonRetryableFailure() throws Exception {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit : undo indentation

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

@shivamVarCS shivamVarCS force-pushed the feature/big-query-delta-plugin-project-dataset-Id branch from bb07af0 to 0fb8884 Compare October 6, 2023 10:53
sahusanket
sahusanket requested changes Oct 11, 2023
View reviewed changes
docs/bigquery-cdcTarget.md Outdated
account to run the job. If a temporary bucket needs to be created, the bucket will also be created in this project and
'GCE Storage Bucket Admin' role on this project must be granted to the specified service account to create buckets.

**Dataset Project ID**: Project the dataset belongs to. This is only required if the dataset is not
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please add "destination dataset" to be more clear

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

widgets/bigquery-cdcTarget.json Outdated
"label": "DataSet Project ID",
"widget-type": "textbox",
"widget-attributes": {
"placeholder": "Project the dataset belongs to, if different from the project ID."
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

similar : Please add "destination dataset" to be more clear

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Updated

@sahusanket
Copy link
Contributor

I tested the changes in my local,
The BQ job runs in DataSet Project ID and not in projectId

@shivamVarCS
Copy link
Contributor Author

I tested the changes in my local, The BQ job runs in DataSet Project ID and not in projectId

I am currently working on fix.

@shivamVarCS shivamVarCS merged commit 5624bac into develop Oct 17, 2023
5 checks passed
This was referenced Oct 18, 2023
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@sahusanket sahusanket sahusanket approved these changes

Assignees
No one assigned
Labels
build
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@shivamVarCS @sahusanket