Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add Source Dropbox #1499

Closed
wants to merge 2 commits into from
Closed

Add Source Dropbox #1499

wants to merge 2 commits into from

Conversation

Z33DD
Copy link
Contributor

@Z33DD Z33DD commented Apr 17, 2024

Description:

This connector uses OAuth for Authentication, so we need to create an app on the Dropbox console like in the image bellow:
image

After setting the Estuary's branding, go to the permission tab and make sure that account_info.read, files.metadata.read and files.content.read are selected.

On the overview tab, copy the OAuth credentials and set the Redirect URIs.

Now, apply to production and wait.
image

Workflow steps:

(How does one use this feature, and how has it changed)

Documentation links affected:

Add docs for this connector: estuary/flow#1458

Notes for reviewers:

(anything that might help someone review this PR)


This change is Reviewable

@Z33DD Z33DD self-assigned this Apr 17, 2024
source-dropbox/main.go Outdated Show resolved Hide resolved
source-dropbox/main.go Outdated Show resolved Hide resolved
source-dropbox/oauth2.go Outdated Show resolved Hide resolved
source-dropbox/oauth2.go Outdated Show resolved Hide resolved
@Z33DD Z33DD marked this pull request as ready for review April 23, 2024 19:45
@Z33DD Z33DD marked this pull request as draft April 23, 2024 21:45
@Z33DD Z33DD marked this pull request as ready for review April 25, 2024 01:35
Copy link
Member

@williamhbaker williamhbaker left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

See inline comments. Please address them, and test the updated connector against a dropbox account with a non-trivial number of files in a directory structure containing subdirectories.

source-dropbox/oauth2.go Outdated Show resolved Hide resolved
source-dropbox/main.go Outdated Show resolved Hide resolved
source-dropbox/main.go Outdated Show resolved Hide resolved
@Z33DD Z33DD requested a review from williamhbaker April 30, 2024 22:01
@Z33DD Z33DD force-pushed the source-dropbox branch from 0a56b54 to ccb0fc5 Compare May 1, 2024 02:10
source-dropbox/oauth2.go Outdated Show resolved Hide resolved
source-dropbox/oauth2.go Outdated Show resolved Hide resolved
source-dropbox/oauth2.go Outdated Show resolved Hide resolved
source-dropbox/oauth2.go Show resolved Hide resolved
source-dropbox/oauth2.go Outdated Show resolved Hide resolved
source-dropbox/main.go Outdated Show resolved Hide resolved
source-dropbox/main.go Outdated Show resolved Hide resolved
source-dropbox/main.go Outdated Show resolved Hide resolved
source-dropbox/main.go Show resolved Hide resolved
source-dropbox/main.go Outdated Show resolved Hide resolved
@williamhbaker
Copy link
Member

In addition to the inline comments, please clean up the commit history prior to submitting this for review again. We'd like the merged set of commits to be useful for future readers of the code, rather than a chronology of all the changes made during the review process.

)

type config struct {
Path string `json:"path" yaml:"path"`
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Those YAML tags are necessary for reading test credentials

@Z33DD
Copy link
Contributor Author

Z33DD commented May 2, 2024

Docs PR: estuary/flow#1458

Add OAuth2 authentication for Dropbox API

Refactor OAuth2Spec function in oauth2.go and made the necessary changes to the connector schema.

Refactor OAuth2Spec function and add Credentials struct for Dropbox API authentication

Refactor Dropbox authentication schema and add new fields

Refactor Dropbox authentication schema and add new fields

Refactor Dropbox authentication schema and add new fields

Refactor Dropbox authentication schema and add new fields

Refactor authentication schema to remove the refresh_token field

Refactor Dropbox authentication schema and remove unnecessary fields

Refactor Dropbox authentication schema and remove unnecessary fields

Fix mapping

Update schema

Update Dropbox authentication schema to add refresh_token

Refactor Dropbox authentication to use OAuth2 client instead of access token

Implement Token source for OAuth2 flow

Simplify code

Revert unnecessary changes

Revert changes to Dockerfile

Remove unnecessary comments

Simplify code

Remove debug logging

Remove TestAzureBlobStore_getConfigSchema from source-dropbox/.snapshots
@Z33DD Z33DD force-pushed the source-dropbox branch from c818900 to d1a19fc Compare May 3, 2024 11:49
Minor refactor

Fix recursive queries problem

Implement proposed changes

Fix pagination issues and update tests

Remove pagination limit in Dropbox connector
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants