Added Salesforce source #1355

Luishfs · 2024-03-11T17:59:55Z

Description:

(Describe the high level scope of new or changed features)

Workflow steps:

(How does one use this feature, and how has it changed)

Documentation links affected:

(list any documentation links that you created, or existing ones that you've identified as needing updates, along with a brief description)

Notes for reviewers:

(anything that might help someone review this PR)
@jgraettinger

This change is

Fixed flow.yaml file configs

Remote Repo URL: git@github.com:airbytehq/airbyte.git Source name: 0f887bc Source Commit ID: 0f887bc143ee14726983816c9972b9398e500bc0 Source Repo Prefix: airbyte-integrations/connectors/source-salesforce/ Import Path: source-salesforce/ License Type: MIT License Path: airbyte-integrations/connectors/source-salesforce/metadata.yaml git-merge-subpath: 0f887bc143ee14726983816c9972b9398e500bc0 airbyte-integrations/connectors/source-salesforce source-salesforce

Removed integration tests Due to the source-salesforce connector having a big number of schema files AND the connector not running capture tests, the necessity of schema files is undermined. Since that, this commit rebases into removing those

At this point in time, all unit_tests shall pass, only snapshot ones "fail"

jgraettinger · 2024-03-11T18:16:56Z

.github/workflows/python.yaml

@@ -48,6 +50,9 @@ jobs:
            type: capture
            version: v1
            usage_rate: "0.0"
+          - name: source-salesforce
+            type: capture
+            version: v2


this will need usage_rate: "1.0"

Fixed in a1c34e9

jgraettinger · 2024-03-11T18:18:17Z

source-salesforce/tests/conftest.py

@@ -19,14 +19,14 @@ def time_sleep_mock(mocker):

 @pytest.fixture(scope="module")
 def bulk_catalog():
-    with open("unit_tests/bulk_catalog.json") as f:
+    with open("source-salesforce/tests/bulk_catalog.json") as f:


does this scoping change if you add an empty __init__.py to this tests/ folder ?

The __init__.py is already at the folder =P

jgraettinger · 2024-03-11T18:20:20Z

source-salesforce/README.md

@@ -0,0 +1,132 @@
+# Salesforce Source


This readme can be removed, as it's relevant only for Airbyte.

Better to not have it, then to have one that's wrong.

Removed in a1c34e9

jgraettinger · 2024-03-11T18:20:33Z

source-salesforce/icon.svg

This can be removed

removed in a1c34e9

jgraettinger · 2024-03-11T18:20:54Z

source-salesforce/metadata.yaml

@@ -0,0 +1,24 @@
+data:


this can be removed

removed in a1c34e9

jgraettinger · 2024-03-11T18:23:13Z

source-salesforce/pyproject.toml

+
+[tool.poetry.dependencies]
+python = ">=3.11,<3.12"
+flow-sdk = {path="../python", develop=true}


This is using the older flow-sdk, which we've now replaced with the estuary-cdk.

Switching everything over has been a work in progress that's happened concurrently with your efforts here (see this just-landed PR). Would you please update accordingly?

Modified in a1c34e9

jgraettinger · 2024-03-11T18:24:11Z

source-salesforce/test.flow.yaml

+          - "-m"
+          - source-salesforce
+        config: config.yaml
+    bindings:


Please remove these, leaving just bindings: [].

They're already covered by the discovery snapshot.

updated in a1c34e9

jgraettinger · 2024-03-11T18:25:32Z

source-salesforce/tests/snapshots/snapshots__spec__capture.stdout.json

      ],
      "title": "Salesforce Source Spec",
      "type": "object"
    },
-    "documentationUrl": "https://go.estuary.dev/AyWXIh",
+    "documentationUrl": "https://docs.airbyte.com/integrations/sources/salesforce",


We should restore this. It's threaded through in __main__.py as I recall.

restored in e039e79

jgraettinger · 2024-03-11T18:27:51Z

source-salesforce/tests/snapshots/snapshots__spec__capture.stdout.json

-          ],
-          "type": "object",
-          "x-oauth2-provider": "salesforce"
+        "auth_type": {


This schema change is ... wrong. I'm not sure what's happened here. Note that auth_type is top-level, but should be nested under credentials -- compare to spec.yaml.

Yeah, it doesn't make sense, probably lost it in the rebase, will replace it.

restored in e039e79

jgraettinger · 2024-03-11T18:28:32Z

source-salesforce/tests/snapshots/snapshots__discover__capture.stdout.json

Ouch, there's an awful lot of churn in here in switching over the connector -- so much that Github won't even load it.

What's the nature of the changes that are happening?

So, the salesforce connector generates its own schema when running, whats going on is that the discovery is generating all 600 schemas ( 128.793 lines ) and displaying them

Understood, but I was talking about the changes in this snapshot between the baseline (from snapshots of the docker image) and this commit which switches over to the imported connector. I would not expect nearly as much churn here.

It's possible it may just be re-orderings and the diff will get much smaller after switching to the estuary-cdk, which preserves the sorted order used by the original Airbyte connector (while the flow-sdk randomizes it).

…rce to the new CDK

Correct version was probably lost during rebase, this represents the spec state after correcting spec.yaml file

Luishfs · 2024-03-13T16:41:20Z

Closing due to review, this PR needs rebasing.

Luishfs added 8 commits March 8, 2024 14:50

Added initial files & test file

10819c7

Added tests snapshots

d53ec84

Fixed flow.yaml file configs

Added connector to CI

99234a4

Added unit_tests under test folder

1424acf

Removed integration tests Due to the source-salesforce connector having a big number of schema files AND the connector not running capture tests, the necessity of schema files is undermined. Since that, this commit rebases into removing those

Fix-ups & added unit-tests

22b3a7f

Fixed all connector unit_tests

ae13497

At this point in time, all unit_tests shall pass, only snapshot ones "fail"

updated snapshots

c6b4021

Luishfs self-assigned this Mar 11, 2024

jgraettinger reviewed Mar 11, 2024

View reviewed changes

Luishfs added 2 commits March 11, 2024 16:29

Fixed CI, Removed unused files, Cleaned test.flow.yaml, Added salesfo…

a1c34e9

…rce to the new CDK

Correcting spec snapshot

e039e79

Correct version was probably lost during rebase, this represents the spec state after correcting spec.yaml file

Luishfs force-pushed the salesforce-source branch from 13ae043 to e039e79 Compare March 11, 2024 19:38

Merge branch 'main' into salesforce-source

da6949c

Luishfs closed this Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Added Salesforce source #1355

Added Salesforce source #1355

Luishfs commented Mar 11, 2024 •

edited by jgraettinger

Loading

jgraettinger Mar 11, 2024

Luishfs Mar 11, 2024

jgraettinger Mar 11, 2024

Luishfs Mar 11, 2024

jgraettinger Mar 11, 2024

Luishfs Mar 11, 2024

jgraettinger Mar 11, 2024

Luishfs Mar 11, 2024

jgraettinger Mar 11, 2024

Luishfs Mar 11, 2024

jgraettinger Mar 11, 2024

Luishfs Mar 11, 2024

jgraettinger Mar 11, 2024

Luishfs Mar 11, 2024

jgraettinger Mar 11, 2024

Luishfs Mar 11, 2024 •

edited

Loading

jgraettinger Mar 11, 2024

Luishfs Mar 11, 2024

Luishfs Mar 11, 2024

jgraettinger Mar 11, 2024

Luishfs Mar 11, 2024

jgraettinger Mar 11, 2024

Luishfs commented Mar 13, 2024

Added Salesforce source #1355

Added Salesforce source #1355

Conversation

Luishfs commented Mar 11, 2024 • edited by jgraettinger Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Luishfs Mar 11, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Luishfs commented Mar 13, 2024

Luishfs commented Mar 11, 2024 •

edited by jgraettinger

Loading

Luishfs Mar 11, 2024 •

edited

Loading