Remove unused Accounts DB queries #5978

akkomar · 2024-07-26T14:58:46Z

Checklist for reviewer:

Commits should reference a bug or github issue, if relevant (if a bug is referenced, the pull request should include the bug number in the title).
If the PR comes from a fork, trigger integration CI tests by running the Push to upstream workflow and provide the <username>:<branch> of the fork as parameter. The parameter will also show up
in the logs of the manual-trigger-required-for-fork CI task together with more detailed instructions.
If adding a new field to a query, ensure that the schema and dependent downstream schemas have been updated.
When adding a new derived dataset, ensure that data is not available already (fully or partially) and recommend extending an existing dataset in favor of creating new ones. Data can be available in the bigquery-etl repository, looker-hub or in looker-spoke-default.

For modifications to schemas in restricted namespaces (see CODEOWNERS):

Follow the change control procedure

┆Issue is synchronized with this Jira Task

dataops-ci-bot · 2024-07-26T15:22:19Z

Integration report for "Remove unused Accounts DB queries"

`sql.diff`

Click to expand!

diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_accounts_backend_external.py /tmp/workspace/generated-sql/dags/bqetl_accounts_backend_external.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_accounts_backend_external.py	2024-07-26 15:05:05.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_accounts_backend_external.py	2024-07-26 15:21:19.000000000 +0000
@@ -53,54 +53,6 @@
     tags=tags,
 ) as dag:
 
-    accounts_backend_external__accounts__v1 = bigquery_etl_query(
-        task_id="accounts_backend_external__accounts__v1",
-        destination_table="accounts_v1",
-        dataset_id="accounts_backend_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="akomar@mozilla.com",
-        email=["akomar@mozilla.com", "telemetry-alerts@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    accounts_backend_external__emails__v1 = bigquery_etl_query(
-        task_id="accounts_backend_external__emails__v1",
-        destination_table="emails_v1",
-        dataset_id="accounts_backend_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="akomar@mozilla.com",
-        email=["akomar@mozilla.com", "telemetry-alerts@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    accounts_backend_external__nonprod_accounts__v1 = bigquery_etl_query(
-        task_id="accounts_backend_external__nonprod_accounts__v1",
-        destination_table="nonprod_accounts_v1",
-        dataset_id="accounts_backend_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="akomar@mozilla.com",
-        email=["akomar@mozilla.com", "telemetry-alerts@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
-    accounts_backend_external__nonprod_emails__v1 = bigquery_etl_query(
-        task_id="accounts_backend_external__nonprod_emails__v1",
-        destination_table="nonprod_emails_v1",
-        dataset_id="accounts_backend_external",
-        project_id="moz-fx-data-shared-prod",
-        owner="akomar@mozilla.com",
-        email=["akomar@mozilla.com", "telemetry-alerts@mozilla.com"],
-        date_partition_parameter=None,
-        depends_on_past=False,
-        task_concurrency=1,
-    )
-
     accounts_db_derived__accounts_aggregates__v1 = bigquery_etl_query(
         task_id="accounts_db_derived__accounts_aggregates__v1",
         destination_table="accounts_aggregates_v1",
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/dags/bqetl_generated_funnels.py /tmp/workspace/generated-sql/dags/bqetl_generated_funnels.py
--- /tmp/workspace/main-generated-sql/dags/bqetl_generated_funnels.py	2024-07-26 15:05:05.000000000 +0000
+++ /tmp/workspace/generated-sql/dags/bqetl_generated_funnels.py	2024-07-26 15:21:20.000000000 +0000
@@ -276,6 +276,21 @@
         )
     )
 
+    monitor_frontend_derived__monitor_dashboard_user_journey_funnels__v1 = bigquery_etl_query(
+        task_id="monitor_frontend_derived__monitor_dashboard_user_journey_funnels__v1",
+        destination_table="monitor_dashboard_user_journey_funnels_v1",
+        dataset_id="monitor_frontend_derived",
+        project_id="moz-fx-data-shared-prod",
+        owner="ksiegler@mozilla.org",
+        email=[
+            "ascholtz@mozilla.com",
+            "ksiegler@mozilla.org",
+            "telemetry-alerts@mozilla.com",
+        ],
+        date_partition_parameter="submission_date",
+        depends_on_past=False,
+    )
+
     accounts_frontend_derived__email_first_reg_login_funnels__v1.set_upstream(
         wait_for_copy_deduplicate_all
     )
@@ -325,3 +340,7 @@
     firefox_accounts_derived__registration_funnels_legacy_events__v1.set_upstream(
         wait_for_firefox_accounts_derived__fxa_stdout_events__v1
     )
+
+    monitor_frontend_derived__monitor_dashboard_user_journey_funnels__v1.set_upstream(
+        wait_for_copy_deduplicate_all
+    )
Only in /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend: accounts
Only in /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend: nonprod_accounts
Only in /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod: accounts_backend_external
Only in /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/internet_outages/global_outages: schema.yaml
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/accounts/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/accounts/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/accounts/metadata.yaml	2024-07-26 15:00:02.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/accounts/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,22 +0,0 @@
----
-friendly_name: Accounts table from production FxA database
-description: |-
-  An authorized view on top of the `accounts_backend_external.accounts_v1` table
-  that only includes non-sensitive fields.
-  Some fields in this table are converted to a more user-friendly, BigQuery-native format:
-    - `uid` is converted from bytes to a hex string
-    - boolean integer columns are converted to BOOL
-    - timestamp columns are converted to TIMESTAMP
-
-  See https://mozilla.github.io/ecosystem-platform/reference/database-structure#database-fxa
-labels:
-  authorized: true
-workgroup_access:
-  - role: roles/bigquery.dataViewer
-    members:
-      - workgroup:mozilla-confidential
-
-# Generated by bigquery_etl.dependency
-references:
-  view.sql:
-  - moz-fx-data-shared-prod.accounts_backend_external.accounts_v1
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/accounts/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/accounts/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/accounts/schema.yaml	2024-07-26 14:59:53.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/accounts/schema.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,42 +0,0 @@
-fields:
-- name: uid
-  type: STRING
-  mode: NULLABLE
-  description: 'Account ID in hexadecimal format.
-
-    FxA stores this as bytes, for purposes of logging or integration
-
-    with other systems we convert it to a hex string.'
-- name: emailVerified
-  type: BOOLEAN
-  mode: NULLABLE
-- name: verifierVersion
-  type: INTEGER
-  mode: NULLABLE
-- name: verifierSetAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: createdAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: locale
-  type: STRING
-  mode: NULLABLE
-- name: lockedAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: profileChangedAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: keysChangedAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: ecosystemAnonId
-  type: STRING
-  mode: NULLABLE
-- name: disabledAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: metricsOptOutAt
-  type: TIMESTAMP
-  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/accounts/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/accounts/view.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/accounts/view.sql	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/accounts/view.sql	1970-01-01 00:00:00.000000000 +0000
@@ -1,18 +0,0 @@
-CREATE OR REPLACE VIEW
-  `moz-fx-data-shared-prod.accounts_backend.accounts`
-AS
-SELECT
-  uid,
-  emailVerified,
-  verifierVersion,
-  verifierSetAt,
-  createdAt,
-  locale,
-  lockedAt,
-  profileChangedAt,
-  keysChangedAt,
-  ecosystemAnonId,
-  disabledAt,
-  metricsOptOutAt,
-FROM
-  `moz-fx-data-shared-prod.accounts_backend_external.accounts_v1`
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/nonprod_accounts/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/nonprod_accounts/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/nonprod_accounts/metadata.yaml	2024-07-26 15:00:02.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/nonprod_accounts/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,22 +0,0 @@
----
-friendly_name: Accounts table from FxA database in stage environment
-description: |-
-  An authorized view on top of the `accounts_backend_external.nonprod_accounts_v1` table
-  that only includes non-sensitive fields.
-  Some fields in this table are converted to a more user-friendly, BigQuery-native format:
-    - `uid` is converted from bytes to a hex string
-    - boolean integer columns are converted to BOOL
-    - timestamp columns are converted to TIMESTAMP
-
-  See https://mozilla.github.io/ecosystem-platform/reference/database-structure#database-fxa
-labels:
-  authorized: true
-workgroup_access:
-  - role: roles/bigquery.dataViewer
-    members:
-      - workgroup:mozilla-confidential
-
-# Generated by bigquery_etl.dependency
-references:
-  view.sql:
-  - moz-fx-data-shared-prod.accounts_backend_external.nonprod_accounts_v1
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/nonprod_accounts/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/nonprod_accounts/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/nonprod_accounts/schema.yaml	2024-07-26 14:59:53.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/nonprod_accounts/schema.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,42 +0,0 @@
-fields:
-- name: uid
-  type: STRING
-  mode: NULLABLE
-  description: 'Account ID in hexadecimal format.
-
-    FxA stores this as bytes, for purposes of logging or integration
-
-    with other systems we convert it to a hex string.'
-- name: emailVerified
-  type: BOOLEAN
-  mode: NULLABLE
-- name: verifierVersion
-  type: INTEGER
-  mode: NULLABLE
-- name: verifierSetAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: createdAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: locale
-  type: STRING
-  mode: NULLABLE
-- name: lockedAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: profileChangedAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: keysChangedAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: ecosystemAnonId
-  type: STRING
-  mode: NULLABLE
-- name: disabledAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: metricsOptOutAt
-  type: TIMESTAMP
-  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/nonprod_accounts/view.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/nonprod_accounts/view.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/nonprod_accounts/view.sql	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend/nonprod_accounts/view.sql	1970-01-01 00:00:00.000000000 +0000
@@ -1,18 +0,0 @@
-CREATE OR REPLACE VIEW
-  `moz-fx-data-shared-prod.accounts_backend.nonprod_accounts`
-AS
-SELECT
-  uid,
-  emailVerified,
-  verifierVersion,
-  verifierSetAt,
-  createdAt,
-  locale,
-  lockedAt,
-  profileChangedAt,
-  keysChangedAt,
-  ecosystemAnonId,
-  disabledAt,
-  metricsOptOutAt,
-FROM
-  `moz-fx-data-shared-prod.accounts_backend_external.nonprod_accounts_v1`
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/accounts_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/accounts_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/accounts_v1/metadata.yaml	2024-07-26 15:00:59.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/accounts_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,26 +0,0 @@
-friendly_name: Accounts table from production FxA database
-description: |
-  A mirror of the `accounts` table from the production FxA CloudSQL database, updated daily to match the current state of the table. Some fields in this table are converted to a more user-friendly, BigQuery-native format:
-    - `uid` is converted from bytes to a hex string
-    - boolean integer columns are converted to BOOL
-    - timestamp columns are converted to TIMESTAMP
-
-  See https://mozilla.github.io/ecosystem-platform/reference/database-structure#database-fxa
-owners:
-- akomar@mozilla.com
-labels:
-  application: accounts_backend
-  schedule: daily
-  dag: bqetl_accounts_backend_external
-  owner1: akomar
-scheduling:
-  dag_name: bqetl_accounts_backend_external
-  date_partition_parameter: null
-bigquery: null
-workgroup_access:
-- role: roles/bigquery.dataViewer
-  members:
-  - workgroup:accounts-confidential
-references:
-  query.sql:
-  - EXTERNAL_QUERY('moz-fx-fxa-prod.us.fxa-rds-prod-prod-fxa', 'SELECT\n         uid,\n         normalizedEmail,\n         email,\n         emailVerified,\n         verifierVersion,\n         verifierSetAt,\n         createdAt,\n         locale,\n         lockedAt,\n         profileChangedAt,\n         keysChangedAt,\n         ecosystemAnonId,\n         disabledAt,\n         metricsOptOutAt\n       FROM\n         accounts\n    ')
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/accounts_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/accounts_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/accounts_v1/query.sql	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/accounts_v1/query.sql	1970-01-01 00:00:00.000000000 +0000
@@ -1,37 +0,0 @@
-SELECT
-  TO_HEX(uid) AS uid,
-  normalizedEmail,
-  email,
-  SAFE_CAST(emailVerified AS BOOL) AS emailVerified,
-  verifierVersion,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(verifierSetAt AS INT)) AS verifierSetAt,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(createdAt AS INT)) AS createdAt,
-  locale,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(lockedAt AS INT)) AS lockedAt,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(profileChangedAt AS INT)) AS profileChangedAt,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(keysChangedAt AS INT)) AS keysChangedAt,
-  ecosystemAnonId,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(disabledAt AS INT)) AS disabledAt,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(metricsOptOutAt AS INT)) AS metricsOptOutAt,
-FROM
-  EXTERNAL_QUERY(
-    "moz-fx-fxa-prod.us.fxa-rds-prod-prod-fxa",
-    """SELECT
-         uid,
-         normalizedEmail,
-         email,
-         emailVerified,
-         verifierVersion,
-         verifierSetAt,
-         createdAt,
-         locale,
-         lockedAt,
-         profileChangedAt,
-         keysChangedAt,
-         ecosystemAnonId,
-         disabledAt,
-         metricsOptOutAt
-       FROM
-         accounts
-    """
-  )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/accounts_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/accounts_v1/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/accounts_v1/schema.yaml	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/accounts_v1/schema.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,47 +0,0 @@
-fields:
-  - name: uid
-    type: STRING
-    mode: NULLABLE
-    description: |-
-      Account ID in hexadecimal format.
-      FxA stores this as bytes, for purposes of logging or integration
-      with other systems we convert it to a hex string.
-  - name: normalizedEmail
-    type: STRING
-    mode: NULLABLE
-  - name: email
-    type: STRING
-    mode: NULLABLE
-  - name: emailVerified
-    type: BOOLEAN
-    mode: NULLABLE
-  - name: verifierVersion
-    type: INTEGER
-    mode: NULLABLE
-  - name: verifierSetAt
-    type: TIMESTAMP
-    mode: NULLABLE
-  - name: createdAt
-    type: TIMESTAMP
-    mode: NULLABLE
-  - name: locale
-    type: STRING
-    mode: NULLABLE
-  - name: lockedAt
-    type: TIMESTAMP
-    mode: NULLABLE
-  - name: profileChangedAt
-    type: TIMESTAMP
-    mode: NULLABLE
-  - name: keysChangedAt
-    type: TIMESTAMP
-    mode: NULLABLE
-  - name: ecosystemAnonId
-    type: STRING
-    mode: NULLABLE
-  - name: disabledAt
-    type: TIMESTAMP
-    mode: NULLABLE
-  - name: metricsOptOutAt
-    type: TIMESTAMP
-    mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/dataset_metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/dataset_metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/dataset_metadata.yaml	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/dataset_metadata.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,14 +0,0 @@
-friendly_name: Firefox Accounts Backend External
-description: |-
-  Data extracted from the FxA backend services databases.
-  See https://mozilla.github.io/ecosystem-platform/reference/database-structure#database-fxa for more information.
-
-  Access to this dataset is restricted to accounts-confidential workgroup because some tables here contain sensitive data.
-  `mozilla-confidential` user-facing views are available in the `accounts_backend` dataset.
-dataset_base_acl: restricted
-user_facing: false
-labels: {}
-workgroup_access:
-- role: roles/bigquery.dataViewer
-  members:
-  - workgroup:accounts-confidential
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/emails_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/emails_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/emails_v1/metadata.yaml	2024-07-26 15:00:59.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/emails_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,28 +0,0 @@
-friendly_name: Emails table from production FxA database
-description: |-
-  A mirror of the `emails` table from the production FxA CloudSQL database,
-  updated daily to match the current state of the table.
-  Some fields in this table are converted to a more user-friendly, BigQuery-native format:
-    - `uid` is converted from bytes to a hex string
-    - boolean integer columns are converted to BOOL
-    - timestamp columns are converted to TIMESTAMP
-
-  See https://mozilla.github.io/ecosystem-platform/reference/database-structure#database-fxa
-owners:
-- akomar@mozilla.com
-labels:
-  application: accounts_backend
-  schedule: daily
-  dag: bqetl_accounts_backend_external
-  owner1: akomar
-scheduling:
-  dag_name: bqetl_accounts_backend_external
-  date_partition_parameter: null
-bigquery: null
-workgroup_access:
-- role: roles/bigquery.dataViewer
-  members:
-  - workgroup:accounts-confidential
-references:
-  query.sql:
-  - EXTERNAL_QUERY('moz-fx-fxa-prod.us.fxa-rds-prod-prod-fxa', 'SELECT\n         id,\n         normalizedEmail,\n         email,\n         uid,\n         isVerified,\n         isPrimary,\n         verifiedAt,\n         createdAt\n       FROM\n         emails\n    ')
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/emails_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/emails_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/emails_v1/query.sql	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/emails_v1/query.sql	1970-01-01 00:00:00.000000000 +0000
@@ -1,25 +0,0 @@
-SELECT
-  id,
-  normalizedEmail,
-  email,
-  TO_HEX(uid) AS uid,
-  SAFE_CAST(isVerified AS BOOL) AS isVerified,
-  SAFE_CAST(isPrimary AS BOOL) AS isPrimary,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(verifiedAt AS INT)) AS verifiedAt,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(createdAt AS INT)) AS createdAt,
-FROM
-  EXTERNAL_QUERY(
-    "moz-fx-fxa-prod.us.fxa-rds-prod-prod-fxa",
-    """SELECT
-         id,
-         normalizedEmail,
-         email,
-         uid,
-         isVerified,
-         isPrimary,
-         verifiedAt,
-         createdAt
-       FROM
-         emails
-    """
-  )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/emails_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/emails_v1/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/emails_v1/schema.yaml	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/emails_v1/schema.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,29 +0,0 @@
-fields:
-- name: id
-  type: INTEGER
-  mode: NULLABLE
-- name: normalizedEmail
-  type: STRING
-  mode: NULLABLE
-- name: email
-  type: STRING
-  mode: NULLABLE
-- name: uid
-  type: STRING
-  mode: NULLABLE
-  description: |-
-    Account ID in hexadecimal format.
-    FxA stores this as bytes, for purposes of logging or integration
-    with other systems we convert it to a hex string.
-- name: isVerified
-  type: BOOLEAN
-  mode: NULLABLE
-- name: isPrimary
-  type: BOOLEAN
-  mode: NULLABLE
-- name: verifiedAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: createdAt
-  type: TIMESTAMP
-  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_accounts_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_accounts_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_accounts_v1/metadata.yaml	2024-07-26 15:00:59.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_accounts_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,26 +0,0 @@
-friendly_name: Accounts table from FxA database in stage environment
-description: |
-  A mirror of the `accounts` table from the staging FxA CloudSQL database, updated daily to match the current state of the table. Some fields in this table are converted to a more user-friendly, BigQuery-native format:
-    - `uid` is converted from bytes to a hex string
-    - boolean integer columns are converted to BOOL
-    - timestamp columns are converted to TIMESTAMP
-
-  See https://mozilla.github.io/ecosystem-platform/reference/database-structure#database-fxa
-owners:
-- akomar@mozilla.com
-labels:
-  application: accounts_backend
-  schedule: daily
-  dag: bqetl_accounts_backend_external
-  owner1: akomar
-scheduling:
-  dag_name: bqetl_accounts_backend_external
-  date_partition_parameter: null
-bigquery: null
-workgroup_access:
-- role: roles/bigquery.dataViewer
-  members:
-  - workgroup:accounts-confidential
-references:
-  query.sql:
-  - EXTERNAL_QUERY('moz-fx-fxa-nonprod.us.fxa-rds-nonprod-stage-fxa', 'SELECT\n         uid,\n         normalizedEmail,\n         email,\n         emailVerified,\n         verifierVersion,\n         verifierSetAt,\n         createdAt,\n         locale,\n         lockedAt,\n         profileChangedAt,\n         keysChangedAt,\n         ecosystemAnonId,\n         disabledAt,\n         metricsOptOutAt\n       FROM\n         accounts\n    ')
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_accounts_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_accounts_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_accounts_v1/query.sql	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_accounts_v1/query.sql	1970-01-01 00:00:00.000000000 +0000
@@ -1,37 +0,0 @@
-SELECT
-  TO_HEX(uid) AS uid,
-  normalizedEmail,
-  email,
-  SAFE_CAST(emailVerified AS BOOL) AS emailVerified,
-  verifierVersion,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(verifierSetAt AS INT)) AS verifierSetAt,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(createdAt AS INT)) AS createdAt,
-  locale,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(lockedAt AS INT)) AS lockedAt,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(profileChangedAt AS INT)) AS profileChangedAt,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(keysChangedAt AS INT)) AS keysChangedAt,
-  ecosystemAnonId,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(disabledAt AS INT)) AS disabledAt,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(metricsOptOutAt AS INT)) AS metricsOptOutAt,
-FROM
-  EXTERNAL_QUERY(
-    "moz-fx-fxa-nonprod.us.fxa-rds-nonprod-stage-fxa",
-    """SELECT
-         uid,
-         normalizedEmail,
-         email,
-         emailVerified,
-         verifierVersion,
-         verifierSetAt,
-         createdAt,
-         locale,
-         lockedAt,
-         profileChangedAt,
-         keysChangedAt,
-         ecosystemAnonId,
-         disabledAt,
-         metricsOptOutAt
-       FROM
-         accounts
-    """
-  )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_accounts_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_accounts_v1/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_accounts_v1/schema.yaml	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_accounts_v1/schema.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,47 +0,0 @@
-fields:
-- name: uid
-  type: STRING
-  mode: NULLABLE
-  description: |-
-    Account ID in hexadecimal format.
-    FxA stores this as bytes, for purposes of logging or integration
-    with other systems we convert it to a hex string.
-- name: normalizedEmail
-  type: STRING
-  mode: NULLABLE
-- name: email
-  type: STRING
-  mode: NULLABLE
-- name: emailVerified
-  type: BOOLEAN
-  mode: NULLABLE
-- name: verifierVersion
-  type: INTEGER
-  mode: NULLABLE
-- name: verifierSetAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: createdAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: locale
-  type: STRING
-  mode: NULLABLE
-- name: lockedAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: profileChangedAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: keysChangedAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: ecosystemAnonId
-  type: STRING
-  mode: NULLABLE
-- name: disabledAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: metricsOptOutAt
-  type: TIMESTAMP
-  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_emails_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_emails_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_emails_v1/metadata.yaml	2024-07-26 15:00:59.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_emails_v1/metadata.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,28 +0,0 @@
-friendly_name: Emails table from FxA database in stage environment
-description: |-
-  A mirror of the `emails` table from the staging FxA CloudSQL database,
-  updated daily to match the current state of the table.
-  Some fields in this table are converted to a more user-friendly, BigQuery-native format:
-    - `uid` is converted from bytes to a hex string
-    - boolean integer columns are converted to BOOL
-    - timestamp columns are converted to TIMESTAMP
-
-  See https://mozilla.github.io/ecosystem-platform/reference/database-structure#database-fxa
-owners:
-- akomar@mozilla.com
-labels:
-  application: accounts_backend
-  schedule: daily
-  dag: bqetl_accounts_backend_external
-  owner1: akomar
-scheduling:
-  dag_name: bqetl_accounts_backend_external
-  date_partition_parameter: null
-bigquery: null
-workgroup_access:
-- role: roles/bigquery.dataViewer
-  members:
-  - workgroup:accounts-confidential
-references:
-  query.sql:
-  - EXTERNAL_QUERY('moz-fx-fxa-nonprod.us.fxa-rds-nonprod-stage-fxa', 'SELECT\n         id,\n         normalizedEmail,\n         email,\n         uid,\n         isVerified,\n         isPrimary,\n         verifiedAt,\n         createdAt\n       FROM\n         emails\n    ')
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_emails_v1/query.sql /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_emails_v1/query.sql
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_emails_v1/query.sql	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_emails_v1/query.sql	1970-01-01 00:00:00.000000000 +0000
@@ -1,25 +0,0 @@
-SELECT
-  id,
-  normalizedEmail,
-  email,
-  TO_HEX(uid) AS uid,
-  SAFE_CAST(isVerified AS BOOL) AS isVerified,
-  SAFE_CAST(isPrimary AS BOOL) AS isPrimary,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(verifiedAt AS INT)) AS verifiedAt,
-  SAFE.TIMESTAMP_MILLIS(SAFE_CAST(createdAt AS INT)) AS createdAt,
-FROM
-  EXTERNAL_QUERY(
-    "moz-fx-fxa-nonprod.us.fxa-rds-nonprod-stage-fxa",
-    """SELECT
-         id,
-         normalizedEmail,
-         email,
-         uid,
-         isVerified,
-         isPrimary,
-         verifiedAt,
-         createdAt
-       FROM
-         emails
-    """
-  )
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_emails_v1/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_emails_v1/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_emails_v1/schema.yaml	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/accounts_backend_external/nonprod_emails_v1/schema.yaml	1970-01-01 00:00:00.000000000 +0000
@@ -1,29 +0,0 @@
-fields:
-- name: id
-  type: INTEGER
-  mode: NULLABLE
-- name: normalizedEmail
-  type: STRING
-  mode: NULLABLE
-- name: email
-  type: STRING
-  mode: NULLABLE
-- name: uid
-  type: STRING
-  mode: NULLABLE
-  description: |-
-    Account ID in hexadecimal format.
-    FxA stores this as bytes, for purposes of logging or integration
-    with other systems we convert it to a hex string.
-- name: isVerified
-  type: BOOLEAN
-  mode: NULLABLE
-- name: isPrimary
-  type: BOOLEAN
-  mode: NULLABLE
-- name: verifiedAt
-  type: TIMESTAMP
-  mode: NULLABLE
-- name: createdAt
-  type: TIMESTAMP
-  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates/schema.yaml	2024-07-26 15:10:30.000000000 +0000
@@ -1,49 +1,49 @@
 fields:
-- mode: NULLABLE
-  name: submission_date
+- name: submission_date
   type: DATE
-- mode: NULLABLE
-  name: source
+  mode: NULLABLE
+- name: source
   type: STRING
-- mode: NULLABLE
-  name: event_type
+  mode: NULLABLE
+- name: event_type
   type: STRING
-- mode: NULLABLE
-  name: form_factor
+  mode: NULLABLE
+- name: form_factor
   type: STRING
-- mode: NULLABLE
-  name: country
+  mode: NULLABLE
+- name: country
   type: STRING
-- mode: NULLABLE
-  name: subdivision1
+  mode: NULLABLE
+- name: subdivision1
   type: STRING
-- mode: NULLABLE
-  name: advertiser
+  mode: NULLABLE
+- name: advertiser
   type: STRING
-- mode: NULLABLE
-  name: release_channel
+  mode: NULLABLE
+- name: release_channel
   type: STRING
-- mode: NULLABLE
-  name: position
+  mode: NULLABLE
+- name: position
   type: INTEGER
-- mode: NULLABLE
-  name: provider
+  mode: NULLABLE
+- name: provider
   type: STRING
-- mode: NULLABLE
-  name: match_type
+  mode: NULLABLE
+- name: match_type
   type: STRING
-- mode: NULLABLE
-  name: normalized_os
+  mode: NULLABLE
+- name: normalized_os
   type: STRING
-- mode: NULLABLE
-  name: suggest_data_sharing_enabled
+  mode: NULLABLE
+- name: suggest_data_sharing_enabled
   type: BOOLEAN
-- mode: NULLABLE
-  name: event_count
+  mode: NULLABLE
+- name: event_count
   type: INTEGER
-- mode: NULLABLE
-  name: user_count
+  mode: NULLABLE
+- name: user_count
   type: INTEGER
-- mode: NULLABLE
-  name: query_type
+  mode: NULLABLE
+- name: query_type
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/contextual_services/event_aggregates_suggest/schema.yaml	2024-07-26 15:10:30.000000000 +0000
@@ -1,40 +1,40 @@
 fields:
-- mode: NULLABLE
-  name: submission_date
+- name: submission_date
   type: DATE
-- mode: NULLABLE
-  name: form_factor
+  mode: NULLABLE
+- name: form_factor
   type: STRING
-- mode: NULLABLE
-  name: country
+  mode: NULLABLE
+- name: country
   type: STRING
-- mode: NULLABLE
-  name: advertiser
+  mode: NULLABLE
+- name: advertiser
   type: STRING
-- mode: NULLABLE
-  name: normalized_os
+  mode: NULLABLE
+- name: normalized_os
   type: STRING
-- mode: NULLABLE
-  name: release_channel
+  mode: NULLABLE
+- name: release_channel
   type: STRING
-- mode: NULLABLE
-  name: position
+  mode: NULLABLE
+- name: position
   type: INTEGER
-- mode: NULLABLE
-  name: provider
+  mode: NULLABLE
+- name: provider
   type: STRING
-- mode: NULLABLE
-  name: match_type
+  mode: NULLABLE
+- name: match_type
   type: STRING
-- mode: NULLABLE
-  name: suggest_data_sharing_enabled
+  mode: NULLABLE
+- name: suggest_data_sharing_enabled
   type: BOOLEAN
-- mode: NULLABLE
-  name: impression_count
+  mode: NULLABLE
+- name: impression_count
   type: INTEGER
-- mode: NULLABLE
-  name: click_count
+  mode: NULLABLE
+- name: click_count
   type: INTEGER
-- mode: NULLABLE
-  name: query_type
+  mode: NULLABLE
+- name: query_type
   type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_clients/schema.yaml	2024-07-26 15:09:04.000000000 +0000
@@ -26,6 +26,9 @@
 - name: adjust_network
   type: STRING
   mode: NULLABLE
+- name: install_source
+  type: STRING
+  mode: NULLABLE
 - name: retained_week_2
   type: BOOLEAN
   mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/fenix/funnel_retention_week_4/schema.yaml	2024-07-26 15:09:05.000000000 +0000
@@ -48,6 +48,10 @@
   description: 'The type of source of a client installation.
 
     '
+- name: install_source
+  type: STRING
+  mode: NULLABLE
+  description: null
 - name: new_profiles
   type: INTEGER
   mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/focus_android/use_counters/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/focus_android/use_counters/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/focus_android/use_counters/schema.yaml	2024-07-26 14:59:53.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/focus_android/use_counters/schema.yaml	2024-07-26 15:15:03.000000000 +0000
@@ -1,11 +1,11 @@
 fields:
 - name: normalized_app_id
-  mode: NULLABLE
   type: STRING
+  mode: NULLABLE
   description: App ID of the channel data was received from
 - name: normalized_channel
-  mode: NULLABLE
   type: STRING
+  mode: NULLABLE
   description: Normalized channel name
 - name: additional_properties
   type: STRING
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_page/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_page/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_page/schema.yaml	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_page/schema.yaml	2024-07-26 15:09:56.000000000 +0000
@@ -6,9 +6,10 @@
 - name: site_url
   type: STRING
   mode: NULLABLE
-  description: |-
-    For domain properties, this will be `sc-domain:` followed by the domain name.
-    For URL-prefix properties, it will be the full URL of the property definition.
+  description: 'For domain properties, this will be `sc-domain:` followed by the domain
+    name.
+
+    For URL-prefix properties, it will be the full URL of the property definition.'
 - name: site_domain_name
   type: STRING
   mode: NULLABLE
@@ -16,103 +17,110 @@
 - name: page_url
   type: STRING
   mode: NULLABLE
-  description: |-
-    The final page URL linked by a search result after any skip redirects.
-    This will be null for anonymized Discover impressions.
+  description: 'The final page URL linked by a search result after any skip redirects.
+
+    This will be null for anonymized Discover impressions.'
 - name: page_domain_name
   type: STRING
   mode: NULLABLE
-  description: |-
-    Domain name of the page URL.
-    This will be null for anonymized Discover impressions.
+  description: 'Domain name of the page URL.
+
+    This will be null for anonymized Discover impressions.'
 - name: page_path
   type: STRING
   mode: NULLABLE
-  description: |-
-    The path part of the page URL.
-    This will be null for anonymized Discover impressions.
+  description: 'The path part of the page URL.
+
+    This will be null for anonymized Discover impressions.'
 - name: localized_site_code
   type: STRING
   mode: NULLABLE
-  description: |-
-    Localized site code such as `en-US` or `de` found in the first segment of the page URL path (if any).
-    This will be null for anonymized Discover impressions.
+  description: 'Localized site code such as `en-US` or `de` found in the first segment
+    of the page URL path (if any).
+
+    This will be null for anonymized Discover impressions.'
 - name: localized_site
   type: STRING
   mode: NULLABLE
-  description: |-
-    Description of the localized site language and/or country based on `localized_site_code` (if any).
-    This will be null for anonymized Discover impressions.
+  description: 'Description of the localized site language and/or country based on
+    `localized_site_code` (if any).
+
+    This will be null for anonymized Discover impressions.'
 - name: localized_site_language_code
   type: STRING
   mode: NULLABLE
-  description: |-
-    Localized site language code in ISO-639-alpha-2 format found in the first segment of the page URL path (if any).
-    This will be null for anonymized Discover impressions.
+  description: 'Localized site language code in ISO-639-alpha-2 format found in the
+    first segment of the page URL path (if any).
+
+    This will be null for anonymized Discover impressions.'
 - name: localized_site_language
   type: STRING
   mode: NULLABLE
-  description: |-
-    Localized site language based on `localized_site_language_code` (if any).
-    This will be null for anonymized Discover impressions.
+  description: 'Localized site language based on `localized_site_language_code` (if
+    any).
+
+    This will be null for anonymized Discover impressions.'
 - name: query
   type: STRING
   mode: NULLABLE
-  description: |-
-    The search query.
-    This will be null for anonymized search impressions, and all Discover and Google News search impressions.
+  description: 'The search query.
+
+    This will be null for anonymized search impressions, and all Discover and Google
+    News search impressions.'
 - name: query_type
   type: STRING
   mode: NULLABLE
-  description: |-
-    Type of search query:
-      * Anonymized: Query was redacted by Google to protect the users' privacy.
-      * Brand: Query contained one or more Mozilla brand keywords.
-      * Non-Brand: Query didn't contain any Mozilla brand keywords.
-      * Unknown: Query couldn't be classified.
-    This will be null for all Discover and Google News search impressions.
+  description: "Type of search query:\n  * Anonymized: Query was redacted by Google\
+    \ to protect the users' privacy.\n  * Brand: Query contained one or more Mozilla\
+    \ brand keywords.\n  * Non-Brand: Query didn't contain any Mozilla brand keywords.\n\
+    \  * Unknown: Query couldn't be classified.\nThis will be null for all Discover\
+    \ and Google News search impressions."
 - name: is_anonymized
   type: BOOLEAN
   mode: NULLABLE
-  description: |-
-    Whether Google has anonymized the search impression to protect the users' privacy.
+  description: 'Whether Google has anonymized the search impression to protect the
+    users'' privacy.
+
     The `query` field will be null for anonymized search impressions.
-    The `country_code`, `page_url`, and related fields will be null for anonymized Discover impressions.
+
+    The `country_code`, `page_url`, and related fields will be null for anonymized
+    Discover impressions.'
 - name: has_good_page_experience
   type: BOOLEAN
   mode: NULLABLE
-  description: |-
-    Whether Google Search considers the page to be providing a good page experience.
-    This will be null when the source data wasn't exported directly to BigQuery by Google.
+  description: 'Whether Google Search considers the page to be providing a good page
+    experience.
+
+    This will be null when the source data wasn''t exported directly to BigQuery by
+    Google.'
 - name: search_type
   type: STRING
   mode: NULLABLE
-  description: |-
-    Where the link was seen by the user:
-      * Web: In Google Search's default "All" tab.
-      * Image: In Google Search's "Images" tab.
-      * Video: In Google Search's "Videos" tab.
-      * News: In Google Search's "News" tab.
-      * Discover: In Google's Discover feed.
-      * Google News: On news.google.com or in the Google News app on Android and iOS.
+  description: "Where the link was seen by the user:\n  * Web: In Google Search's\
+    \ default \"All\" tab.\n  * Image: In Google Search's \"Images\" tab.\n  * Video:\
+    \ In Google Search's \"Videos\" tab.\n  * News: In Google Search's \"News\" tab.\n\
+    \  * Discover: In Google's Discover feed.\n  * Google News: On news.google.com\
+    \ or in the Google News app on Android and iOS."
 - name: search_appearance
   type: STRING
   mode: NULLABLE
-  description: |-
-    How the search result appeared (e.g. normal result, translated result, video).
-    This will be null when the source data wasn't exported directly to BigQuery by Google.
+  description: 'How the search result appeared (e.g. normal result, translated result,
+    video).
+
+    This will be null when the source data wasn''t exported directly to BigQuery by
+    Google.'
 - name: user_country_code
   type: STRING
   mode: NULLABLE
-  description: |-
-    Country from which the user was searching, in ISO-3166-1-alpha-3 format.
-    This will be null for anonymized Discover impressions.
+  description: 'Country from which the user was searching, in ISO-3166-1-alpha-3 format.
+
+    This will be null for anonymized Discover impressions.'
 - name: user_country
   type: STRING
   mode: NULLABLE
-  description: |-
-    Country from which the user was searching.
-    This will be null for anonymized Discover impressions.
+  description: 'Country from which the user was searching.
+
+    This will be null for anonymized Discover impressions.'
 - name: user_region
   type: STRING
   mode: NULLABLE
@@ -124,13 +132,15 @@
 - name: device_type
   type: STRING
   mode: NULLABLE
-  description: |-
-    The type of device on which the user was searching: Desktop, Mobile, or Tablet.
-    This will be null for Discover impressions.
+  description: 'The type of device on which the user was searching: Desktop, Mobile,
+    or Tablet.
+
+    This will be null for Discover impressions.'
 - name: impressions
   type: INTEGER
   mode: NULLABLE
-  description: The number of times that search results with a link to the page were shown to a user.
+  description: The number of times that search results with a link to the page were
+    shown to a user.
 - name: clicks
   type: INTEGER
   mode: NULLABLE
@@ -138,6 +148,7 @@
 - name: average_position
   type: FLOAT
   mode: NULLABLE
-  description: |-
-    The average position of the page in the search results, where `1` is the topmost position.
-    This will be null for Discover and Google News search impressions.
+  description: 'The average position of the page in the search results, where `1`
+    is the topmost position.
+
+    This will be null for Discover and Google News search impressions.'
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_site/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_site/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_site/schema.yaml	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/google_search_console/search_impressions_by_site/schema.yaml	2024-07-26 15:09:57.000000000 +0000
@@ -6,9 +6,10 @@
 - name: site_url
   type: STRING
   mode: NULLABLE
-  description: |-
-    For domain properties, this will be `sc-domain:` followed by the domain name.
-    For URL-prefix properties, it will be the full URL of the property definition.
+  description: 'For domain properties, this will be `sc-domain:` followed by the domain
+    name.
+
+    For URL-prefix properties, it will be the full URL of the property definition.'
 - name: site_domain_name
   type: STRING
   mode: NULLABLE
@@ -20,27 +21,23 @@
 - name: query_type
   type: STRING
   mode: NULLABLE
-  description: |-
-    Type of search query:
-      * Anonymized: Query was redacted by Google to protect the users' privacy.
-      * Brand: Query contained one or more Mozilla brand keywords.
-      * Non-Brand: Query didn't contain any Mozilla brand keywords.
-      * Unknown: Query couldn't be classified.
+  description: "Type of search query:\n  * Anonymized: Query was redacted by Google\
+    \ to protect the users' privacy.\n  * Brand: Query contained one or more Mozilla\
+    \ brand keywords.\n  * Non-Brand: Query didn't contain any Mozilla brand keywords.\n\
+    \  * Unknown: Query couldn't be classified."
 - name: is_anonymized
   type: BOOLEAN
   mode: NULLABLE
-  description: |-
-    Whether Google has anonymized the search impression to protect the users' privacy.
-    The `query` field will be null for anonymized search impressions.
+  description: 'Whether Google has anonymized the search impression to protect the
+    users'' privacy.
+
+    The `query` field will be null for anonymized search impressions.'
 - name: search_type
   type: STRING
   mode: NULLABLE
-  description: |-
-    Where the link was seen by the user:
-      * Web: In Google Search's default "All" tab.
-      * Image: In Google Search's "Images" tab.
-      * Video: In Google Search's "Videos" tab.
-      * News: In Google Search's "News" tab.
+  description: "Where the link was seen by the user:\n  * Web: In Google Search's\
+    \ default \"All\" tab.\n  * Image: In Google Search's \"Images\" tab.\n  * Video:\
+    \ In Google Search's \"Videos\" tab.\n  * News: In Google Search's \"News\" tab."
 - name: user_country_code
   type: STRING
   mode: NULLABLE
@@ -60,17 +57,20 @@
 - name: device_type
   type: STRING
   mode: NULLABLE
-  description: |-
-    The type of device on which the user was searching: Desktop, Mobile, or Tablet.
+  description: 'The type of device on which the user was searching: Desktop, Mobile,
+    or Tablet.'
 - name: impressions
   type: INTEGER
   mode: NULLABLE
-  description: The number of times that search results with at least one link to the site were shown to a user.
+  description: The number of times that search results with at least one link to the
+    site were shown to a user.
 - name: clicks
   type: INTEGER
   mode: NULLABLE
-  description: The number of times a user clicked at least one search result link to the site.
+  description: The number of times a user clicked at least one search result link
+    to the site.
 - name: average_top_position
   type: FLOAT
   mode: NULLABLE
-  description: The average top position of the site in the search results, where `1` is the topmost position.
+  description: The average top position of the site in the search results, where `1`
+    is the topmost position.
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/internet_outages/global_outages/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/internet_outages/global_outages/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/internet_outages/global_outages/schema.yaml	1970-01-01 00:00:00.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/internet_outages/global_outages/schema.yaml	2024-07-26 15:10:32.000000000 +0000
@@ -0,0 +1,58 @@
+fields:
+- name: country
+  type: STRING
+  mode: NULLABLE
+- name: city
+  type: STRING
+  mode: NULLABLE
+- name: geo_subdivision1
+  type: STRING
+  mode: NULLABLE
+- name: geo_subdivision2
+  type: STRING
+  mode: NULLABLE
+- name: datetime
+  type: TIMESTAMP
+  mode: NULLABLE
+- name: proportion_undefined
+  type: FLOAT
+  mode: NULLABLE
+- name: proportion_timeout
+  type: FLOAT
+  mode: NULLABLE
+- name: proportion_abort
+  type: FLOAT
+  mode: NULLABLE
+- name: proportion_unreachable
+  type: FLOAT
+  mode: NULLABLE
+- name: proportion_terminated
+  type: FLOAT
+  mode: NULLABLE
+- name: proportion_channel_open
+  type: FLOAT
+  mode: NULLABLE
+- name: avg_dns_success_time
+  type: FLOAT
+  mode: NULLABLE
+- name: missing_dns_success
+  type: FLOAT
+  mode: NULLABLE
+- name: avg_dns_failure_time
+  type: FLOAT
+  mode: NULLABLE
+- name: missing_dns_failure
+  type: FLOAT
+  mode: NULLABLE
+- name: count_dns_failure
+  type: FLOAT
+  mode: NULLABLE
+- name: ssl_error_prop
+  type: FLOAT
+  mode: NULLABLE
+- name: avg_tls_handshake_time
+  type: FLOAT
+  mode: NULLABLE
+- name: dataset_version
+  type: STRING
+  mode: NULLABLE
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/klar_android/use_counters/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/klar_android/use_counters/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/klar_android/use_counters/schema.yaml	2024-07-26 14:59:53.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/klar_android/use_counters/schema.yaml	2024-07-26 15:14:08.000000000 +0000
@@ -1,11 +1,11 @@
 fields:
 - name: normalized_app_id
-  mode: NULLABLE
   type: STRING
+  mode: NULLABLE
   description: App ID of the channel data was received from
 - name: normalized_channel
-  mode: NULLABLE
   type: STRING
+  mode: NULLABLE
   description: Normalized channel name
 - name: additional_properties
   type: STRING
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/event_monitoring_live_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/event_monitoring_live_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/event_monitoring_live_v1/metadata.yaml	2024-07-26 15:00:24.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/event_monitoring_live_v1/metadata.yaml	2024-07-26 15:16:06.000000000 +0000
@@ -1,10 +1,19 @@
+friendly_name: Event Monitoring Live
+description: |-
+  Materialized view of experimentation related events
+  coming from monitor_frontend.
+owners:
+- ascholtz@mozilla.com
+- akomar@mozilla.com
+labels:
+  materialized_view: true
+  owner1: ascholtz
+  owner2: akomar
+bigquery: null
 workgroup_access:
 - role: roles/bigquery.dataViewer
   members:
   - workgroup:mozilla-confidential
-  - workgroup:dataops-managed/external-fides
-
-# Generated by bigquery_etl.dependency
 references:
   materialized_view.sql:
   - moz-fx-data-shared-prod.monitor_frontend_live.events_v1
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/monitor_dashboard_user_journey_funnels_v1/metadata.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/monitor_dashboard_user_journey_funnels_v1/metadata.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/monitor_dashboard_user_journey_funnels_v1/metadata.yaml	2024-07-26 15:00:25.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitor_frontend_derived/monitor_dashboard_user_journey_funnels_v1/metadata.yaml	2024-07-26 15:16:06.000000000 +0000
@@ -1,10 +1,26 @@
+friendly_name: Monitor Dashboard User Journey Funnels
+description: |-
+  Please provide a description for the query
+owners:
+- ksiegler@mozilla.org
+labels:
+  incremental: true
+  dag: bqetl_generated_funnels
+  owner1: ksiegler
+scheduling:
+  dag_name: bqetl_generated_funnels
+bigquery:
+  time_partitioning:
+    type: day
+    field: submission_date
+    require_partition_filter: false
+    expiration_days: null
+  range_partitioning: null
+  clustering: null
 workgroup_access:
 - role: roles/bigquery.dataViewer
   members:
   - workgroup:mozilla-confidential
-  - workgroup:dataops-managed/external-fides
-
-# Generated by bigquery_etl.dependency
 references:
   query.sql:
   - mozdata.monitor_frontend.events_unnested
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitoring/airflow_trigger/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring/airflow_trigger/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/monitoring/airflow_trigger/schema.yaml	2024-07-26 14:59:53.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/monitoring/airflow_trigger/schema.yaml	2024-07-26 15:10:38.000000000 +0000
@@ -17,3 +17,11 @@
 - name: triggerer_id
   type: INTEGER
   mode: NULLABLE
+- name: trigger_id
+  type: INTEGER
+  mode: NULLABLE
+  description: null
+- name: created_at
+  type: TIMESTAMP
+  mode: NULLABLE
+  description: null
diff -bur --no-dereference --new-file /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/geckoview_version/schema.yaml /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/geckoview_version/schema.yaml
--- /tmp/workspace/main-generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/geckoview_version/schema.yaml	2024-07-26 14:59:10.000000000 +0000
+++ /tmp/workspace/generated-sql/sql/moz-fx-data-shared-prod/org_mozilla_fenix/geckoview_version/schema.yaml	2024-07-26 15:10:05.000000000 +0000
@@ -1,7 +1,13 @@
 fields:
-- type: DATETIME
-  name: build_hour
-- type: INTEGER
-  name: geckoview_major_version
-- type: INTEGER
-  name: n_pings
+- name: build_hour
+  type: DATETIME
+  mode: NULLABLE
+  description: null
+- name: geckoview_major_version
+  type: INTEGER
+  mode: NULLABLE
+  description: null
+- name: n_pings
+  type: INTEGER
+  mode: NULLABLE
+  description: null

Link to full diff

Remove unused Accounts DB queries

3706f31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Remove unused Accounts DB queries #5978

Remove unused Accounts DB queries #5978

akkomar commented Jul 26, 2024 •

edited by data-sync-user

Loading

dataops-ci-bot commented Jul 26, 2024

Remove unused Accounts DB queries #5978

Are you sure you want to change the base?

Remove unused Accounts DB queries #5978

Conversation

akkomar commented Jul 26, 2024 • edited by data-sync-user Loading

dataops-ci-bot commented Jul 26, 2024

Integration report for "Remove unused Accounts DB queries"

sql.diff

akkomar commented Jul 26, 2024 •

edited by data-sync-user

Loading

`sql.diff`