source-postgres: Support _citext columns
#2166
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Arrays of `citext` are tricky because `citext` is an extension type which means that it has no stable OID, and as a result the PGX client library doesn't have any default type mappings for it. In the absence of any registered codec, values are always captured in their stringified text-protocol format. However our discovery logic still notices that it's an array and emits the appropriate schema for an array of text values, which means there's a mismatch and we're not satisfying our own schema. Which is bad. This commit fixes that by extending the "datatype tweaks" logic with a query against `pg_catalog.pg_type` looking for specific extension types we support (which currently is just `citext`) and for each result row we register custom handlers for the base type and arrays of that type. Since this is all getting fairly complex, I've also gone ahead and added test cases for `citext` and `_citext` datatype support. But of course those don't work against a database without that extension installed, so the CI database setup script has been modified to install it. (The `citext` extension is shipped as part of Postgres itself, it just isn't enabled by default, so we just have to run `CREATE EXTENSION citext`).
willdonnelly
added
the
change:unplanned
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description:
Arrays of
citextare tricky becausecitextis an extension type which means that it has no stable OID, and as a result the PGX client library doesn't have any default type mappings for it. In the absence of any registered codec, values are always captured in their stringified text-protocol format.However our discovery logic still notices that it's an array and emits the appropriate schema for an array of text values, which means there's a mismatch and we're not satisfying our own schema. Which is bad. While
citextis an extension type, it's a common one and the citext extension ships as part of the Postgres release, so I believe we should still support it as best we can.This commit fixes that by extending the "datatype tweaks" logic with a query against
pg_catalog.pg_typelooking for specific extension types we support (which currently is justcitext) and for each result row we register custom handlers for the base type and arrays of that type.Since this is all getting fairly complex, I've also gone ahead and added test cases for
citextand_citextdatatype support. But of course those don't work against a database without that extension installed, so the CI database setup script has been modified to install it. (Thecitextextension is shipped as part of Postgres itself, it just isn't enabled by default, so we just have to runCREATE EXTENSION citext).Workflow steps:
Columns of type
citext[]should just work going forward.This change is