From 7fe568e3690b1c9759cfd83c2e9bc4c4b30fbcfb Mon Sep 17 00:00:00 2001 From: Sergey Motornyuk Date: Tue, 26 Nov 2024 11:14:14 +0200 Subject: [PATCH] Deployed cda1cb6 with MkDocs version: 1.6.1 --- search/search_index.json | 2 +- sitemap.xml | 20 ++++++++++---------- sitemap.xml.gz | Bin 272 -> 272 bytes usage/transmutators/index.html | 14 +++++++------- 4 files changed, 18 insertions(+), 18 deletions(-) diff --git a/search/search_index.json b/search/search_index.json index 885b214..241a307 100644 --- a/search/search_index.json +++ b/search/search_index.json @@ -1 +1 @@ -{"config":{"lang":["en"],"separator":"[\\s\\-\\.\\_]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":""},{"location":"#ckanext-transmute","title":"ckanext-transmute","text":"

Pipeline for data validation and conversion using schemas.

Read the documentation for a full user guide.

"},{"location":"#quickstart","title":"Quickstart","text":"

Install the extension 

pip install ckanext-transmute\n

Add transmute to the list of enabled plugins in the CKAN config file.

Transform data using inline schema 

ckanapi action tsm_transmute root=example \\\n    data:'{\"greeting\": \"hello\"}' \\\n    schema:'{\n  \"root\": \"example\",\n  \"types\": {\n    \"example\": {\n      \"fields\": {\n        \"message\": {\n          \"validate_missing\": true,\n          \"validators\": [\n            [\n              \"tsm_concat\",\n              \"$greeting\",\n              \", \",\n              \"$name\",\n              \"!\"\n            ]\n          ],\n          \"weight\": 2\n        },\n        \"name\": {\n          \"default\": \"transmute\"\n        },\n        \"greeting\": {\n          \"default\": \"Hi\"\n        }\n      },\n      \"post-fields\": {\n        \"greeting\": {\n          \"remove\": true\n        },\n        \"name\": {\n          \"remove\": true\n        }\n      }\n    }\n  }\n}'\n

"},{"location":"#developer-installation","title":"Developer installation","text":"

Install the extension

git clone https://github.com/DataShades/ckanext-transmute.git\ncd ckanext-transmute\npip install -e '.[dev]'\n

Run tests

pytest\n
"},{"location":"#license","title":"License","text":"

AGPL

"},{"location":"api/","title":"API","text":""},{"location":"api/#transmute.logic.action.tsm_transmute","title":"tsm_transmute(context, data_dict)","text":"

Transmute data using the schema.

The function creates a deep copy of the data and performs all modifications on the copy.

PARAMETER DESCRIPTION data

A data dict to transmute

TYPE: dict[str, Any]

schema

schema to transmute data

TYPE: dict[str, Any]

root

a root schema type

TYPE: str

RETURNS DESCRIPTION dict[str, Any]

Transmuted data

"},{"location":"changelog/","title":"Changelog","text":"

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog and this project adheres to Semantic Versioning.

"},{"location":"changelog/#v170-2024-10-30","title":"v1.7.0 - 2024-10-30","text":"

Compare with first commit

"},{"location":"changelog/#features","title":"Features","text":""},{"location":"changelog/#bug-fixes","title":"Bug Fixes","text":""},{"location":"configuration/","title":"Configuration","text":""},{"location":"configuration/#ckanexttransmuteschemaname","title":"ckanext.transmute.schema.<NAME>","text":"

Path to the JSON file with definition of the named schema.

"},{"location":"installation/","title":"Installation","text":""},{"location":"installation/#requirements","title":"Requirements","text":"

Compatibility with core CKAN versions:

CKAN version Compatible? 2.9 no 2.10 yes 2.11 yes master yes

Note

It's recommended to install the extension via pip. If you are using GitHub version of the extension, stick to the vX.Y.Z tags to avoid breaking changes. Check the changelog before upgrading the extension.

"},{"location":"installation/#installation_1","title":"Installation","text":"

Install the extension

pip install ckanext-transmute\n

Add transmute to the ckan.plugins setting in your CKAN config file.

"},{"location":"interfaces/","title":"Interfaces","text":""},{"location":"interfaces/#transmute.interfaces.ITransmute","title":"ITransmute","text":"

Main extension point of ckanext-transmute.

"},{"location":"interfaces/#transmute.interfaces.ITransmute.get_transmutation_schemas","title":"get_transmutation_schemas()","text":"

Register definitions of named schemas.

These schemas can be reffered by name in code. In this way you can define static schema and apply in multiple places it to arbitrary data.

Example 
def get_transmutation_schemas(self):\n    person = {\n        \"fields\": {\n            \"age\": {\"validators\": [\"int_validator\"]},\n            \"name\": {\"default\": \"John Doe\"},\n        }\n    }\n\n    schema = {\n        \"root\": \"person\",\n        \"types\": {\"person\": person}\n    }\n\n    return {\"person\": schema}\n
RETURNS DESCRIPTION dict[str, dict[str, Any]]

Mapping with definitions of named schemas.

"},{"location":"interfaces/#transmute.interfaces.ITransmute.get_transmutators","title":"get_transmutators()","text":"

Register custom transmutation functions.

Example 
def get_transmutators(self):\n    return {\n        \"tsm_title_case\": tsm_title_case,\n        \"tsm_is_email\": tsm_is_email\n    }\n
RETURNS DESCRIPTION dict[str, Any]

Mapping with transmutaion functions.

"},{"location":"usage/","title":"Overview","text":"

ckanext-transmute registers an action tsm_transmute to transmute data using the provided conversion scheme. The action doesn't change the original data but creates a new data dict. There are two mandatory arguments: data and schema. data is a data dict you need to transform, and schema contains the rules describing all the transformation steps.

Typical use-case for it is transforming existing data, like this:

{\n  \"title\": \"Test-dataset\",\n  \"email\": \"test@test.ua\",\n  \"metadata_created\": \"\",\n  \"metadata_modified\": \"\",\n  \"metadata_reviewed\": \"\",\n  \"resources\": [\n    {\n      \"title\": \"test-res\",\n      \"extension\": \"xml\",\n      \"web\": \"https://stackoverflow.com/\",\n      \"sub-resources\": [\n        {\n          \"title\": \"sub-res\",\n          \"extension\": \"csv\",\n          \"extra\": \"should-be-removed\"\n        }\n      ]\n    },\n    {\n      \"title\": \"test-res2\",\n      \"extension\": \"csv\",\n      \"web\": \"https://stackoverflow.com/\"\n    }\n  ]\n}\n

into expected data, like this:

{\n    \"name\": \"test-dataset\",\n    \"email\": \"test@test.ua\",\n    \"metadata_created\": datetime.datetime(2022, 2, 3, 15, 54, 26, 359453),\n    \"metadata_modified\": datetime.datetime(2022, 2, 3, 15, 54, 26, 359453),\n    \"metadata_reviewed\": datetime.datetime(2022, 2, 3, 15, 54, 26, 359453),\n    \"attachments\": [\n        {\n            \"name\": \"test-res\",\n            \"format\": \"XML\",\n            \"url\": \"https://stackoverflow.com/\",\n            \"sub-resources\": [{\"name\": \"SUB-RES\", \"format\": \"CSV\"}]\n        },\n        {\n            \"name\": \"test-res2\",\n            \"format\": \"CSV\",\n            \"url\": \"https://stackoverflow.com/\"\n        }\n    ]\n}\n

To achieve this goal, the following schema definition can be used: 

{\n    \"root\": \"Dataset\",\n    \"types\": {\n        \"Dataset\": {\n            \"fields\": {\n                \"title\": {\n                    \"validators\": [\n                        \"tsm_string_only\",\n                        \"tsm_to_lowercase\",\n                        \"tsm_name_validator\",\n                    ],\n                    \"map\": \"name\",\n                },\n                \"resources\": {\n                    \"type\": \"Resource\",\n                    \"multiple\": True,\n                    \"map\": \"attachments\",\n                },\n                \"metadata_created\": {\n                    \"validators\": [\"tsm_isodate\"],\n                    \"default\": \"2022-02-03T15:54:26.359453\",\n                },\n                \"metadata_modified\": {\n                    \"validators\": [\"tsm_isodate\"],\n                    \"default_from\": \"metadata_created\",\n                },\n                \"metadata_reviewed\": {\n                    \"validators\": [\"tsm_isodate\"],\n                    \"replace_from\": \"metadata_modified\",\n                },\n            }\n        },\n        \"Resource\": {\n            \"fields\": {\n                \"title\": {\n                    \"validators\": [\"tsm_string_only\"],\n                    \"map\": \"name\",\n                },\n                \"extension\": {\n                    \"validators\": [\"tsm_string_only\", \"tsm_to_uppercase\"],\n                    \"map\": \"format\",\n                },\n                \"web\": {\n                    \"validators\": [\"tsm_string_only\"],\n                    \"map\": \"url\",\n                },\n                \"sub-resources\": {\n                    \"type\": \"Sub-Resource\",\n                    \"multiple\": True,\n                },\n            },\n        },\n        \"Sub-Resource\": {\n            \"fields\": {\n                \"title\": {\n                    \"validators\": [\"tsm_string_only\", \"tsm_to_uppercase\"],\n                    \"map\": \"name\",\n                },\n                \"extension\": {\n                    \"validators\": [\"tsm_string_only\", \"tsm_to_uppercase\"],\n                    \"map\": \"format\",\n                },\n                \"extra\": {\n                    \"remove\": True,\n                },\n            }\n        },\n    },\n}\n

This is an example of schema with nested types. The root field defines the type of the outer layer of data, while sub-resources field inside the definition of the root type contain type references to Sub-Resource definition.

"},{"location":"usage/schema/","title":"Schema","text":"

Transmutation schema represented by a dictionary that contains descriptions of all data types used for transmutation and the name of the root type.

{\n    \"root\": \"main\",\n    \"types\": {\n        \"main\": {},\n        \"secondary\": {}\n    }\n}\n

The root type is used for the initial transformation. If, during this transformation, some of multi-values fields contain reference to other types defined in schema, these types will be used for further transformation of data.

{\n    \"root\": \"main\",\n    \"types\": {\n        \"main\": {\n            \"drop_unknown_fields\": true,\n            \"fields\": {\n                \"child\": {\"type\": \"secondary\", \"multiple\": true}\n            }\n        },\n        \"secondary\": {\n            \"drop_unknown_fields\": true,\n            \"fields\": {\"name\": {}}\n        }\n    }\n}\n

Note

At the moment, only multivalued fields can be transformed using nested types. In future support for single-valued nested field will be added

"},{"location":"usage/transmutators/","title":"Transmutators","text":"

Transmutators are similar to CKAN validators. They accept the field and modify it. But unlike validators, transmutators work with field and have access to the whole schema definition.

Usually, transmutator is defined as a function with a single argument. This argument always receives the instance of validated field. It's a dataclass with

Transmutator modifies field in place and returns the whole field when job is done.

ckanext-transmute contains a number of transmutators that can be used without additional configuration. And if you need more, you can define a custom transmutator with the ITransmute interface.

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_concat","title":"tsm_concat(field, *strings)","text":"

Concatenate strings to build a new one.

Use $self to point on the current field value.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

strings

strings to concat with

TYPE: Any DEFAULT: ()

RETURNS DESCRIPTION Field

the same Field with new value

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_get_nested","title":"tsm_get_nested(field, *path)","text":"

Fetches a nested value from a field.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

path

Iterable with path segments

TYPE: str DEFAULT: ()

RAISES DESCRIPTION Invalid

raises if path doesn't exist

RETURNS DESCRIPTION Field

the same Field with new value

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_isodate","title":"tsm_isodate(field)","text":"

Validates datetime string Mutates an iso-like string to datetime object.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RAISES DESCRIPTION Invalid

raises if date format is incorrect

RETURNS DESCRIPTION Field

the same Field with casted value

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_list_mapper","title":"tsm_list_mapper(field, mapping, remove=False)","text":"

Maps values within a list to corresponding values from the provided dictionary.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

mapping

A dictionary representing the mapping of values.

TYPE: dict[Any, Any]

remove

If set to True, removes values from the list if they don't have a corresponding mapping. Defaults to False.

TYPE: bool DEFAULT: False

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_map_value","title":"tsm_map_value(field, test_value, if_same, if_different=SENTINEL)","text":"

Replace value with other value.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

test_value

value that will be compared to field value

TYPE: Any

if_same

value to use if test_value matches the field value

TYPE: Any

if_different

value to use if test_value does not matche the field value. Leave empty to keep original value of the field.

TYPE: Any DEFAULT: SENTINEL

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_mapper","title":"tsm_mapper(field, mapping, default=None)","text":"

Map a value with a new value. The initial value must serve as a key within a mapping dictionary, while the dict value will represent the updated value.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

mapping

A dictionary representing the mapping of values.

TYPE: dict[Any, Any]

default

The default value to be used when the key is not found. If the default value is not provided, the current value will be used as it.

TYPE: Any DEFAULT: None

RETURNS DESCRIPTION Field

the same Field with new value

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_name_validator","title":"tsm_name_validator(field)","text":"

Wrapper over CKAN default name_validator validator.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RAISES DESCRIPTION Invalid

if value is not a valid name

RETURNS DESCRIPTION Field

the same Field object if it's valid

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_stop_on_empty","title":"tsm_stop_on_empty(field)","text":"

Stop transmutation if field is empty.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RETURNS DESCRIPTION Field

the same Field

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_string_only","title":"tsm_string_only(field)","text":"

Validates if field.value is string.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RAISES DESCRIPTION Invalid

raises is the field.value is not string

RETURNS DESCRIPTION Field

the same Field object if it's valid

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_to_lowercase","title":"tsm_to_lowercase(field)","text":"

Casts string value to lowercase.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RETURNS DESCRIPTION Field

Field object with mutated string

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_to_string","title":"tsm_to_string(field)","text":"

Casts field.value to str.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RETURNS DESCRIPTION Field

the same Field with new value

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_to_uppercase","title":"tsm_to_uppercase(field)","text":"

Casts string value to uppercase.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RETURNS DESCRIPTION Field

Field object with mutated string

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_trim_string","title":"tsm_trim_string(field, max_length)","text":"

Trim string lenght.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

max_length

String max length

TYPE: int

RETURNS DESCRIPTION Field

the same Field object if it's valid

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_unique_only","title":"tsm_unique_only(field)","text":"

Preserve only unique values from list.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RETURNS DESCRIPTION Field

the same Field with new value

TYPE: Field

"},{"location":"usage/type/","title":"Type","text":"

Type description contains definition of its fields and a number of additional settings.

{\n    \"root\": \"main\",\n    \"types\": {\n        \"main\": {\n            \"drop_unknown_fields\": true,\n            \"fields\": {\n                \"first\": {},\n                \"second\": {}\n            }\n        }\n    }\n}\n

Every field either refers a different type if it's definded with multiple: true and type: TYPE_NAME, or contains inline definition. Inline fields are used most often and their definition is flexible enough to cover majority of use-cases.

{\n    \"root\": \"main\",\n    \"types\": {\n        \"main\": {\n            \"fields\": {\n                \"inline_field\": {\"default\": 42},\n                \"sub_type\": {\"multiple\": true, \"type\": \"secondary\"}\n            }\n        },\n        \"secondary\": {}\n    }\n}\n

Here's the list of attributes that can be used in the field definition:

Attribute Description map New name of the field validators List of transmutators applied to the field remove Flag that removes field from data when enabled default Default value if field is missing default_from Name of the field used as source of default value value Static value that replaces any existing value of the field replace_from Name of the field used as a source of value validate_missing Flag that applies validation even if data does not contains the field weight Weight that controls order of field processing"}]} \ No newline at end of file +{"config":{"lang":["en"],"separator":"[\\s\\-\\.\\_]+","pipeline":["stopWordFilter"]},"docs":[{"location":"","title":"Home","text":""},{"location":"#ckanext-transmute","title":"ckanext-transmute","text":"

Pipeline for data validation and conversion using schemas.

Read the documentation for a full user guide.

"},{"location":"#quickstart","title":"Quickstart","text":"

Install the extension 

pip install ckanext-transmute\n

Add transmute to the list of enabled plugins in the CKAN config file.

Transform data using inline schema 

ckanapi action tsm_transmute root=example \\\n    data:'{\"greeting\": \"hello\"}' \\\n    schema:'{\n  \"root\": \"example\",\n  \"types\": {\n    \"example\": {\n      \"fields\": {\n        \"message\": {\n          \"validate_missing\": true,\n          \"validators\": [\n            [\n              \"tsm_concat\",\n              \"$greeting\",\n              \", \",\n              \"$name\",\n              \"!\"\n            ]\n          ],\n          \"weight\": 2\n        },\n        \"name\": {\n          \"default\": \"transmute\"\n        },\n        \"greeting\": {\n          \"default\": \"Hi\"\n        }\n      },\n      \"post-fields\": {\n        \"greeting\": {\n          \"remove\": true\n        },\n        \"name\": {\n          \"remove\": true\n        }\n      }\n    }\n  }\n}'\n

"},{"location":"#developer-installation","title":"Developer installation","text":"

Install the extension

git clone https://github.com/DataShades/ckanext-transmute.git\ncd ckanext-transmute\npip install -e '.[dev]'\n

Run tests

pytest\n
"},{"location":"#license","title":"License","text":"

AGPL

"},{"location":"api/","title":"API","text":""},{"location":"api/#transmute.logic.action.tsm_transmute","title":"tsm_transmute(context, data_dict)","text":"

Transmute data using the schema.

The function creates a deep copy of the data and performs all modifications on the copy.

PARAMETER DESCRIPTION data

A data dict to transmute

TYPE: dict[str, Any]

schema

schema to transmute data

TYPE: dict[str, Any]

root

a root schema type

TYPE: str

RETURNS DESCRIPTION dict[str, Any]

Transmuted data

"},{"location":"changelog/","title":"Changelog","text":"

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog and this project adheres to Semantic Versioning.

"},{"location":"changelog/#v170-2024-10-30","title":"v1.7.0 - 2024-10-30","text":"

Compare with first commit

"},{"location":"changelog/#features","title":"Features","text":""},{"location":"changelog/#bug-fixes","title":"Bug Fixes","text":""},{"location":"configuration/","title":"Configuration","text":""},{"location":"configuration/#ckanexttransmuteschemaname","title":"ckanext.transmute.schema.<NAME>","text":"

Path to the JSON file with definition of the named schema.

"},{"location":"installation/","title":"Installation","text":""},{"location":"installation/#requirements","title":"Requirements","text":"

Compatibility with core CKAN versions:

CKAN version Compatible? 2.9 no 2.10 yes 2.11 yes master yes

Note

It's recommended to install the extension via pip. If you are using GitHub version of the extension, stick to the vX.Y.Z tags to avoid breaking changes. Check the changelog before upgrading the extension.

"},{"location":"installation/#installation_1","title":"Installation","text":"

Install the extension

pip install ckanext-transmute\n

Add transmute to the ckan.plugins setting in your CKAN config file.

"},{"location":"interfaces/","title":"Interfaces","text":""},{"location":"interfaces/#transmute.interfaces.ITransmute","title":"ITransmute","text":"

Main extension point of ckanext-transmute.

"},{"location":"interfaces/#transmute.interfaces.ITransmute.get_transmutation_schemas","title":"get_transmutation_schemas()","text":"

Register definitions of named schemas.

These schemas can be reffered by name in code. In this way you can define static schema and apply in multiple places it to arbitrary data.

Example 
def get_transmutation_schemas(self):\n    person = {\n        \"fields\": {\n            \"age\": {\"validators\": [\"int_validator\"]},\n            \"name\": {\"default\": \"John Doe\"},\n        }\n    }\n\n    schema = {\n        \"root\": \"person\",\n        \"types\": {\"person\": person}\n    }\n\n    return {\"person\": schema}\n
RETURNS DESCRIPTION dict[str, dict[str, Any]]

Mapping with definitions of named schemas.

"},{"location":"interfaces/#transmute.interfaces.ITransmute.get_transmutators","title":"get_transmutators()","text":"

Register custom transmutation functions.

Example 
def get_transmutators(self):\n    return {\n        \"tsm_title_case\": tsm_title_case,\n        \"tsm_is_email\": tsm_is_email\n    }\n
RETURNS DESCRIPTION dict[str, Any]

Mapping with transmutaion functions.

"},{"location":"usage/","title":"Overview","text":"

ckanext-transmute registers an action tsm_transmute to transmute data using the provided conversion scheme. The action doesn't change the original data but creates a new data dict. There are two mandatory arguments: data and schema. data is a data dict you need to transform, and schema contains the rules describing all the transformation steps.

Typical use-case for it is transforming existing data, like this:

{\n  \"title\": \"Test-dataset\",\n  \"email\": \"test@test.ua\",\n  \"metadata_created\": \"\",\n  \"metadata_modified\": \"\",\n  \"metadata_reviewed\": \"\",\n  \"resources\": [\n    {\n      \"title\": \"test-res\",\n      \"extension\": \"xml\",\n      \"web\": \"https://stackoverflow.com/\",\n      \"sub-resources\": [\n        {\n          \"title\": \"sub-res\",\n          \"extension\": \"csv\",\n          \"extra\": \"should-be-removed\"\n        }\n      ]\n    },\n    {\n      \"title\": \"test-res2\",\n      \"extension\": \"csv\",\n      \"web\": \"https://stackoverflow.com/\"\n    }\n  ]\n}\n

into expected data, like this:

{\n    \"name\": \"test-dataset\",\n    \"email\": \"test@test.ua\",\n    \"metadata_created\": datetime.datetime(2022, 2, 3, 15, 54, 26, 359453),\n    \"metadata_modified\": datetime.datetime(2022, 2, 3, 15, 54, 26, 359453),\n    \"metadata_reviewed\": datetime.datetime(2022, 2, 3, 15, 54, 26, 359453),\n    \"attachments\": [\n        {\n            \"name\": \"test-res\",\n            \"format\": \"XML\",\n            \"url\": \"https://stackoverflow.com/\",\n            \"sub-resources\": [{\"name\": \"SUB-RES\", \"format\": \"CSV\"}]\n        },\n        {\n            \"name\": \"test-res2\",\n            \"format\": \"CSV\",\n            \"url\": \"https://stackoverflow.com/\"\n        }\n    ]\n}\n

To achieve this goal, the following schema definition can be used: 

{\n    \"root\": \"Dataset\",\n    \"types\": {\n        \"Dataset\": {\n            \"fields\": {\n                \"title\": {\n                    \"validators\": [\n                        \"tsm_string_only\",\n                        \"tsm_to_lowercase\",\n                        \"tsm_name_validator\",\n                    ],\n                    \"map\": \"name\",\n                },\n                \"resources\": {\n                    \"type\": \"Resource\",\n                    \"multiple\": True,\n                    \"map\": \"attachments\",\n                },\n                \"metadata_created\": {\n                    \"validators\": [\"tsm_isodate\"],\n                    \"default\": \"2022-02-03T15:54:26.359453\",\n                },\n                \"metadata_modified\": {\n                    \"validators\": [\"tsm_isodate\"],\n                    \"default_from\": \"metadata_created\",\n                },\n                \"metadata_reviewed\": {\n                    \"validators\": [\"tsm_isodate\"],\n                    \"replace_from\": \"metadata_modified\",\n                },\n            }\n        },\n        \"Resource\": {\n            \"fields\": {\n                \"title\": {\n                    \"validators\": [\"tsm_string_only\"],\n                    \"map\": \"name\",\n                },\n                \"extension\": {\n                    \"validators\": [\"tsm_string_only\", \"tsm_to_uppercase\"],\n                    \"map\": \"format\",\n                },\n                \"web\": {\n                    \"validators\": [\"tsm_string_only\"],\n                    \"map\": \"url\",\n                },\n                \"sub-resources\": {\n                    \"type\": \"Sub-Resource\",\n                    \"multiple\": True,\n                },\n            },\n        },\n        \"Sub-Resource\": {\n            \"fields\": {\n                \"title\": {\n                    \"validators\": [\"tsm_string_only\", \"tsm_to_uppercase\"],\n                    \"map\": \"name\",\n                },\n                \"extension\": {\n                    \"validators\": [\"tsm_string_only\", \"tsm_to_uppercase\"],\n                    \"map\": \"format\",\n                },\n                \"extra\": {\n                    \"remove\": True,\n                },\n            }\n        },\n    },\n}\n

This is an example of schema with nested types. The root field defines the type of the outer layer of data, while sub-resources field inside the definition of the root type contain type references to Sub-Resource definition.

"},{"location":"usage/schema/","title":"Schema","text":"

Transmutation schema represented by a dictionary that contains descriptions of all data types used for transmutation and the name of the root type.

{\n    \"root\": \"main\",\n    \"types\": {\n        \"main\": {},\n        \"secondary\": {}\n    }\n}\n

The root type is used for the initial transformation. If, during this transformation, some of multi-values fields contain reference to other types defined in schema, these types will be used for further transformation of data.

{\n    \"root\": \"main\",\n    \"types\": {\n        \"main\": {\n            \"drop_unknown_fields\": true,\n            \"fields\": {\n                \"child\": {\"type\": \"secondary\", \"multiple\": true}\n            }\n        },\n        \"secondary\": {\n            \"drop_unknown_fields\": true,\n            \"fields\": {\"name\": {}}\n        }\n    }\n}\n

Note

At the moment, only multivalued fields can be transformed using nested types. In future support for single-valued nested field will be added

"},{"location":"usage/transmutators/","title":"Transmutators","text":"

Transmutators are similar to CKAN validators. They accept the field and modify it. But unlike validators, transmutators work with field and have access to the whole schema definition.

Usually, transmutator is defined as a function with a single argument. This argument always receives the instance of validated field. It's a dataclass with

Transmutator modifies field in place and returns the whole field when job is done.

ckanext-transmute contains a number of transmutators that can be used without additional configuration. And if you need more, you can define a custom transmutator with the ITransmute interface.

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_concat","title":"tsm_concat(field, *strings)","text":"

Concatenate strings to build a new one.

Use $self to point on the current field.value.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

strings

strings to concat with

TYPE: Any DEFAULT: ()

RETURNS DESCRIPTION Field

the same Field with new value

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_get_nested","title":"tsm_get_nested(field, *path)","text":"

Fetches a nested value from a field.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

path

Iterable with path segments

TYPE: str DEFAULT: ()

RAISES DESCRIPTION Invalid

raises if path doesn't exist

RETURNS DESCRIPTION Field

the same Field with new value

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_isodate","title":"tsm_isodate(field)","text":"

Validates datetime string Mutates an iso-like string to datetime object.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RAISES DESCRIPTION Invalid

raises if date format is incorrect

RETURNS DESCRIPTION Field

the same Field with casted value

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_list_mapper","title":"tsm_list_mapper(field, mapping, remove=False)","text":"

Maps values within a list to corresponding values from the provided dictionary.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

mapping

A dictionary representing the mapping of values.

TYPE: dict[Any, Any]

remove

If set to True, removes values from the list if they don't have a corresponding mapping. Defaults to False.

TYPE: bool DEFAULT: False

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_map_value","title":"tsm_map_value(field, test_value, if_same, if_different=SENTINEL)","text":"

Replace value with other value.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

test_value

value that will be compared to field.value

TYPE: Any

if_same

value to use if test_value matches the field.value

TYPE: Any

if_different

value to use if test_value does not matche the field.value. Leave empty to keep original value of the field.

TYPE: Any DEFAULT: SENTINEL

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_mapper","title":"tsm_mapper(field, mapping, default=None)","text":"

Map a value with a new value. The initial value must serve as a key within a mapping dictionary, while the dict value will represent the updated value.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

mapping

A dictionary representing the mapping of values.

TYPE: dict[Any, Any]

default

The default value to be used when the key is not found. If the default value is not provided, the current value will be used as it.

TYPE: Any DEFAULT: None

RETURNS DESCRIPTION Field

the same Field with new value

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_name_validator","title":"tsm_name_validator(field)","text":"

Wrapper over CKAN default name_validator validator.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RAISES DESCRIPTION Invalid

if value is not a valid name

RETURNS DESCRIPTION Field

the same Field object if it's valid

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_stop_on_empty","title":"tsm_stop_on_empty(field)","text":"

Stop transmutation if field is empty.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RETURNS DESCRIPTION Field

the same Field

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_string_only","title":"tsm_string_only(field)","text":"

Validates if field.value is string.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RAISES DESCRIPTION Invalid

raises is the field.value is not string

RETURNS DESCRIPTION Field

the same Field object if it's valid

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_to_lowercase","title":"tsm_to_lowercase(field)","text":"

Casts string value to lowercase.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RETURNS DESCRIPTION Field

Field object with mutated string

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_to_string","title":"tsm_to_string(field)","text":"

Casts field.value to str.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RETURNS DESCRIPTION Field

the same Field with new value

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_to_uppercase","title":"tsm_to_uppercase(field)","text":"

Casts string value to uppercase.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RETURNS DESCRIPTION Field

Field object with mutated string

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_trim_string","title":"tsm_trim_string(field, max_length)","text":"

Trim string lenght.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

max_length

String max length

TYPE: int

RETURNS DESCRIPTION Field

the same Field object if it's valid

TYPE: Field

"},{"location":"usage/transmutators/#transmute.transmutators.tsm_unique_only","title":"tsm_unique_only(field)","text":"

Preserve only unique values from list.

PARAMETER DESCRIPTION field

Field object

TYPE: Field

RETURNS DESCRIPTION Field

the same Field with new value

TYPE: Field

"},{"location":"usage/type/","title":"Type","text":"

Type description contains definition of its fields and a number of additional settings.

{\n    \"root\": \"main\",\n    \"types\": {\n        \"main\": {\n            \"drop_unknown_fields\": true,\n            \"fields\": {\n                \"first\": {},\n                \"second\": {}\n            }\n        }\n    }\n}\n

Every field either refers a different type if it's definded with multiple: true and type: TYPE_NAME, or contains inline definition. Inline fields are used most often and their definition is flexible enough to cover majority of use-cases.

{\n    \"root\": \"main\",\n    \"types\": {\n        \"main\": {\n            \"fields\": {\n                \"inline_field\": {\"default\": 42},\n                \"sub_type\": {\"multiple\": true, \"type\": \"secondary\"}\n            }\n        },\n        \"secondary\": {}\n    }\n}\n

Here's the list of attributes that can be used in the field definition:

Attribute Description map New name of the field validators List of transmutators applied to the field remove Flag that removes field from data when enabled default Default value if field is missing default_from Name of the field used as source of default value value Static value that replaces any existing value of the field replace_from Name of the field used as a source of value validate_missing Flag that applies validation even if data does not contains the field weight Weight that controls order of field processing"}]} \ No newline at end of file diff --git a/sitemap.xml b/sitemap.xml index 09a87fb..fb9a0d6 100644 --- a/sitemap.xml +++ b/sitemap.xml @@ -2,42 +2,42 @@ https://datashades.github.io/ckanext-transmute/ - 2024-11-23 + 2024-11-26 https://datashades.github.io/ckanext-transmute/api/ - 2024-11-23 + 2024-11-26 https://datashades.github.io/ckanext-transmute/changelog/ - 2024-11-23 + 2024-11-26 https://datashades.github.io/ckanext-transmute/configuration/ - 2024-11-23 + 2024-11-26 https://datashades.github.io/ckanext-transmute/installation/ - 2024-11-23 + 2024-11-26 https://datashades.github.io/ckanext-transmute/interfaces/ - 2024-11-23 + 2024-11-26 https://datashades.github.io/ckanext-transmute/usage/ - 2024-11-23 + 2024-11-26 https://datashades.github.io/ckanext-transmute/usage/schema/ - 2024-11-23 + 2024-11-26 https://datashades.github.io/ckanext-transmute/usage/transmutators/ - 2024-11-23 + 2024-11-26 https://datashades.github.io/ckanext-transmute/usage/type/ - 2024-11-23 + 2024-11-26 \ No newline at end of file diff --git a/sitemap.xml.gz b/sitemap.xml.gz index 08346e29b2dc820f7fdd004c481acb229bf31ef7..4d533aceaa955a6ea6b82c58ab1a4a1716238e7d 100644 GIT binary patch delta 253 zcmV?4o0A6rIw2+|KjPC6hN;mDkN#ILj z%lZFLa^(7W9bEQ-k!v{xI6j8FUZcpbKVulHf1aOc6MJ zoXu&hCkE&C0>cn#c+R^P7^65hx(**zk Dby9sm delta 253 zcmV>(oMT>68MtX za{m959JxMT2bXAm1lHRvY8Q07s8EvWfPs4GkfgEPPjAu&rQdDA0OpQv^;Q zXLB0siNU$Oz%a%OXr#?iCIW*yp;(v0PGG)s;1KsRCjK1%Fss{

Concatenate strings to build a new one.

-

Use $self to point on the current field value.

+

Use $self to point on the current field.value.

@@ -1331,7 +1331,7 @@

-

value that will be compared to field value

+

value that will be compared to field.value

@@ -1347,7 +1347,7 @@

-

value to use if test_value matches the field value

+

value to use if test_value matches the field.value

@@ -1363,7 +1363,7 @@

-

value to use if test_value does not matche the field value. +

value to use if test_value does not matche the field.value. Leave empty to keep original value of the field.

@@ -1677,7 +1677,7 @@

-

Validates if field.value is string.

+

Validates if field.value is string.

@@ -1724,7 +1724,7 @@

@@ -1847,7 +1847,7 @@

-

Casts field.value to str.

+

Casts field.value to str.

-

raises is the field.value is not string

+

raises is the field.value is not string