Boolean type

[afolarin]

Quick question:

An Issue arose with merging the pyspark dataframes. It is related to the boolean/string issue we faced while reading from the schema. The reason for the inconsistency was that in some schema, the datatype is defined as:
'type': 'boolean'
While for some others, it is defined as:

'type': {
    'type': 'boolean'
}

@blootsvoets any reason for this and would it make sense to try validating for a consistent form?

cc/ @thepushkarp @Hsankesara

[blootsvoets]

Indeed simple types should not be nested like that. It’s only used for complex types: lists, maps, enums and records. If the schema compiles (does it?) it is hard to check this programmatically, because in the compiled schema you cannot distinguish these notations.

Feel free to add an entry to the readme and/or pyspark Github actions test