Generate JSON schema's from spec schema to validate values in sidecars. #2020
Conversation
Codecov ReportAttention: Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## master #2020 +/- ##
==========================================
+ Coverage 85.75% 87.10% +1.34%
==========================================
Files 91 133 +42
Lines 3785 6411 +2626
Branches 1218 1545 +327
==========================================
+ Hits 3246 5584 +2338
- Misses 453 737 +284
- Partials 86 90 +4 ☔ View full report in Codecov by Sentry. |
|
This isn't using AJV to validate column value types yet. Currently all values read in from a tsv are treated as strings. The schema specifies the type of some columns as numbers. For now the code looks up the format associated with the type and runs that regex on the values. I'll need to attempt coercion of values if we want type checking there to be similar to json files. |
| this.dataset_description = description | ||
| this.files = [] | ||
| this.tree = {} | ||
| this.ignored = [] | ||
| this.modalities = [] | ||
| this.ajv = new Ajv({strictSchema: false}) | ||
| // @ts-expect-error | ||
| this.ajv.compile = memoize(this.ajv.compile) |
There was a problem hiding this comment.
Haven't been able to figure out how to make these types mesh. Could store it as another entry in the dataset context/.
There was a problem hiding this comment.
Maybe AJV itself should be a private field and compile is just a method on the context?
I think the regex checking is probably going to be faster than coercion and validation, and it gets us where we want to be. |
Co-authored-by: Chris Markiewicz <effigies@gmail.com>
| @@ -47,6 +47,10 @@ export const filenameIssues: IssueDefinitionRecord = { | |||
| severity: "warning", | |||
| reason: "A data files JSON sidecar is missing a key listed as recommended.", | |||
| }, | |||
| JSON_SCHEMA_VALIDATION_ERROR: { | |||
There was a problem hiding this comment.
We should pull this code from schema.rules.errors (and make all other errors there available. Internally represent with code as the key and rest of object as contents. add addSchemaIssue to src/issues/schemaIssues.ts or something.
| const validate = context.dataset.ajv.compile(schema.objects.metadata[keyName]) | ||
| const result = validate(context.sidecar[keyName]) | ||
| if (result === false) { | ||
| const evidenceBase = `Failed for this file.key: ${originFileKey} Schema path: ${schemaPath}` |
There was a problem hiding this comment.
Errors could take schema path as a seperate argument and print the entire rule. https://deno.land/x/json2yaml@v1.0.1
Co-authored-by: Nell Hardcastle <nell@dev-nell.com>
…il out of json validation earlier if key not in sidecar
…ds-validator into schema/json_array_type_checks
… schema/json_array_type_checks
… schema/json_array_type_checks
|
|
|
Actually a schema bug. Should be: diff --git a/src/schema/rules/sidecars/meg.yaml b/src/schema/rules/sidecars/meg.yaml
index 33d851f6..5391c829 100644
--- a/src/schema/rules/sidecars/meg.yaml
+++ b/src/schema/rules/sidecars/meg.yaml
@@ -205,7 +205,7 @@ MEGCoordsystemDigitizedHeadPoints:
- datatype == "meg"
- suffix == "coordsystem"
fields:
- DigitizedHeadPoints: optional
+ DigitizedHeadPoints__coordsystem: optional
DigitizedHeadPointsCoordinateSystem: optional
DigitizedHeadPointsCoordinateUnits: optional
DigitizedHeadPointsCoordinateSystemDescription: |
|
Should be able to test the above patch with: |
effigies
force-pushed
the
schema/json_array_type_checks
branch
from
675c62d to
6b61737
Compare
No description provided.