This repository contains a set of JSON schemas that can be used to test JSON Schema containment checkers.
It is meant to be language agnostic and should require only a JSON parser.
The structure of the repository, as well as the general concept, is greatly inspired by the JSON Schema Test Suite.
The tests in this suite are contained in the tests
directory at the root of
this repository. Inside that directory is a subdirectory for each draft or
version of the specification.
Inside each draft directory, there are a number of .json
files and one or more
special subdirectories. The subdirectories contain .json
files meant for a
specific testing purpose, and each .json
file logically groups a set of test
cases together. Often the grouping is by property under test, but not always.
The subdirectories are described in the next section.
Inside each .json
file is a single array containing objects. It's easiest to illustrate the structure of these with an example:
[
{
"id": 1,
"schema1": {
"anyOf": [
{
"enum": [
1,
2,
3
]
},
{
"not": {
"enum": [
1,
2,
3
]
}
}
]
},
"schema2": true,
"tests": {
"s1SubsetEqOfs2": true,
"s2SubsetEqOfs1": true
}
}
]
In short, each object has the following attributes:
- id: Uniquely identifies a set of tests within the same file.
- schema1: The first schema for JSON Schema containment checking.
- schema2: The second schema for JSON Schema containment checking.
- tests: The groundtruth of tests.
- s1SubsetEqOfs2: Whether schema1 is a subset of schema2, or both schemas are equivalent.
- s2SubsetEqOfs1: Whether schema2 is a subset of schema1, or both schemas are equivalent.
There is currently only one subdirectory that may exist within each draft directory. This is:
optional/
: Contains tests that are considered optional.
Python 3 is used for test generation.
// Clone repository with submodules
git clone --recurse-submodules https://github.com/sdbs-uni-p/json-schema-containment-testsuite.git
or
// Clone repository
git clone https://github.com/sdbs-uni-p/json-schema-containment-testsuite.git
// Get the submodules
git submodule update --init --recursive
To regenerate a new version of the test suite as shown in the tests
folder:
-
Delete all folders under
tests
directory in the main repository; -
Run the following commands to generate the test suite based on
JSON Schema Test Suite
:
// Update submodules (JSON-Schema-Test-Suite) to latest commit
git submodule foreach git pull origin master
// Generate tests
python bin/testsuite_generation.py
Validates all tests in the tests
directory.
python bin/test_json.py
There is a demo paper for this test suite, presented at ER 2021: A Test Suite for JSON Schema Containment.
To refer to this test suite in a publication, please cite the demo paper above and/or use this BibTex entry:
@Misc{jsc_test_suite,
author = {Lyes Attouche and Mohamed Amine Baazizi and Dario Colazzo and Yunchen Ding and Luca Escher and Michael Fruth and Giorgio Ghelli and Carlo Sartiani and Stefanie Scherzinger},
title = {A Test Suite for Checking JSON Schema Containment},
note = {\url{https://github.com/sdbs-uni-p/json-schema-containment-testsuite}},
year = 2021
}
This work was partly funded by Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) grant #385808805.