Using DirectoryTemplates to save/store json structures in directory trees

[pljakobs]

my (inherited) application uses a rather large global config struct that is only growing. This struct is represented on storage as well as on the API level by a json object, there's two sets of code to construct the object (one for storage, one for the API) and two sets to deserialize it again.

My idea to save both code and, more importantly, RAM, would be to map the json object to a directory structure on storage, files matching leaf objects in the json object, directories representing objects containing other objects.

Now, in digging into Sming classes and samples, I was intrigued by the DirectoryTemplate and the ability to use a small template to list a directory as json as seen in Basic_Storage.

I was wondering: how difficult could it be to build on this templating to at least be able to stream a directory structure as mentioned above as json? I feel that "only" two things are missing: the ability to read file content from the template (which is certainly possible but not from the DirectoryTemplate class, right?) and the ability to recurse into directories, reading the whole tree in one go.
Also, there would need to be a way to differentiate directories representing an array from directories representing an object. In my current code, I'm doing that by going through all the children, if they all have numeric names, I deem it an array. Probably not the best criteria, especially if the numeric names are non-contiguous.

I figure that the other way round - taking in a json file and turning it into a directory is a bit more difficult, as, while the StreamTemplate engine could be used to dissect the json, there would still have to be a bit of code specific to storing json.

But I wonder: given IFS and it's copy on write feature, this might make a pretty descent way of storing large configurations. Since directories can be directly accessed, full as well as partial reads / updates / adds should be easy, element order doesn't matter in json and element hierarchy would be reflected 1:1 by directories. To clearly mark arrays from objects of objects, I figure a user attr could be used on the containing directory?

From my application's point of view, I believe this might actually allow me to completely remove ArduinoJSON from it, as any serialization / deseralization / object value read could be done through this. And since the app mostly uses json for configuration, the performance impact of going through flash should not be an issue. Maybe I could even use Stream templates to read values from the api / websocket messages which are json.

I will say that I'm nowhere near understanding the stream classes well enough to implement this, but I would appreciate feedback, if this might be a good idea and even a valuable add-on to Sming as well as maybe pointers in the right direction?

regards

pj

[mikee47]

Sounds like an interesting idea! There is going to be a trade-off since too many files and the filing system overhead becomes excessive. In general terms, partitioning the data structure into a few separate JSON files might work best. Merging those together doesn't necessarily require them to be hierarchical.

Can you provide an example configuration (suitable sanitised, of course)? It would be helpful to aid discussion; we should consider your specific use case before attempting to generalise a solution.

NB. Differentiating between array and object nodes is kind of implicit since each leaf (JSON file) would either contain [ ... ] or { ... }.

[pljakobs]

this is the current config json. As said, my primary goal is to minimize RAM usage:

{"network":{"connection":{"dhcp":true,"ip":"0.0.0.0","netmask":"0.0.0.0","gateway":"0.0.0.0"},"ap":{"secured":false,"password":"rgbwwctrl","ssid":""},"mqtt":{"enabled":false,"server":"mqtt.local","port":1883,"username":"","password":"","topic_base":"home/"}},"color":{"outputmode":0,"startup_color":"last","hsv":{"model":0,"red":0,"yellow":0,"green":0,"cyan":0,"blue":0,"magenta":0},"brightness":{"red":100,"green":100,"blue":100,"ww":100,"cw":100},"colortemp":{"ww":2700,"cw":6000}},"security":{"api_secured":false},"ota":{"url":"https://lightinator.de/version.json"},"sync":{"clock_master_enabled":false,"clock_master_interval":30,"clock_slave_enabled":false,"clock_slave_topic":"home/led1/clock","cmd_master_enabled":false,"cmd_slave_enabled":false,"cmd_slave_topic":"home/led1/command","color_master_enabled":false,"color_master_interval_ms":0,"color_slave_enabled":false,"color_slave_topic":"home/led1/color"},"events":{"color_interval_ms":500,"color_mininterval_ms":500,"server_enabled":true,"transfin_interval_ms":1000},"general":{"device_name":"","pin_config":"13,12,14,5,4","buttons_config":"","buttons_debounce_ms":50,"current_pin_config_name":"mrpj","pin_config_url":"https://raw.githubusercontent.com/pljakobs/esp_rgb_webapp2/devel/public/config/pinconfig.json","channels":[{"pin":13,"name":"red"},{"pin":12,"name":"green"},{"pin":14,"name":"blue"},{"pin":5,"name":"warmwhite"},{"pin":4,"name":"coldwhite"}],"supported_color_models":["RGB","RGBW","RGBWW","RAW"]}}

(sorry for not having a pretty version)

the raw config struct is about 500-600 byte in RAM but the two string arrays add considerably.
From what I looked at with LittleFS, I believe the point where number of files becomes an issue (especially on a fs that is primarily read), we're talking about 10000s of files. This should not be more than a few 100 max - of which most are only ever read.
There are three access cases for both read and write:

• full structure read / write - primarily when the config is updated through the API or backed up.
• single element read / write - whenever the application reads or writes a single element - that might happen more often and might be a bit more time critical - I would need to look into this, but I don't really think so. Worst case, a few values could be kept in variables
• single object read / write - to allow loops, I would expect an array to be read as one element or maybe used via an enumerator to iterate over the elements.

The way this structure is held now is a global struct. I find that most un-elegant.

On partitioning the file: my thought is to get rid of the whole "load the data into ram" idea - a class that would take a json document and translate it into an on-strage representation and would, reversely, take a json path and provide the value or object that path points to seems like an interesting alternative as it would avoid the whole serialization / deserialization and RAM storage.
Also, as said, that happens twice in two different places of the code, I don't know why and I never liked it.

[pljakobs]

(btw, just noticed I said IFS and copy on write - of course I meant LittelFS)

[mikee47]

My idea to save both code and, more importantly, RAM, would be to map the json object to a directory structure on storage, files matching leaf objects in the json object, directories representing objects containing other objects.

So if I understand correctly, we'd have something like this:

config/
    network/
        connection.json
            {
                "dhcp": true,
                "ip": "0.0.0.0",
                "netmask": "0.0.0.0",
                "gateway": "0.0.0.0"
            }
        ap.json
            {
                "secured": false,
                "password": "rgbwwctrl",
                "ssid": ""
            }
        mqtt.json
            {
                "enabled": false,
                "server": "mqtt.local",
                "port": 1883,
                "username": "",
                "password": "",
                "topic_base": "home/"
            }
    color.json
        {
            "outputmode": 0,
            "startup_color": "last",
        }
    color/
        hsv.json
            {
                "model": 0,
                "red": 0,
                "yellow": 0,
                "green": 0,
                "cyan": 0,
                "blue": 0,
                "magenta": 0
            }
        brightness.json
            {
                "red": 100,
                "green": 100,
                "blue": 100,
                "ww": 100,
                "cw": 100
            }
        colortemp.json
            {
                "ww": 2700,
                "cw": 6000
            }
    security.json
        {
            "api_secured": false
        }
    ota.json
        {
            "url": "https://lightinator.de/version.json"
        }
    sync.json
        {
            "clock_master_enabled": false,
            "clock_master_interval": 30,
            "clock_slave_enabled": false,
            "clock_slave_topic": "home/led1/clock",
            "cmd_master_enabled": false,
            "cmd_slave_enabled": false,
            "cmd_slave_topic": "home/led1/command",
            "color_master_enabled": false,
            "color_master_interval_ms": 0,
            "color_slave_enabled": false,
            "color_slave_topic": "home/led1/color"
        }
    events.json
        {
            "color_interval_ms": 500,
            "color_mininterval_ms": 500,
            "server_enabled": true,
            "transfin_interval_ms": 1000
        }
    general.json
        {
            "device_name": "",
            "pin_config": "13,12,14,5,4",
            "buttons_config": "",
            "buttons_debounce_ms": 50,
            "current_pin_config_name": "mrpj",
            "pin_config_url": "https://raw.githubusercontent.com/pljakobs/esp_rgb_webapp2/devel/public/config/pinconfig.json",
        }
    general/
        channels.json
            [
                {
                    "pin": 13,
                    "name": "red"
                },
                {
                    "pin": 12,
                    "name": "green"
                },
                {
                    "pin": 14,
                    "name": "blue"
                },
                {
                    "pin": 5,
                    "name": "warmwhite"
                },
                {
                    "pin": 4,
                    "name": "coldwhite"
                }
            ]
    supported_color_models.json
        [
            "RGB",
            "RGBW",
            "RGBWW",
            "RAW"
        ]

[mikee47]

Without file content:

config/
    color.json
    security.json
    ota.json
    sync.json
    events.json
    general.json
    supported_color_models.json
    network/
        connection.json
        ap.json
        mqtt.json
    color/
        hsv.json
        brightness.json
        colortemp.json
    general/
        channels.json

[mikee47]

Maybe just grouping settings more sensibly would be simpler:

config/
    network.json
        {
            "network": {
                "connection": {
                    "dhcp": true,
                    "ip": "0.0.0.0",
                    "netmask": "0.0.0.0",
                    "gateway": "0.0.0.0"
                },
                "ap": {
                    "secured": false,
                    "password": "rgbwwctrl",
                    "ssid": ""
                },
                "mqtt": {
                    "enabled": false,
                    "server": "mqtt.local",
                    "port": 1883,
                    "username": "",
                    "password": "",
                    "topic_base": "home/"
                }
            },
            "security": {
                "api_secured": false
            },
            "ota": {
                "url": "https://lightinator.de/version.json"
            }
        }

etc.

[mikee47]

My idea to save both code and, more importantly, RAM, would be to map the json object to a directory structure on storage, files matching leaf objects in the json object, directories representing objects containing other objects.
Now, in digging into Sming classes and samples, I was intrigued by the DirectoryTemplate and the ability to use a small template to list a directory as json as seen in Basic_Storage Basic_IFS.

Templating works well here since each entry is for a simple Stat structure. That can't really be extended to more complex structures as we're considering here.

I was wondering: how difficult could it be to build on this templating to at least be able to stream a directory structure as mentioned above as json? I feel that "only" two things are missing: the ability to read file content from the template (which is certainly possible but not from the DirectoryTemplate class, right?) and the ability to recurse into directories, reading the whole tree in one go.

A flat directory structure is fairly easy to handle like this since it requires only one or two file handles to be open at once. As soon as you start nesting directories it becomes much more complex.

A more general problem here is synchronisation of settings if related values are distributed across multiple files.

From my application's point of view, I believe this might actually allow me to completely remove ArduinoJSON from it, as any serialization / deseralization / object value read could be done through this. And since the app mostly uses json for configuration, the performance impact of going through flash should not be an issue. Maybe I could even use Stream templates to read values from the api / websocket messages which are json.

I think this would end up being a lot more complicated than simply breaking the configuration into logical chunks. Each chunk would be stored in a separate file and would be accessed using ArduinoJson as usual.

Serialising those files to a web client can be done using a custom stream class which just serves up each file in turn. The MultipartStream class does this to encode HTTP multipart content.

A more optimised example can be found in the UPnP library, which contains a DescriptionStream class to produce XML-encoded output. The actual encoding is done in sections using the RapidXML library which keeps the RAM footprint low.

It also has the DescriptionParser class which does the reverse, allowing large .xml configuration files to be parsed in sections. Parsing is tricker than encoding because we need to detect when we have a complete chunk ready for de-serialisation.

I think a more generalised class would be useful to allow chunked serialisation and de-serialisation of large JSON files, so happy to work on this with you?

[mikee47]

Dealing with string paths and variants is flexible, but error-prone.
Defining the application's database structure using C++ classes enables compile-time type checking and simplified coding.

Let's defined a Group class to support this:

namespace ConfigDB
{
    /**
     * @brief Access a group of simple key/value pairs within a store
     */
    class Group
    {
    public:
        Group(Config& db, const String& path, Mode mode)
        {
            // e.g. given "network.connection" split up (probably usign CStringArray)
            String storeName = ...; // "network"
            path = ...; // "connection"
            store = db.open(storeName, mode);
        }

        bool setValue(const String& key, const Value& value)
        {
            return store->setValue(path + key, value);
        }

        Value getValue(const String& key)
        {
            return store->getValue(path + key);
        }

    private:
        StorePtr store;
        String path;
    };

}

We can then use this in a set of concrete classes which reflect the structure of our specific database.
In practice, this code would be generated (e.g. with python) from a database schema.

namespace MyConfig
{
    namespace Network {
        class Connection: public ConfigDB::Group
        {
            Connection(ConfigDB::Config& db): Group(db, "network.connection")
            {
            }

            bool getDhcp()
            {
                return Group::getValue("dhcp");
            }

            void setDhcp(bool value)
            {
                Group::setValue("dhcp", value);
            }

            IpAddress getIp()
            {
                return IpAddress(Group::getValue("ip").as<String>());
            }

            void setIp(const IpAddress& ip)
            {
                Group::setValue("ip", String(ip));
            }
        };

        class Ap: public ConfigDB::Group
        {
            Ap(ConfigDB::Config& db): Group(db, "network.ap")
            {
            }

            // ...
        };

        // ... etc
    }

    namespace Color
    {
        // ...
    }

    namespace Sync
    {
        // ...
    }

    // ... etc.
} // namespace MyConfig

And we'd use it like this:

void update()
{
    ConfigDB::Config db(...);
    MyConfig::Network::Connection connection(db);
    IpAddress ip = connection.getIp();
    connection.setDhcp(false);
    connection.setIp("1.2.3.4");
    connection.commit();
}

[mikee47]

The UPnP library uses code generation in this way. Since the entire standard uses XML I could use XSL transforms to do this. We'd probably have to invent something here, probably using JSON.

[pljakobs]

So what you're suggesting is to have a json prototype and, possibly during the build process, generate it's implementation as a c++ class?
That would be close to magic!

[mikee47]

We already use JsonSchema to validate hardware configs (Sming/Components/Storage/schema.json), IFS build scripts (Sming/Components/IFS/tools/fsbuild/schema.json) and for USB (Sming/Libraries/USB/schema.json). This allows config files to be validated during pre-build.

If you're not aware, we can also use this schema for auto-completion in VsCode (and probably other IDEs):

• Go to a sample project, e.g. Basic_Storage
• Run make ide-vscode to generate appropriate vscode project files
• Open vscode (or vscodium) and open the samples/Basic_Storage/sming.code-workspace (which matches .hw files with the storage schema).
• Open basic_storage.hw: note that VSCode recognises this as JSON
• Type " somewhere and auto-completion will show options from the schema

We can do a similar thing for ConfigDB, so for .cdb or whatever is associated with the schema. Our schema would define a much simpler and more specific syntax for the actual database configuration.

Schema can be mostly generated automatically from existing datasets. At https://app.quicktype.io/, for example, you can paste in a JSON data file on the left and select 'Json Schema' as the output format. I've found this provides a good starting point from which to refine the schema.

[mikee47]

https://github.com/saasquatch/json-schema-inferrer Has an online demo which generates much simpler and more configurable schema.

[mikee47]

I've started a library based on this discussion at https://github.com/mikee47/ConfigDB/tree/develop. There's a working sample based on your dataset so have a play and we can perhaps work towards something useful?

[pljakobs]

you are amazing!
will have a look at it later.