All changes that impact users of this module are documented in this file, in the Common Changelog format with some additional specifications defined in the CONTRIBUTING file. This codebase adheres to Semantic Versioning.
Full changeset and discussions: #1088.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Display engine version in logs
- Fix the number of terms displayed in logs
- Fix Reporter module instantiation
Full changeset and discussions: #1087.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Set default tracking frequency to two per day
Full changeset and discussions: #1086.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Make tracking schedule configurable
- Make publishing schedule configurable
- Add tracking overrun protection
Full changeset and discussions: #1085.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Corrected a missing negation in a Reporter error log
Full changeset and discussions: #1071.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Nest all configurations under the
@opentermsarchive/enginekey; wrap the content ofconfig/production.jsonlike this:{ @opentermsarchive/engine: { < your previous configuration >}} - Breaking: Rename the
apiconfiguration key tocollection-api; update this in yourconfig/production.json - Breaking: Remove default values for
portandbasePathincollection-api; specify these configurations explicitly in yourconfig/production.json - Breaking: Prefix all environment variables with
OTA_ENGINE_; rename variables such asSENDINBLUE_API_KEYtoOTA_ENGINE_SENDINBLUE_API_KEY,SMTP_PASSWORDtoOTA_ENGINE_SMTP_PASSWORD, andGITHUB_TOKENtoOTA_ENGINE_GITHUB_TOKEN
Full changeset and discussions: #1076.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Enable a service to be renamed into several services when exporting a dataset
Full changeset and discussions: #1075.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Exclude non-record files (e.g., README, LICENSE) from dataset
Full changeset and discussions: #1070.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix
ota lintcommand
Full changeset and discussions: #1069.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add label
empty contentto reported issues on GitHub when server returns empty content or when PDF content is unextractable - Add label
invalid selectorto reported issues on GitHub when CSS selector is invalid
- Consider extraction errors as expected operational errors instead of crashing the engine
- Do not crash when source documents links point to invalid URLs
- Take into account the
--typesCLI option - Fix counts of terms when they are specified via the CLI option
--types - Fix display issues for service ID and terms type associated with errors
Full changeset and discussions: #1068.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix validation of modified declarations when declaration history is modified; bug introduced in v1.1.2
Full changeset and discussions: #1066.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix validation of modified declarations when new services were added
- Fix validation of modified declarations when filters were modified
Full changeset and discussions: #1065.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Ensure to reopen associated issue if terms become inaccessible again
Full changeset and discussions: #1063.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Introduce
pluginErrorerror type to isolate plugin errors from the engine and increase robustness
Full changeset and discussions: #1061.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
No code changes were made in this release, it simply signifies that the public API can now be considered stable.
Full changeset and discussions: #1056.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix support of service names with special character in the validation CLI
Full changeset and discussions: #1054.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Minimize required privileges associated to the GitHub token to create issues in the declarations repository
Full changeset and discussions: #1053.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Log a warning in case log emails cannot be sent because of a missing config
Full changeset and discussions: #1052.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Base the loading of the Notifier module on the presence of required configuration, not on the environment; only define the
SENDINBLUE_API_KEYenvironment variable if you want the Notifier to be loaded, avoid relying onNODE_ENV=production
Full changeset and discussions: #1051.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Consider EAI_AGAIN errors as a legitimate part of the tracking lifecycle rather than causing the engine to crash
Full changeset and discussions: #1050.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix engine hang with non-existent service ID
Full changeset and discussions: #1045.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix support of special character in service names
Full changeset and discussions: #1037.
Development of this release was made on a volunteer basis by @Ndpnt.
- Improve performance for large-scale MongoDB databases
Full changeset and discussions: #1033.
Development of this release was supported by the NGI0 Entrust Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Expose versions data through the collection API (#1003). When using Git as storage for versions, this API relies on the assumption that the commit date matches the author date, introduced in the engine in June 2022 (#875). If your collection was created before this date, inconsistencies in the API results may arise. You can verify if your version history includes commits with commit dates differing from author dates by executing the following command at the root of your versions repository:
git log --format="%H %ad %cI" --date=iso-strict | awk '{if ($2 != $3) print "Author date", $2, "and commit date", $3, "mismatch for commit", $1 }'. You can correct the history with the command:git rebase --committer-date-is-author-date $(git rev-list --max-parents=0 HEAD). Since the entire history will be rewritten, a force push may be required for distributed repositories
- Provide a succinct JSON-formatted error message as response in API errors
Full changeset and discussions: #1032.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Update ESLint to use the stylistic plugin following deprecation of formatting rules, correcting a randomly-appearing
Configuration for rule "lines-between-class-members" is invaliderror
Full changeset and discussions: #1031.
Development of this release was supported by the NGI0 Entrust Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Add
termsattribute to/servicesAPI response, containing declared term types for each service
Full changeset and discussions: #1026.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Ensure
Reporterconfiguration is defined before instantiating it - Ensure to add links to
versionsandsnapshotsin issues only if repositories are defined in configuration
- No longer define default configuration for
Reportermodule
Full changeset and discussions: #1025.
Development of this release was supported by GitHub Social Impact through its DPG Open Source Community Manager Program.
- Auto create labels for declarations repository on GitHub
- Assign a label based on the error type to issues managed by the engine
- Breaking: Revise the formatting of reported issues, with notable adjustments to the title, transitioning from
Fix <service_id> - <terms_type>to<service_id> ‧ <terms_type> ‧ not tracked anymore - Breaking: Rename
trackermodule intoreporter; update your configuration file by renamingtrackerintoreporter - Breaking: Change configuration for
reportermodule; update yourreporterconfiguration accordingly
- Breaking: No longer assign the label
bot-reportto issues managed by the engine
Full changeset and discussions: #1024.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Upgrade dependencies patches and ensure compatibility with Node v20
Full changeset and discussions: #1021.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Update
opentermsarchive/terms-typesdependency; update your declarations following the instructions in the changelog for validating declarations
Full changeset and discussions: #1020.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix validation for removed terms types
Full changeset and discussions: #1015.
Development of this release was supported by the NGI0 Entrust Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Embed Swagger UI for graphical user interface documentation of the API; access it on
/docs
- Breaking: Change path from
/services/:serviceIdto/service/:serviceIdfor direct access to aserviceresource; update paths accordingly in your own codebase - Breaking: Change path from
/specsto/docsfor OpenAPI specification; update paths accordingly in your own codebase and set the HTTP headerAccept: application/json
Full changeset and discussions: #1013.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Fix issue creation on GitHub tracker (#1012)
Full changeset and discussions: #1011.
Development of this release was supported by the NGI0 Entrust Fund, a fund established by NLnet with financial support from the European Commission's Next Generation Internet programme, under the aegis of DG CNECT under grant agreement N°101069594.
- Add collection metadata API (#1003)
- Add
ota servecommand to CLI to start the server and expose the API
- Include collection name in dataset name
- Clean up README
Full changeset and discussions: #996, #999, #1000.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Rename CLI option
--terms-typesto--typesin API; simply rename accordingly in your own codebase - Breaking: Rename CLI option
--refilter-only,-rto--extract-only,-ein API; simply rename accordingly in your own codebase - Breaking: Rename class
PageDeclarationtoSourceDocumentand its atributenoiseSelectorstoinsignificantContentSelectorsin API; simply rename accordingly in your own codebase - Breaking: Rename function and its parameters
filter({ content, mimeType, pageDeclaration })toextract(sourceDocument)in API;contentandmimeTypeare embeddedsourceDocumentattributes; rename accordingly in your own codebase and setcontentandmimeTypein thesourceDocumentpassed as a parameter to the function - Breaking: Rephrase commit messages in Git storage:
Start trackingis changed toFirst record of,RefiltertoApply technical or declaration upgrade onandUpdatetoRecord new changes of; existing data will still be loaded, but new commits will use these new messages, if you have scripts that parse commit messages directly, update them accordingly - Breaking: Rename document attribute
isRefiltertoisExtractOnlyin MongoDB storage; existing data will still be loaded, but new entries will use this new attribute, if you have scripts that query the Mongo database directly, update them accordingly - Make vocabulary consistent throughout the codebase (#971)
- Breaking: Remove
npm run extractcommand; usenpm run start -- --extract-onlyinstead
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Change URL to engine repo from
ambanum/OpenTermsArchivetoOpenTermsArchive/engine - Replace documentation with links to the documentation website to avoid redundancy
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Split validation and linting of declaration files in 2 commands.
ota validatedoes not test the linting of files anymore. - Breaking:
ota lintdoes not fix files by default.ota lint --fixmust be used for that.
Full changeset and discussions: #981.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Specification of changelog format
- Specification of changelog format is now in CONTRIBUTING
- Improve documentation for CLI commands
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Replace behaviour of
ota track --schedule. It now tracks the changes only on scheduled hours. It was before running also on launch.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Trigger a release event on CI to deploy documentation website
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Remove obsolete ansible deployment recipes as it was extracted in a dedicated repository. Look at the README to know how to deploy the engine.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Replace embedded terms types list with the one defined in the dedicated repository
@opentermsarchive/terms-types.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add dataset command to CLI; this command can be discovered in the documentation and by running
ota dataset help
Full changeset and discussions: #959.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Improved reliability and expanded coverage of email protection global filter
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Add missing configuration for production
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Breaking: Unify all CLI commands as subcommands of one single
otacommand and rename some options; the new CLI can be discovered in the documentation and by runningota help(#978)
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Ensure paths for given Git storage configuration are relative to current working directory instead of engine module folder
Full changeset and discussions: #979.
- Add missing configuration for production
Full changeset and discussions: #976.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Load both the configurations defined in this module and the configurations defined in the module which use it as dependency
Full changeset and discussions: #977.
- Add missing file required for CLI in the packaging process
Full changeset and discussions: #971.
- Rewrite documentation to bring it up to date with current behaviour
Full changeset and discussions: #957.
Development of this release was supported by the French Ministry for Foreign Affairs through its ministerial State Startups incubator under the aegis of the Ambassador for Digital Affairs.
- Publish package on NPM under name
@opentermsarchive/engine - Export
filter,pageDeclarationandfetchin NPM module - Add changelog