Datasources - Part 2: main extension + existStrict + builder #3102

rolnico · 2024-07-12T07:52:58Z

Please check if the PR fulfills these requirements

The commit message follows our guidelines
Tests for the changes have been added (for bug fixes / features)
Docs have been added / updated (for bug fixes / features)

Does this PR already have an issue describing the problem?

What kind of change does this PR introduce?
Feature + bug fix

What is the current behavior?

What is the new behavior (if this is a feature change)?
Features:

The parameter dataExtension corresponds to the extension of the files the users want to consider in their datasources. For example, if dataExtension == "xiidm", only files ending with ".xiidm" will be considered. The value can be anything. If the parameter is empty, it won't be considered in the file listing.
"Double" extensions (.iidm.xml, .iidm.json, .iidm.bin) are not anymore considered by the importers.
A DataSourceBuilder is now provided in addition to usual datasources creation methods.

Bug fixes:

When multiple files with the same base name are located next to each other, the correct importer will now be used.
When providing a filename to a DirectoryDataSource method (exists, newOutputStream, etc.), the datasource compression format extension is added to the filename only if the filename doesn't have one already.

Does this PR introduce a breaking change or deprecate an API?

Yes
No

If yes, please check if the following requirements are fulfilled

The Breaking Change or Deprecated label has been added
The migration steps are described in the following section

What changes might users need to make in their application due to this PR? (migration steps)

Some datasource constructors have changed or have been deleted. Change accordingly:

Old constructor	New constructor
`Bzip2DirectoryDataSource(Path directory, String baseName, DataSourceObserver observer)` or `Bzip2DirectoryDataSource(Path directory, String baseName)`	`Bzip2DirectoryDataSource(Path directory, String baseName, String dataExtension, DataSourceObserver observer)`
`GzDirectoryDataSource(Path directory, String baseName, DataSourceObserver observer)` or `GzDirectoryDataSource(Path directory, String baseName)`	`GzDirectoryDataSource(Path directory, String baseName, String dataExtension, DataSourceObserver observer)`
`XZDirectoryDataSource(Path directory, String baseName, DataSourceObserver observer)` or `XZDirectoryDataSource(Path directory, String baseName)`	`XZDirectoryDataSource(Path directory, String baseName, String dataExtension, DataSourceObserver observer)`
`ZstdDirectoryDataSource(Path directory, String baseName, DataSourceObserver observer)` or `ZstdDirectoryDataSource(Path directory, String baseName)`	`ZstdDirectoryDataSource(Path directory, String baseName, String dataExtension, DataSourceObserver observer)`
`XZDirectoryDataSource(Path directory, String baseName, DataSourceObserver observer)` or `XzDirectoryDataSource(Path directory, String baseName)`	`XzDirectoryDataSource(Path directory, String baseName, String dataExtension, DataSourceObserver observer)`
`ZipArchiveDataSource(Path directory, String zipFileName, String baseName, DataSourceObserver observer)`	`ZipArchiveDataSource(Path directory, String zipFileName, String baseName, String dataExtension, DataSourceObserver observer)`
`ZipArchiveDataSource(Path directory, String zipFileName, String baseName)`	`ZipArchiveDataSource(Path directory, String zipFileName, String baseName, String dataExtension)`

"Double" extensions (.iidm.xml, .iidm.json, .iidm.bin) are not anymore considered by the importers. If you used them, please use those instead:

Old double extension	Extension to use
`.iidm.xml`	`.xiidm`
`.iidm.json`	`.jiidm`
`.iidm.bin`	`.biidm`

Previously, when providing a filename to a datasource method (exists, newOutputStream, etc.), the datasource compression format extension was always added to the filename. Now it is added to the filename only if the filename doesn't have one already.

Other information:

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

…ataSource Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

…asources_2_main_extension # Conflicts: # commons/src/main/java/com/powsybl/commons/datasource/ReadOnlyDataSource.java

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

jonenst · 2024-07-15T15:48:55Z

commons/src/main/java/com/powsybl/commons/datasource/DirectoryDataSource.java

@@ -59,7 +62,8 @@ protected OutputStream getCompressedOutputStream(OutputStream os) throws IOExcep

 private Path getPath(String fileName) {
 Objects.requireNonNull(fileName);
- return directory.resolve(fileName + getCompressionExt());
+ FileInformation fileInformation = new FileInformation(fileName, false);
+ return directory.resolve(fileInformation.getCompressionFormat() == null ? fileName + getCompressionExtension() : fileName);


mention this fix/change or put in a separate PR ?

This changes compressed directory data source to allow newinputstream newoutputstream and exists to work with "foo.xiidm" when the file is "foo.xiidm.gz"

so it introduces the concept of "filenameOrUncompressedFilename" ?

added info about this in the PR comment + in the breaking change section

jonenst · 2024-07-15T16:16:17Z

iidm/iidm-serde/src/test/java/com/powsybl/iidm/serde/ConversionTest.java

+ }
+
+ @Test
+ void testConversionZip() throws IOException {


very nice that you added a test !

jonenst · 2024-07-15T16:26:12Z

since everything can be implemented only with getBaseExtension ( see #3085 ) , do we still want to add an API existsStrict ? Maybe yes but i'm not sure

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

rolnico · 2024-07-16T15:00:50Z

since everything can be implemented only with getBaseExtension ( see #3085 ) , do we still want to add an API existsStrict ? Maybe yes but i'm not sure

I find it better to do the check on the datasource side and not on the importer side. However, after discussion with @flo-dup, I changed existsStrict(String suffix, String ext) to isMainExtension(String ext) and fixed the uses. This method is more understandable and still keeps the logic on the datasource side.

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

# Conflicts: # commons/src/main/java/com/powsybl/commons/datasource/AbstractArchiveDataSource.java # commons/src/main/java/com/powsybl/commons/datasource/AbstractFileSystemDataSource.java # commons/src/main/java/com/powsybl/commons/datasource/Bzip2DirectoryDataSource.java # commons/src/main/java/com/powsybl/commons/datasource/DataSourceUtil.java # commons/src/main/java/com/powsybl/commons/datasource/DirectoryDataSource.java # commons/src/main/java/com/powsybl/commons/datasource/GenericReadOnlyDataSource.java # commons/src/main/java/com/powsybl/commons/datasource/GzDirectoryDataSource.java # commons/src/main/java/com/powsybl/commons/datasource/ReadOnlyDataSource.java # commons/src/main/java/com/powsybl/commons/datasource/XZDirectoryDataSource.java # commons/src/main/java/com/powsybl/commons/datasource/ZipArchiveDataSource.java # commons/src/main/java/com/powsybl/commons/datasource/ZstdDirectoryDataSource.java # commons/src/test/java/com/powsybl/commons/datasource/AbstractArchiveDataSourceTest.java # commons/src/test/java/com/powsybl/commons/datasource/AbstractFileSystemDataSourceTest.java # commons/src/test/java/com/powsybl/commons/datasource/Bzip2DirectoryDataSourceTest.java # commons/src/test/java/com/powsybl/commons/datasource/DirectoryDataSourceTest.java # commons/src/test/java/com/powsybl/commons/datasource/GzDirectoryDataSourceTest.java # commons/src/test/java/com/powsybl/commons/datasource/MultipleReadOnlyDataSourceTest.java # commons/src/test/java/com/powsybl/commons/datasource/XZDirectoryDataSourceTest.java # commons/src/test/java/com/powsybl/commons/datasource/ZipArchiveDataSourceTest.java # commons/src/test/java/com/powsybl/commons/datasource/ZstdDirectoryDataSourceTest.java

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

commons/src/main/java/com/powsybl/commons/datasource/FileInformation.java

commons/src/main/java/com/powsybl/commons/datasource/ReadOnlyDataSource.java

commons/src/main/java/com/powsybl/commons/datasource/FileInformation.java

commons/src/main/java/com/powsybl/commons/datasource/ReadOnlyDataSource.java

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

cgmes/cgmes-model/src/main/java/com/powsybl/cgmes/model/CgmesOnDataSource.java

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

sonarcloud · 2024-07-31T15:09:11Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
92.4% Coverage on New Code
0.2% Duplication on New Code

See analysis details on SonarCloud

jonenst · 2024-08-02T07:56:07Z

cgmes/cgmes-model/src/main/java/com/powsybl/cgmes/model/CgmesOnDataSource.java

+ if (dataSource.getDataExtension() == null || dataSource.getDataExtension().isEmpty()) {
+ return false;
+ } else if (EXTENSIONS.equals(dataSource.getDataExtension())) {
+ try (InputStream is = dataSource.newInputStream(null, EXTENSIONS)) {


Does this do an NPE if the user specified a data extension but the file doesn't actually exist ?

It would throw an exception but not a NPE. It will be fixed in #3114

rolnico added 6 commits July 10, 2024 17:37

reorganize datasource classes + add tests

e5c4154

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

fix issues + make a constructor protected

f7be34e

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

add mainExtension

7231deb

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

add mainExtension parameter to datasource

f785eab

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

fix issues

b4134bd

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

fix issues

ca8a7f2

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

rolnico self-assigned this Jul 12, 2024

rolnico added 6 commits July 12, 2024 10:01

add builder and file information

cc1bcc1

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

add existsStrict method

1efab79

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

make importers use existStrict

0b22416

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

coverage + fix mainExtension

64c474e

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

fix test

b3eb831

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

coverage on ZipArchiveDataSource

ab30f56

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

rolnico added Breaking Change API is broken bug labels Jul 15, 2024

rolnico added 8 commits July 15, 2024 12:46

add a conversion test

272c8d8

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

use existing file in conversion test

2f54962

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

fix test

0f85f55

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

move boolean exists(String suffix, String ext) to AbstractFileSystemD…

2a59770

…ataSource Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

remove mention to mainExtension in tests

5db7158

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

remove semicolon

77b606f

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

Merge branch 'refs/heads/nro/datasources_reorganisation' into nro/dat…

a298501

…asources_2_main_extension # Conflicts: # commons/src/main/java/com/powsybl/commons/datasource/ReadOnlyDataSource.java

revert commit mainExtension in tests

4fd2dba

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

rolnico force-pushed the nro/datasources_2_main_extension branch from c76c4eb to 4fd2dba Compare July 15, 2024 14:45

jonenst reviewed Jul 15, 2024

View reviewed changes

rolnico requested a review from flo-dup July 16, 2024 11:56

rolnico added 2 commits July 16, 2024 16:55

revert commit mainExtension in tests

da3727f

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

fix issue

dcf8bb9

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

fix import

ba347f1

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

rolnico marked this pull request as ready for review July 18, 2024 08:46

Base automatically changed from nro/datasources_reorganisation to main July 19, 2024 13:55

rolnico added 4 commits July 22, 2024 09:29

fix test

c9b6a46

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

fix GenericReadOnlyDataSource + test

e455986

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

fix checkstyle

f7187a6

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

flo-dup reviewed Jul 26, 2024

View reviewed changes

fix javadoc

25588fe

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

flo-dup reviewed Jul 31, 2024

View reviewed changes

rolnico added 5 commits July 31, 2024 15:36

factorise some code

10bd397

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

simplify expression

3ae79a6

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

change mainExtension to dataExtension

1f806e9

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

fix EXTENSIONS in CgmesOnDataSource

d90024c

Signed-off-by: Nicolas Rol <nicolas.rol@rte-france.com>

Merge branch 'refs/heads/main' into nro/datasources_2_main_extension

2f047be

flo-dup approved these changes Jul 31, 2024

View reviewed changes

flo-dup merged commit 4b4f148 into main Aug 1, 2024
7 checks passed

flo-dup deleted the nro/datasources_2_main_extension branch August 1, 2024 14:39

jonenst reviewed Aug 2, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Datasources - Part 2: main extension + existStrict + builder #3102

Datasources - Part 2: main extension + existStrict + builder #3102

rolnico commented Jul 12, 2024 •

edited by flo-dup

Loading

jonenst Jul 15, 2024

rolnico Jul 16, 2024

jonenst Jul 15, 2024

jonenst commented Jul 15, 2024

rolnico commented Jul 16, 2024

sonarcloud bot commented Jul 31, 2024

jonenst Aug 2, 2024

rolnico Aug 2, 2024

Datasources - Part 2: main extension + existStrict + builder #3102

Datasources - Part 2: main extension + existStrict + builder #3102

Conversation

rolnico commented Jul 12, 2024 • edited by flo-dup Loading

jonenst Jul 15, 2024

Choose a reason for hiding this comment

rolnico Jul 16, 2024

Choose a reason for hiding this comment

jonenst Jul 15, 2024

Choose a reason for hiding this comment

jonenst commented Jul 15, 2024

rolnico commented Jul 16, 2024

sonarcloud bot commented Jul 31, 2024

Quality Gate passed

jonenst Aug 2, 2024

Choose a reason for hiding this comment

rolnico Aug 2, 2024

Choose a reason for hiding this comment

rolnico commented Jul 12, 2024 •

edited by flo-dup

Loading