RDF4J 4.0 is in the making! Have your say

[abrokenjester]

We have started development work on the next major release of RDF4J: version 4.0.

A large part of this release will be about some compatibility-breaking changes that we have been saving up over the past couple of years. In particular:

1. we will upgrade to Java 11 as the minimally required version of Java for RDF4J
2. The SeRQL query engine and all related code will be removed
3. The Lucene-Spin Sail and the Spin Sail will both be removed
4. several obsolete classes and methods throughout the code base will be removed
5. we are reorganizing parts of the Maven module structure to give us more flexibility and set us up for using Java modularity

Apart from this, we'll also introduce several new features of course, including things like easier integration with Spring, to allowing concurrent reads from a single SailConnection.

For an overview of what's planned, currently, see https://github.com/eclipse/rdf4j/milestone/30 .

We'd also like to invite you to think along, and let us know what you'd like to see changed in the current framework (now that we have a chance to make some breaking changes), or what you definitely do not want to see thrown out, or anything else that you'd like to see and/or would be interested in helping out with!

[neumarcx]

Thank you for the update Jenn, with regards to

3. The Lucene-Spin Sail and the Spin Sail will both be removed

Is the projected feature replacement here to use ShEx and/or SHACL?

Marco

[reckart]

The simple "Lucene Sail" will remain though, will it?

[barthanssens]

Yes the Lucene Sail (full text search) will continue to be part of RDF4J.

For RDF4J v4.0 we plan to upgrade the Lucene dependencies to 8.x (exact version also depends on ES/Solr, which depend on Lucene)

[reckart]

Do you have any strategy regarding dealing with the switch of ES to another license? In particular, do you limit the upgrade to that last ES version under the old license or do you plan to upgrade to a version under the new license? Maybe support both?

[barthanssens]

The plan is to at least go to the latest 7.x with the old license, but probably to be discussed with the Eclipse foundation / IP team if/how we can use later versions without too much hassle...

Also depends on the number of people actually using ES :-)

RDF4J 4.0 dev branch includes ES 7.x, but not the latest version nor the latest with the "old" license (have to look deeper into it, but there is a Solr test failing with latest Lucense 8.x, and we try to keep the libraries in sync).

[reckart]

In my situation, I'm not using the ES support of RDF4J at all, but I have dependencies on ES because it is used for a different purpose - and it is a mess. In order to be able to depend on RDF4J+Lucene-Sail, ES, Lucene, and MTAS (another search library build on top of Lucene), it is always a bit of a pain to find a sweet spot of versions where all these four are compatible. IMHO ES doesn't necessarily make it easy on people by depending on Lucene instead of e.g. including a shaded version of whatever they need.

[seralf]

Hi, are there any discussion for version 4 plan on rebuilding the server/workbench components into something more self-contained? (following from the discussion here: #1502).

[abrokenjester]

Nothing really beyond what is discussed there. There's the docker images of course - we keep meaning to give those a bit more prominence. But for actually doing a self-contained executable, we'd need some contributors with the capacity to spend some serious time on that.

[hmottestad]

Should we remove URI?

[abrokenjester]

@knoan we are accepting PRs for 4.0, yes, against the develop branch, so we'd be very happy to see your improvements! Don't forget to create an issue for whatever improvement you're submitting though (and I'd suggest keeping things as self-contained as possible).

As for what can be removed: of course we can only really remove things that have been marked deprecated at least a few minor releases back (or unless it's something hidden deep in the bowels where we're fairly sure no-one is likely to be using it anyway).

As for removing dependencies on javax.xml.datatype - can you elaborate on why do you consider that desirable? Edit: never mind, saw your other reply below, that jogged my memory.

[knoan]

As for what can be removed: of course we can only really remove things that have been marked deprecated at least a few minor releases back (or unless it's something hidden deep in the bowels where we're fairly sure no-one is likely to be using it anyway).

Got it. IMHO, We should at least consider deprecating Simple* classes and start upgrading internals to o.e.r.model, leveraging their Generic* counterparts already available within o.e.r.model.base.

As for removing dependencies on javax.xml.datatype - can you elaborate on why do you consider that desirable?

1. From a cursory inspection I'd say that most of javax.xml.datatype dependencies relate to legacy XML date types
2. javax.* packages are unlikely to be available on constrained devices, whose support was one of the original motivations for decoupling the abstract model to a dedicated lightweight package
3. java.xml.* packages were released with Java 1.5 and never really updated to XSD 1.1, which is the reference standard for SPARQL 1.1 datatypes

Dropping legacy/XML date support from Literal and ValueFactory would very likely remove javax.* dependencies at least from o.e.r.model; however, most of those methods aren't as yet deprecated, so according to the previous guidelines we are not at liberty to remove them…

[knoan]

I see that work is already in progress on URI and related deprecations removal at #3288 .

I could consider #2565 and removal of legacy calendric values in the context of #2563, but both actions are probably better delayed until #3288 is merged.

Concerning #2563, let me know if we consider acceptable removing legacy calendrical methods from Literal and related classes without previous deprecation notice.

[abrokenjester]

Concerning #2563, let me know if we consider acceptable removing legacy calendrical methods from Literal and related classes without previous deprecation notice.

We don't, in general, consider this acceptable, as it breaks semantic versioning. In rare cases we can make an exception, but this is part of a very public piece of the API, and migrating to using java.time classes is a non-trivial bit of effort for our users as well. I think adding the new time-based APIs and marking all legacy methods as deprecated is the only way forward here. Full removal will have to wait until the next major release.

[knoan]

Ok. Let's just deprecate existing legacy methods.

[knoan]

A good opportunity for completing #2563 and removing dependencies from legacy Java time APIs and javax.xml.datatype.XMLGregorianCalendar.

[knoan]

<brainstorming possibly far-fetched ideas>

• replace o.e.r.common.iterator/iteration stuff with streams
• review, refactor and migrate o.e.r.common.* classes
• reduce public API surface by downgrading all implementation classes to package-level visibility and hiding complex subsystems behind facades (that would provide more leeway for future internal refactoring)

</brainstorming>

[abrokenjester]

replace o.e.r.common.iterator/iteration stuff with streams

Interesting. What would be the benefit of this according to you? And how would you consider doing this without completely breaking every existing RDF4J-based project?

review, refactor and migrate o.e.r.common.* classes

This has, to a large degree, already happened. See a.o. #2710 .

reduce public API surface by downgrading all implementation classes to package-level visibility and hiding complex subsystems behind facades (that would provide more leeway for future internal refactoring)

It's definitely on my wish list to reduce the public API surface. I'm not sure what exactly you're proposing here though. Hiding all implementation classes to package-level visibility will make the framework unusable. Are there specific packages you're thinking of? Related to this is also the fact that we want make sure each package name is only used in one module (see #1494), as a prelude to possible use of the "new" Java modules to enhance encapsulation (though there are pros and cons to that).

[knoan]

replace o.e.r.common.iterator/iteration stuff with streams

Interesting. What would be the benefit of this according to you? And how would you consider doing this without completely breaking every existing RDF4J-based project?

Not that easy ;-) but with possible minor exceptions like MinusIteration iterator/iterations seem to map fairly well on the Java Stream API: upgrading would simplify the code base with possible performance improvements and provide largely transparent support for parallel processing of query results.

In order to maintain some degree of backward compatibility we could consider something along the lines of deprecating existing classes and providing a reverse adapter for converting streams to iterations, complementing the existing Iterations.stream(Iteration) static adapter method.

review, refactor and migrate o.e.r.common.* classes

This has, to a large degree, already happened. See a.o. #2710 .

I see.

reduce public API surface by downgrading all implementation classes to package-level visibility and hiding complex subsystems behind facades (that would provide more leeway for future internal refactoring)

It's definitely on my wish list to reduce the public API surface. I'm not sure what exactly you're proposing here though. Hiding all implementation classes to package-level visibility will make the framework unusable. Are there specific packages you're thinking of?

Not really sure here, but when wading through older code sometime you get the feeling that it could be better encapsulated. Minor examples for instance within o.e.r.sail.memory/nativerdf, which expose internals like optimised value implementations, datastores/btrees and the like.

[rwalkerands]

3. The Lucene-Spin Sail and the Spin Sail will both be removed

Uh, any impact on any other inferencing, e.g., RDF Schema inferencing and Custom Inferencing as per sections 1.4 and 1.5 of https://rdf4j.org/documentation/programming/repository/ ? Just checking (I hope) ...

[abrokenjester]

No worries, both RDF Schema inferencing and the custom inferencer will remain.

There is a possibility (this hasn't been discussed or scheduled yet but I just thought of it) that we'll remove the deprecated ForwardChainingRDFSchemaInferencer, in favor of the (now standard, and recommended) SchemaCachingInferencer - they both do the same thing, the latter is just better at it. But I'm in no rush on that score, and haven't thought the consequences through yet fully.

[hmottestad]

Was hoping to maybe rename the SchemaCach... to just "RdfsReasoner" or "RdfsInferencer" or "RdfsReasonerSail".

By rename I mean to essentially copy over all the code into new files and then deprecate the old files. 

[fkleedorfer]

Is there an ETA for 4.0 already? I am wondering about this as I have some contributed code in there I would like to switch to but I cannot have a snapshot dependency past a certain date. On the other hand, maintaining two code bases is less than ideal. Knowing a target date would help me make the decision either way.

Cheers!

[abrokenjester]

We don't have a date for a final release yet I think, and due to availability of various people (including myself) we've not quite moved as fast as I thought we would. However what I'm hoping to do is a first milestone build, by the end of the month.

[fkleedorfer]

Thanks, that was helpful!