A "org.eclipse.equinox.p2.transport.javahttpclient" transport using directly Java HTTP client or OSGi services

[mickaelistria]

Currently, the only transport available in p2 goes through ECF, which does the job well. But still it adds 1 layer of indirection which might not always be desired, and Java now should have everything (API-wise) to directly consume it and still have extensibility to add new protocol.
So we should consider incepting a org.eclipse.equinox.p2.transport.javahttpclient which would directly consume Java HTTP Client, as an alternative to ECF-based transport.

[mickaelistria]

Note that m2e for example has "eclipse-m2e/m2e-core@b632cef#diff-1346eb18ae8b11904767f6574e3aebf01bca948f837865a449c8c4b41141b8f1" which seems to hook directly into OSGi. It might be an interesting approach to use the OSGi service instead of Java one (although direct Java services still seem simpler).

[laeubi]

Basically the m2e extends the URL Stream Handler inside java what is about URL>Connection. Such a handler is available by default for http(s). The more interesting part is caching where one can befit from protocol specific things.

Tycho contains a transport that don't uses ECF and integrates with URLStreamHandlerService as well:
https://github.com/eclipse-tycho/tycho/tree/main/p2-maven-plugin/src/main/java/org/eclipse/tycho/p2maven/transport

[merks]

My biggest personal concern is that there is the potential for new problems that I didn't have before and that I have no way of working around. For example, the newer httpclientjava mostly worked well for my use cases, but the problem with GOAWAY exceptions were so common that I needed to switch to the old client for SimRel:

https://github.com/eclipse-simrel/simrel.build/blob/c53643eafb315ddb34dc2df64bec48827f523013/pom.xml#L59

I would hope not to repeat that experience, and then have no recourse.

---

My biggest general concern is to replace something that is tried-test-and-true for umpteen years with something new-and-improved but unproven. Specifically the types of problems (that we also saw from the previous effort) that would affect a significantly large portion of our consumer base:

eclipse/ecf#76

---

None of this is to say we should therefore not try or that trying is a bad things, but if we are to try, we better try really hard and produce something of industrial quality that is confirmed to work in complex/secure network environments. There needs to be significant commitment, including considerations about who is responsible for supporting this replacement.

[laeubi]

I can just add something here, as it is often refereed to use "Java HTTP client" (or here "OSGi services"), the problem is not really the http client or to integrate with other protocols, this is solvable in a few lines and I actually did that for ECF but the main problems are:

1. (Proxy) Authentication, what is not really nicely abstracted in P2 and currently relies on a wild interaction between ECF+P2, that can be solved (see P2) but will still be challenging as it potentially "breaks" things (e.g. Ooomph) that use/rely on internal API... and then there is actually demands to integrate this with the UI what currently only support Username/Password
2. Proxy Server (Authentication) this is basically (again) not using any standard nor is there any useful abstraction in P2, even worse we have no tests and no test infrastructure for that part, it purely works because it worked some time in the past an we never touch this.
3. Caching, is actually best done on transport level, currently is done by P2 but on different levels e.g. metadata caches (what are server specific) and artifact caching on several levels (e.g. bundle pools) what actually is not server specific (see mirrors!)
4. Legacy some things work because they work always that way and changing might break in any way (see @merks )
5. Most worse: No one want this (from business POV), because exchanging something that works with something that works the same is not very profitable and the risk of change is high.

I really would love to work on these issues, but would rather do it for money than for fun (because it is not fun), but doubt anyone will be willing to invest any money in such an area.

Regarding ECF: The most logical way would be to simply fork the classes we need (but what about extension points someone possibly using?) as 95 % of ECF is of no use for P2.

[merks]

@laeubi

I think you captured very well the most relevant details and considerations.

I think these are possible courses of action:

• Support ECF to keep it viable.
• Fork the parts of ECF that are required to keep those specific parts viable.
• Re-implement everything currently implemented by ECF and keep that viable.

Perhaps there are courses of action I've not considered...

[mickaelistria]

Thanks for the input. I have just brought this to discussion after yesterday call where it appeared ECF is at risk. I do believe that for the same feature, the least code the better and that's why Java http client seems promising on the long run; but as you highlighted, adopting it would be a significant effort; which doesn't seem so profitable immediately.
It's more food for future thoughts then.
I leave it to you to discuss the best outcome of this ticket. You can close it if you prefer, or keep the idea pending open.

[laeubi]

I referenced it now also here (found no good way to ping relevant interested users):

• https://gitlab.eclipse.org/eclipse-wg/ide-wg/ide-wg-dev-funded-efforts/ide-wg-dev-funded-program-planning-council-top-issues/-/issues/37#note_2471638

[jonahgraham]

There seems to be an open question as to which approach for handling http going forward should happen. The ECF and Equinox/p2 project teams need to decide on the correct course of action so that the planning council can recommend funding this work to the IDE WG. (See this comment in gitlab)

[akurtakov]

IMO the major thing here is fixing proxy support with java httpclient so the implementation doesn't require extra library Apache httpclient.
The question after that is how to "glue" this code in p2 to make Platform releng/P2/ECF/etc. developers life easier but no one is considering coming up with NEW Java httpclient implementation AFAICT.