Maintaining parity with Rails going forward

[headius]

Today I had a discussion with @enebo and @matthewd about the challenges maintaining JRuby support for ActiveRecord. A few days ago I chatted about similar challenges with @rsim about the same. I think it's past time we get stakeholders like them and @kares @rdubya and others together to see how we can all share our efforts to keep up with Rails releases and make it easier to onboard new databases.

The discussion we had with @enebo and @matthewd was on Matrix here.

The main challenge is that there's varying levels of custom backend driver-specific code across the three main adapters for SQLite3, MySQL2, and PostgreSQL. Code that is specific to the native driver (e.g. pg) is in the same files next to code that is database-specific but not driver-specific.

Our process of updating arjdbc for sqlite and mysql involves us updating code we've largely copied out of Rails and customized with JDBC logic. An example gist of the sqlite driver shows this well: https://gist.github.com/headius/423138b66fdad9012088358ae623b339

In that diff you can see huge parts of the file are identical (this was diff -w since we have some minor whitespace differences). Those pieces obviously could be shared. For this 7.1 version we largely have sqlite working properly.

for MySQL the diff is larger with less shared, but still potentially could be structured in a more shareable way. MySQL support for 7.1 is working but perhaps with less confidence.

PG has the least sharing and seems to have the most driver-specific code, sprinkled generously across nearly all the files in AR's postgresql adapter. We still don't have a working PG for 7.1 yet and it's holding back our users.

How should we move forward working with rails-core and arjdbc stakeholders to make this process easier?

[rdubya]

Hey all,
Glad to hear there is a push to get this stuff easier to maintain. I unfortunately haven't worked professionally with JRuby for a few years now, so most of my recent contributions were just things that I had time to pick at.

To address the Postgres stuff: I think the bulk of the code we had to override/work around was because there was a direct reference to the pg gem which has a totally different structure. There is also some places where we had to do hacky things because of syntax. For example, PG uses $1, $2, etc for parameterized queries and JDBC just wants a bunch of "?"s for identifiers. We also pull type information out of the database response differently because of the JDBC layer. There are a lot of things that could be simplified if the PG gem wasn't so tightly coupled with the AR adapter code.

I don't have much experience with the mysql adapter, but iirc the sqlite adapter has similar issues and has a lot of generic code in the main adapter class, which we had trouble extending. That means our own adapter with a lot of the generic code copied into it. A lot of that could be reduced by either pulling some of it into modules that could be included in both or figuring out a way to have a base sqlite adapter class that was then extended by the different implementations.

Happy to contribute where I can, but with my other commitments, I'll probably be mostly limited to answering questions and providing opinions :/

[rsim]

We are actively using activerecord-jdbc-adapter in our eazyBI application (currently using Rails 6.1). We are supporting multiple database connections in our application and are supporting MySQL, PostgreSQL, Microsoft SQL Server (using patched activerecord-jdbc-adapter) and Oracle (using https://github.com/rsim/oracle-enhanced). We also have minimal support for ClickHouse, Vertica, and Snowflake databases (just for connection management, schema creation, and execution of SQL statements).

Currently, the main problems of using activerecord-jdbc-adapter are:

• The source code contains a lot of adapters that are based on the original base JDBC adapter but now this main JDBC adapter is broken on Rails 6 or later. It would be good to fix the base JDBC adapter.
• Probably it would be good to remove all other database-specific adapters that are not maintained. Probably it would be good to have clear examples and / or specs how to add additional database-specific adapters based on the base JDBC adapter.
• We have copied mssql adapter code from https://github.com/JesseChavez/activerecord-jdbc-adapter to our application to support Microsoft SQL Server. It would be good either to merge it in the main activerecord-jdbc-adapter or to make as extension of the base JDBC adapter.
• We have made several Java class patches master...eazybi:activerecord-jdbc-adapter:eazybi for MySQL and PostgreSQL.
• Other small Ruby code patches for MySQL and PostgreSQL adapters are in our application.

So it would be good to agree on the way forward what should be in the main activerecord-jdbc-adapter and what could be added as extensions in separate repositories / gems. We are ready to submit our patches upstream after deciding on the scope of this repo / gem.

[enebo]

@rsim When I started working on Rails 5 support the plan was to eliminate all adapters we couldn't support. There was some deletion but obviously it never completely removed all remnants of the other adapters. Any PRs to remove the rest I think seems like a great idea.

The second thing also related to the Rails 5 changes was I didn't target the base generic JDBC adapter and I didn't really know how to. Having some basic bare JDBC db which exercised that code seems like something we need added back. Since you have some other dbs which probably does use this I feel like we should perhaps add one or all of those back to this project.

Personally, the main problem of ARJDBC is lack of contribution and thin resources to support it. You actually have commercial need for these adapters to work and I am wondering if it makes sense for you to merge adapters back to this project and become an active member?

I think the only real caveat to desiring multiple repos are:

1. This adapter forces an all adapter release strategy because they all rely on the same Java classes. It is not horrible to release new versions of every adapter whenever any single adapter needs a release but it is confusing. If there is some good solution to not have this happen (shading?) then perhaps this oddity can go away. I still think even with this issue more dbs in here is fine so long as someone maintains them.
2. There is something in your mods which is too specific to what you offer (e.g. it is an application feature and not some general feature for the adapter). In that case, it makes more sense to be external.

My take overall is what can we do to get most of your code back into this repository or barring that how can we get a reasonable stream of fixes for the adapters we do support?

---

My addition to this discussion beyond what @headius had mentioned in the first post from the original conversation is that we still have a lot of conditional behavior associated with JDBC-specific things. I have no comprehension as to whether some of this is old stuff or options we still need to support. So in addition to removing old unsupported adapter vestiges we need someone with knowledge to go over the JDBC-specific code paths. I did remove a lot of code associated with specific-versioned JDBC that was ancient but there is still a lot of that logic.