Skip to content

DB2

Jump to bottom
Leonard Carcaramo edited this page Apr 15, 2021 · 1 revision

DB2 Support For Artifactory (Unable to implement)

Note that the Vault team was unable to add DB2 support to Artifactory given time and resource restraints, and the complexity of Artifactory's database logic. It MAY be possible to add DB2 support to Artifactory given sufficient time and resources. This wiki exists to document what progress the Vault team was able to make in adding DB2 support for anyone to refer to including but not limited to anyone who attempts to add DB2 database support to Artifactory in the future.

Also note that the Vault team did all work for adding DB2 database support to Artifactory building an running Artifactory as a containerized application on zCX.

Lastly, note that Artifactory is a Java application built using Maven, and that it depends on dozens of frameworks and libraries that the Vault team is not familiar with.

⚠️ All code that we wrote and JFrog source code that we modified is available here in this repository to meet the conditions of JFrog Artifactory OSS's license. (AGPL 3.0)

 

Database configuration in Artifactory

All database configuration for Artifactory is kept in a properties file called db.properties.

type=<database type>
url=<url endpoint to database>
driver=<jdbc driver for corresponding database type>
username=<username>
password=<password>

The required properties can vary, but in the context of DB2, below are the required properties in order for Artifactory to establish a connection to a DB2 database.

This config file is stored at /opt/jfrog/artifactory/etc inside the Artifactory container.

Here is an example of what a db.properties file might look like for a DB2 database if Artifactory supported DB2:

type=db2
url=<url endpoint to database>
driver=com.ibm.db2.jcc.DB2Driver
username=<mvs userid>
password=<mvs password>

⚠️ Ensure you are calling your JDBC Driver com.ibm.db2.jcc.DB2Driver and NOT com.ibm.db2.DB2Jcc. We are not entirely sure why, but Artifactory will not be able to load your JDBC Driver if you don't do this. You will get the following error if you don't name your JDBC Driver correctly:

SEVERE: Unable to create initial connections of pool.
java.sql.SQLException: class com.ibm.db2.jcc.DB2Jcc cannot be cast to class java.sql.Driver (com.ibm.db2.jcc.DB2Jcc is in unnamed module of loader java.net.URLClassLoader @2ac273d3; java.sql.Driver is in module java.sql of loader 'platform')

Note that we tried using the other database types (mySQL, Oracle, MariaDB, PostgreSQL, and Derby) to to see if they could utilize a DB2 database, but had no luck because the SQL that each of those database types use is incompatible with DB2.

You can manually configure db.properties, but if you are using Docker or zCX, you can optionally pass in your database configuration as environment variables:

docker run --name <container name> -d -p 8081:8081 \
           -e DB_TYPE=<db type> \
           -e DB_USER=<username> \
           -e DB_PASSWORD=<password> \
           -e DB_URL=<db url> \
           -e SKIP_DB_CHECK=<set true if testing DB2> \
           jfrog/artifactory-oss:<version>

The environment variables in the above command are used to dynamically create the db.properties file. In order for this to work with DB2 you would need to create a db2.properties template file. You can either create it directly in the source code at jfrog-artifactory-oss/distribution/standalone/zip/src/main/install/misc/db, or as part of Artifactory's Dockerfile at /opt/jfrog/artifactory/misc/db. A complete DB2 implementation for Artifactory would include the db2.properties template file in the source code. Additionally, Artifactory's Docker entrypoint script would need to be modified.

// db2.properties template file might look something like this
type=db2
driver=com.ibm.db2.jcc.DB2Driver
url=jdbc:db2://localhost:5432/artifactory
username=artifactory
password=password

 

JDBC Drivers and License files

JDBC drivers for Artifactory are stored at /opt/jfrog/artifactory/tomcat/lib inside the Artifactory container. (The directory structure will be similar for an Artifactory service not running inside a container. The absolute path will likely be different however.)

The aforementioned directory is where the DB2 JDBC driver and DB2 license file should be inserted.

If you are using zCX or Docker, you should copy the JDBC driver and license file to Artifactory as part of your Dockerfile using quay.io/ibm/jfrog-artifactory-oss:<version> as the base image for Artifactory with DB2 support. There are other ways to do this, but this is the easiest way.

# Add this to Dockerfile
COPY <path to DB2 JDBC Driver on local system> /opt/jfrog/artifactory/tomcat/lib/
COPY <path to DB2 license file on local system> /opt/jfrog/artifactory/tomcat/lib/

A complete DB2 implementation for Artifactory would require that users manually put their JDBC Driver and licesne file in the corresponding folder.

Additionally, in order to establish a connection to a DB2 database, the path to the DB2 JDBC Driver and the DB2 license file must be added to the $CLASSPATH (Do this in the Dockerfile if using Docker or zCX).

# Add this to Dockerfile
ENV CLASSPATH=<path to DB2 JDBC Driver inside container>:<path to DB2 license file inside container>:CLASSPATH

We are not sure how Artifactory handles class loading, but all we know is that nothing exists inside the $CLASSPATH environment variable. Anyone who creates a complete DB2 implementation for Artifactory should figure out how Artifactory handles class loading and make it so Artifactory can find DB2 JDBC drivers and license files if necessary. It's likely that one of the frameworks/libraries that Artifactory uses handles class loading in some other way that doesn't involve the $CLASSPATH environment variable.

 

Adding DB2 support to Artifactory (First steps)

  1. Artifactory keeps a list of all the database types it supports in an enum called DbType. This is the first thing that needs to be changed. All you have to do is add DB2 to this enum. Note that this enum is an external dependency that Artifactory downloads from it's Maven repository at build time. This is a JFrog dependency, and it is licensed under the (AGPL 3.0) license, so the sources are publicly available. DbType is part of the org.jfrog.storage package. You can download it here: https://mvnrepository.com/artifact/org.jfrog.storage/jfrog-db-infra?repo=jcenter

  2. The second thing that has to be changed is DbUtils. DbUtils is also part of the org.jfrog.storage package, so it will be downloaded along with DbType. You only need to add DB2 to a couple switch statements in this class. Add DB2 to the switch statement inside the getActiveSchema method and the getActiveCatalog method.

// add this to each switch statement in the methods in DbUtils mentioned above.
case DB2:
  1. This is the point where the Vault team ran into some serious blockers, and it was decided to give up on DB2 and switch to PostgreSQL. At this point Artifactory fails at AccessJdbcHelperImpl:
Caused by: java.lang.UnsupportedOperationException: Create Schema - is not supported in db2 on zOS
        at org.flywaydb.core.internal.dbsupport.db2zos.DB2zosSchema.doCreate(DB2zosSchema.java:62)
        at org.flywaydb.core.internal.dbsupport.Schema.create(Schema.java:112)
        at org.flywaydb.core.internal.command.DbSchemas$1.call(DbSchemas.java:80)
        at org.flywaydb.core.internal.command.DbSchemas$1.call(DbSchemas.java:68)
        at org.flywaydb.core.internal.util.jdbc.TransactionTemplate.execute(TransactionTemplate.java:75)
        at org.flywaydb.core.internal.command.DbSchemas.create(DbSchemas.java:68)
        at org.flywaydb.core.Flyway$5.execute(Flyway.java:1105)
        at org.flywaydb.core.Flyway$5.execute(Flyway.java:1103)
        at org.flywaydb.core.Flyway.execute(Flyway.java:1413)
        at org.flywaydb.core.Flyway.baseline(Flyway.java:1103)
        at org.jfrog.access.server.db.util.AccessJdbcHelperImpl.setFlywayBaseline(AccessJdbcHelperImpl.java:167)
        at org.jfrog.access.server.db.util.AccessJdbcHelperImpl.createTablesIfNeeded(AccessJdbcHelperImpl.java:152)
        at org.jfrog.access.server.db.util.AccessJdbcHelperImpl.initDb(AccessJdbcHelperImpl.java:119)
        at org.jfrog.access.server.db.util.AccessJdbcHelperImpl.init(AccessJdbcHelperImpl.java:58)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:566)

This class is also an external JFrog dependency. AccessJdbcHelperImpl is part of the org.jfrog.access package. Unfortunately, it appears that JFrog provides no source code for this package: https://mvnrepository.com/artifact/org.jfrog.access/access-server-core This package is licensed under the (AGPL 3.0) license, so one should be able to get into contact with JFrog to request the source code for this package under the conditions of the license. The fact that we did not have immediate access to this package, and that it was becoming increasingly evident that adding DB2 support was going to become a difficult and time consuming process prompted the vault team to give up on adding DB2 support.

 

Maven and Replacing Dependencies

We do not have a full understanding of how Artifactory uses Maven, but we do know enough to figure out how Artifactory is getting it's dependencies, where it stores them, and how to modify them to enable DB2 support.

Note that we built Artifactory on zCX using the pom.xml file that is used to build Artifactory for Docker. Artifactory may build differently if you are not building it for Docker. The Artifactory image that we used is built using the following command: mvn clean pacakge -Pdocker using the pom.xml file located here: jfrog-artifactory-oss/distribution/docker

It seems that all of the JFrog dependencies are downloaded as a single War file named access-war-4.14.0.war. JFrog dependencies are the only ones that should need to be modified in order to add DB2 support to Artifactory.

Downloading from central: https://jcenter.bintray.com/org/jfrog/access/access-war/4.14.0/access-war-4.14.0.war

We do not know much about what happens between the maven build and the creation of the Artifactory Docker image, but what we do know is that access-war-4.14.0.war ultimately gets renamed to access.war and gets stored at /org/jfrog/artifactory/webapps. When you start an Artifactory container using this image, Artifactory extracts the contents of this war file to opt/jfrog/artifactory/tomcat/webapps/access. This means that you can create a Dockerfile that uses the Artifactory image as a base image, modify whatever dependencies need to be modified in access.war, and create an Artifactory container using the new image. Given that you modified the dependencies in access.war, you should see the new dependencies working in an Artifactory container that you spin up from the new image.

You also may need to compile the classes that you are replacing as part of your Dockerfile to avoid java version conflicts (We used a shell script that gets executed in our Dockerfile to accomplish this)

# copy java files that need to be compiled and replaced
COPY DbType.java /opt/jfrog/artifactory/webapps/
COPY DbUtils.java /opt/jfrog/artifactory/webapps/

# copy and run shell script that compiles java classes and puts them in the war file
COPY update-dependencies.sh /update-dependencies.sh
RUN ./update-dependencies.sh

update-dependencies.sh

#!/bin/bash

cd /opt/jfrog/artifactory/webapps
javac DbType.java -d tmp/WEB-INF/lib/
cd /opt/jfrog/artifactory/webapps
mv access.war tmp/
cd tmp
jar -xvf access.war
rm -rf access.war
cd WEB-INF/lib
jar -uvf jfrog-db-infra-3.12.0.jar org/jfrog/storage/DbType.class
cd ../../..
javac DbUtils.java -cp /opt/jfrog/artifactory/webapps/tmp/WEB-INF/lib/jfrog-db-infra-3.12.0.jar:/opt/jfrog/artifactory/webapps/tmp/WEB-INF/lib/lombok-1.18.2.jar:/opt/jfrog/artifactory/webapps/tmp/WEB-INF/lib/slf4j-api-1.7.26.jar:/opt/jfrog/artifactory/webapps/tmp/WEB-INF/lib/spring-jdbc-5.1.15.RELEASE.jar:/opt/jfrog/artifactory/webapps/tmp/WEB-INF/lib/javax.annotation-api-1.3.2.jar:/opt/jfrog/artifactory/webapps/tmp/WEB-INF/lib/commons-lang-2.6.jar:/opt/jfrog/artifactory/webapps/tmp/WEB-INF/lib/jsr305-2.0.0.jar -d tmp/WEB-INF/lib/
cd tmp/WEB-INF/lib
jar uvf jfrog-db-infra-3.12.0.jar org/jfrog/storage/util/DbUtils.class
jar uvf jfrog-db-infra-3.12.0.jar org/jfrog/storage/util/DbUtils\$1.class
mv org ../../../
cd ../..
jar -cvf access.war *
mv access.war ../
cd

This is most likely NOT the best way to accomplish this, but given that NO ONE on the Vault team was particularly experienced with Maven, this was the best way for the Vault team to quickly modify dependencies. Otherwise, more time would need to be spent learning about how Artifactory uses Maven, a custom maven repository would need to be created, Artifactory's pom would need to be modified, etc...

If the Vault Team or anyone else works on adding DB2 support to Artifactory in the future, A Maven expert will most likely be needed on the team in order to create a complete DB2 implementation for Artifactory.

Clone this wiki locally