support docker-aio

[pdurbin]

In pull request #7 I started adding dependencies for docker-aio such as a yum install of Docker itself. 😄 Thank you to @donsizemore and @poikilotherm for helping me! I don't think I've ever installed Docker on anything but a Mac or a Linux laptop before.

I'm spinning off this new issue from #1 because my current goal is parity with what I have at https://build.hmdc.harvard.edu:8443/job/phoenix.dataverse.org-apitest-develop/ and I don't use Docker for that. So I don't see Docker as an absolute dependency for parity but it should would be nice to get it working. Easier on the AWS bill for sure.

Parity is the first goal I tried to explain in my "thoughts on automated testing" post at https://groups.google.com/d/msg/dataverse-dev/ISot5k4VjZQ/t-hzPk8tAwAJ which I expressed as " Get https://jenkins.dataverse.org to do the same job that the older Jenkins server and phoenix do which is to catch regressions before a release."

Because I'm having so much trouble getting docker-aio working on a newly spun up Jenkins server I decided to back up and run in on my Mac again. I got a working Dataverse. I'm on IQSS/dataverse@3ce173c9d

Mac

Setting up your glassfish4 to support Dataverse
Glassfish directory: /usr/local/glassfish4
Domain directory:    /usr/local/glassfish4/glassfish/domains/domain1
/usr/local/glassfish4/bin /opt/dv/testdata/scripts/installer
domain running
remote failure: Application dataverse-4.0 is not deployed on this target [server]
Command undeploy failed.
Deleted 1 option(s)
Command delete-jvm-options executed successfully.
Created 1 option(s)
Command create-jvm-options executed successfully.
Created 1 option(s)

On my Jenkins server (a t2.large), I'm getting the following error.

Jenkins server

Setting up your glassfish4 to support Dataverse
Glassfish directory: /usr/local/glassfish4
Domain directory:    /usr/local/glassfish4/glassfish/domains/domain1
./glassfish-setup.sh: line 285: pushd: /usr/local/glassfish4/bin: No such file or directory
./glassfish-setup.sh: line 289: ./asadmin: No such file or directory
domain running
./glassfish-setup.sh: line 49: ./asadmin: No such file or directory
./glassfish-setup.sh: line 53: ./asadmin: No such file or directory
...

Help me @pameyer you're my only hope!

[pameyer]

@pdurbin I'm partly speculating here, but my suspicion is that the base system doesn't have unzip installed; so that script that downloads and patches glassfish can't unpack the distribution archive. If this is on the right track, conf/docker-aio/dv/deps/glassfish4dv.tgz should be a lot smaller than usual, and not have a functioning glassfish inside it.

If that's not the case, it might be image caching (aka - docker images show a different hash for the centos:7 image being used.

[pdurbin]

@pameyer THANK YOU! In my previous commit I had added yum install zip but in 8ea4fc3 I just added yum install unzip. I just spun up a new EC2 instance and ran conf/docker-aio/prep_it.bash which ended with this output:

docker-aio ready to run integration tests
{"status":"OK","data":{"version":"4.14","build":"develop-3c697b1"}}

Fingers crossed!

[pdurbin]

Woo-hoo! Running conf/docker-aio/run-test-suite.sh not only worked, all the tests passed!

[INFO] Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.266 s - in SearchIT
[INFO] 
[INFO] Results:
[INFO] 
[WARNING] Tests run: 92, Failures: 0, Errors: 0, Skipped: 4
[INFO] 
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 4:05.998s
[INFO] Finished at: Fri Jun 07 10:50:59 UTC 2019
[INFO] Final Memory: 65M/755M
[INFO] ------------------------------------------------------------------------

I'm on 8ea4fc3 of this dataverse-jenkins repo (on a newly spun up t2.large EC2 instance) and on IQSS/dataverse@3c697b1 on the dataverse side.

[pdurbin]

I created a new job in Jenkins based on the above but my latest blocker is some sort of permission problem with /tmp. Here's a screenshot from http://ec2-3-81-58-115.compute-1.amazonaws.com:8080/job/IQSS-dataverse-api-test-suite-docker-aio-develop/1/testReport/junit/edu.harvard.iq.dataverse.export/SchemaDotOrgExporterTest/testExportDataset/

[pdurbin]

Over at IQSS/dataverse#5885 @donsizemore reported exactly this problem:

Similarly, in SchemaDotOrgExporterTest.java:
try (PrintWriter printWriter = new PrintWriter("/tmp/dvjsonld.json"))
breaks the SchemaDotOrgExporter test with a permission denied.

[pdurbin]

For now I'm just going to delete the file in /tmp that was created when I did the run as "centos" rather than "jenkins":

[root@ip-172-31-34-143 dataverse-jenkins]# ls -l /tmp/dvjsonld.json
-rw-rw-r--. 1 centos centos 2458 Jun  7 10:35 /tmp/dvjsonld.json
[root@ip-172-31-34-143 dataverse-jenkins]# rm /tmp/dvjsonld.json
rm: remove regular file ‘/tmp/dvjsonld.json’? y
[root@ip-172-31-34-143 dataverse-jenkins]# 

[pdurbin]

Bah!

dvinstall/solrconfig.xml dvinstall/install
  adding: dvinstall/glassfish-setup.sh (deflated 68%)
  adding: dvinstall/pgdriver/ (stored 0%)
  adding: dvinstall/pgdriver/postgresql-42.2.2.jar (deflated 7%)

  adding: dvinstall/dataverse.war
zip I/O error: No space left on device

A t2.large isn't big enough.

[root@ip-172-31-34-143 dataverse-jenkins]# df -hl
Filesystem      Size  Used Avail Use% Mounted on
/dev/xvda1      8.0G  7.9G  176M  98% /
devtmpfs        3.8G     0  3.8G   0% /dev
tmpfs           3.9G     0  3.9G   0% /dev/shm
tmpfs           3.9G   17M  3.9G   1% /run
tmpfs           3.9G     0  3.9G   0% /sys/fs/cgroup
tmpfs           783M     0  783M   0% /run/user/1000
overlay         8.0G  7.9G  176M  98% /var/lib/docker/overlay2/90125d0480750e69da4bb229aa3f1b68e9e0bd4ff9d771429926be3460e559f1/merged
shm              64M  4.0K   64M   1% /var/lib/docker/containers/bbccc5cc756edeb7dbd5391d83f06cceb3607bdf6af598ddef90fea34e780316/mounts/shm
tmpfs           783M     0  783M   0% /run/user/0
[root@ip-172-31-34-143 dataverse-jenkins]#

[pdurbin]

@donsizemore is the boat at https://jenkins.dataverse.org big enough for you to play around with pull request #9? 😄 I just requested a review from you. I'd be happy for @pameyer and others to take a look as well.