Skip to content

Setup Gotcha's

Jump to bottom
martincsimpson edited this page Nov 24, 2023 · 6 revisions

Hyku 5 Setup Instructions

External Solr inside Docker

Solr Configuration

You will need to make some custom modifications to use an external Solr. The first step is to ensure you are using docker-compose-ext-solr.yml as it has some custom configuration to NOT initialize solr as we expect that Solr would already be initialized.

Environment Variable Configuration

As part of the setup you will need to configure the environemnt variables, they can be found in .env

Var Name Value Notes
SOLR_URL https://username:password@host/solr/ Note: must be HTTPS otherwise configset wont upload, must include trailing /solr/
SOLR_HOST host Only the solr host with no other items
SOLR_PORT 443 Must be 443 or another HTTPS port
SOLR_COLLECTION_NAME hydra-development Note, must be manually created
SOLR_CONFIGSET_NAME hyku Can be left as default
SOLR_ADMIN_PASSWORD pass Must be the same as in SOLR_URL
SOLR_ADMIN_USER user Must be the same as in SOLR_URL

Authentication

To securely upload a configset to Solr, you must have authentication enabled on the Solr APIs, this is referenced in the above SOLR_URL, SOLR_ADMIN_PASSWORD and SOLR_ADMIN_USER. The user must be granted administrative privileges.

Solr YML configuration

The existing Solr Configuration ./config/solr.yml is not able to use an external SOLR and requires a slight modification.

You will need to change all lines to support the SOLR_URL || statement.

development:
  url: <%= ENV['SOLR_URL'] + "hydra-development" || "http://127.0.0.1:#{ENV.fetch('SOLR_DEVELOPMENT_PORT', 8983)}/solr/hydra-development" %>
test:
  url: <%= ENV['SOLR_URL'] ? ENV['SOLR_URL'] + 'hydra-test' : "http://127.0.0.1:#{ENV.fetch('SOLR_TEST_PORT', 8985)}/solr/hydra-test" %>
production:
  url: <%= ENV['SOLR_URL'] || "http://#{ENV.fetch('SOLR_ADMIN_USER', 'admin')}:#{ENV.fetch('SOLR_ADMIN_PASSWORD', 'admin')}@#{ENV.fetch('SOLR_HOST', 'solr')}:#{ENV.fetch('SOLR_PORT', '8983')}/solr/" %>

note that the hydra-development collection name must be the same as the environment variable SOLR_COLLECTION_NAME

Running the SOLR Initialize Scripts

As part of the automated setup SOLR needs to be setup using bin/solrcloud-upload-configset.sh script. For this script to work some modifications need to be made to the script:

-solr_config_list_url="http://$SOLR_HOST:$SOLR_PORT/api/cluster/configs?omitHeader=true"
-solr_config_upload_url="http://$SOLR_HOST:$SOLR_PORT/solr/admin/configs?action=UPLOAD&name=$SOLR_CONFIGSET_NAME"
+solr_config_list_url="https://$SOLR_HOST:$SOLR_PORT/api/cluster/configs?omitHeader=true"
+solr_config_upload_url="https://$SOLR_HOST:$SOLR_PORT/solr/admin/configs?action=UPLOAD&name=$SOLR_CONFIGSET_NAME"
-solr_collection_list_url="$SOLR_HOST:$SOLR_PORT/solr/admin/collections?action=LIST"
-solr_collection_modify_url="$SOLR_HOST:$SOLR_PORT/solr/admin/collections?action=MODIFYCOLLECTION&collection=$SOLR_COLLECTION_NAME&collection.configName=$SOLR_CONFIGSET_NAME"
+solr_collection_list_url="https://$SOLR_HOST:$SOLR_PORT/solr/admin/collections?action=LIST"
+solr_collection_modify_url="https://$SOLR_HOST:$SOLR_PORT/solr/admin/collections?action=MODIFYCOLLECTION&collection=$SOLR_COLLECTION_NAME&collection.configName=$SOLR_CONFIGSET_NAME"

The above modifications allow the SOLR_URL and https values to be used with the configset setup script

Running the Solr Setup Script

You will need to run the Solr Setup script outside of docker (as we are using an external Solr), for this you will need to assign the environment variables and the solr configuration directory

  1. SOLR_ADMIN_PASSWORD=PASS SOLR_ADMIN_USER=USER SOLR_COLLECTION_NAME=hydra-development SOLR_CONFIGSET_NAME=hyku SOLR_HOST=HOST SOLR_PORT=443 bash -x solrcloud-upload-configset.sh ../solr/conf/

  2. Login to your Solr service and create the sollection hydra-development (or whatever your SOLR_COLLECTION_NAME is) manually using the hyku configset (or whatever your SOLR_CONFIGSET_NAME is

  3. Run this command again SOLR_ADMIN_PASSWORD=PASS SOLR_ADMIN_USER=USER SOLR_COLLECTION_NAME=hydra-development SOLR_CONFIGSET_NAME=hyku SOLR_HOST=HOST SOLR_PORT=443 bash -x solrcloud-upload-configset.sh ../solr/conf/

Note: Running it again is important as it will modify the collection once created

Note: At the moment it is not clear how the collection is supposed to be automatically created

Note: For some reason the solrcloud-upload-configset.sh script is not capable of running inside docker as it wont pickup the correct environment variables

Note: You can manually upload the configset to solr using the zkcli tool (located in the Solr package, NOT zookeeper). /root/solr-9.3.0/server/scripts/cloud-scripts/zkcli.sh -zkhost XXX:2181 -cmd upconfig -confdir ../conf/ -confname hyku

SMTP Setup

SMTP has not been designed to work at all in the development environment for Hyku5, so to enable it you will have to modify two files

  1. .env
Var Name Value Notes
SMTP_ADDRESS smtp.mailgun.org Host of SMTP Provider
SMTP_DOMAIN ubiquityrepo-eu.website Domain of Email being sent
SMTP_PASSWORD XXX SMTP Login Password
SMTP_USERNAME XXX SMTP Login Username
SMTP_FROM postmaster@ubiquityrepo-eu.website From Address of the emails being sent
SMTP_PORT 587 Note: 587 is the SSL SMTP port
  1. config/environments/development.rb

Change this value to true for debugging purposes

+  config.action_mailer.raise_delivery_errors = true

Add this block

+  config.action_mailer.smtp_settings = {
+      from: ENV['SMTP_FROM'],
+      user_name: ENV['SMTP_USER_NAME'],
+      password: ENV['SMTP_PASSWORD'],
+      address: ENV['SMTP_ADDRESS'],
+      domain: ENV['SMTP_DOMAIN'],
+      port: ENV['SMTP_PORT'],
+      enable_starttls_auto: true,
+      authentication: ENV['SMTP_TYPE']
+    }

This will enable emails to be sent from the development environment

Disabling required files for a work

Edit file config/initializers/hyrax.rb and change config.work_requires_files to false

Enabling hyku-api

Modify the Gemfile and add this line

+gem "hyku-api", github: "ubiquitypress/hyku-api", branch: "main"

Putting it all together

To start the environment run docker-compose -f docker-compose-ext-solr.yml.no-init up -d

Post start-up steps

DOI Integration

Gemfile

For DOI integration firstly make sure the gem is installed (it should be there by default)

Gemfile

gem 'hyrax-doi', git: 'https://github.com/samvera-labs/hyrax-doi.git', branch: 'main'

Installation

Run the install command rails g hyrax:doi:install --datacite

Attaching to Work Types

rails g hyrax:doi:add_to_work_type GenericWork --datacite rails g hyrax:doi:add_to_work_type Image --datacite note: make sure that you specify a real work type, otherwise it will return 0 and no value

This adds a line into the app/model/<work_type>.rb file looking like this:

  # Adds behaviors for hyrax-doi plugin.
  include Hyrax::DOI::DOIBehavior
  include Hyrax::DOI::DataCiteDOIBehavior

It also modifies the following files:

app/models/generic_work.rb
app/forms/hyrax/generic_work_form.rb
app/presenters/hyrax/generic_work_presenter.rb

note: sometimes the --datacite switch doesn't work, and causes the below issues. You may have to manually add the includes to the above files.

Access to class methods

TODO: Getting error

web_1            | ActionView::Template::Error (undefined method `doi_status_when_public' for #<GenericWork:0x00007f7415826528>):
web_1            |     1: <%# TODO: determine these from actual DataCite data instead of the intention field stored in fedora %>
web_1            |     2: <% disable_do_not_mint = @curation_concern.doi_status_when_public != nil %>
web_1            |     3: <% disable_draft = @curation_concern.doi_status_when_public.in?(['registered','findable']) %>
web_1            |     4: 
web_1            |     5: <div>

Integration of Bulkrax::AdvancedSearch

Note: Following this PR works: https://github.com/scientist-softserv/adventist-dl/commit/02db13063585b9ea45b7d5813564a7507a96883c

Issues starting docker-compose because puma not found

What you will need to do is create a new docker-compose file called docker-compose.bundle.yml and fill it with below:

# Copy file to docker-compose.override.yml to override docker-compose.yml
# Only use for local development
version: '3.8'
services:
  # Uncomment to allow for the use of a ruby debugger (byebug, pry, etc) in Docker.
  # See http://playbook-staging.notch8.com/en/devops/docker_debugger for more info.
  base:
    command: sh -l -c "bundle && bundle exec puma -v -b tcp://0.0.0.0:3000"
  web:
    command: sh -l -c "bundle && bundle exec puma -v -b tcp://0.0.0.0:3000"
  worker:
    command: sh -l -c "bundle && bundle exec sidekiq"

What you then do is run docker compose using this command:

docker-compose -f docker-compose.yml -f docker-compose.bundle.yml up

Clone this wiki locally