Skip to content

Latest commit

 

History

History
280 lines (183 loc) · 8.18 KB

readme.md

File metadata and controls

280 lines (183 loc) · 8.18 KB
Raw

Create a THREDDS AWS Nexrad VM on Jetstream

Create a THREDDS VM on Jetstream

Create an m1.large VM with the Jetstream OpenStack API. Work with Unidata system administrator staff to have this VM's IP address resolve to tds-nexrad.scigw.unidata.ucar.edu

Clone the science-gateway Repository

We will be making heavy use of the Unidata/science-gateway git repository.

git clone https://github.com/Unidata/science-gateway ~/science-gateway

Build the AWS Nexrad TDS Docker Container

From the ~/science-gateway/vms/thredds-aws directory:

docker build -t unidata/thredds-docker:<5-version> .

Start TDS With Docker and docker-compose

With the help of Docker and docker-compose, starting a VM with the TDS should be fairly easy. There are a few directories you will need to map from outside to within the container. See here to install Docker and docker-compose.

TDS Configuration

Download Configuration

mkdir -p ~/tdsconfig/
wget http://unidata-tds.s3.amazonaws.com/tdsConfig/awsL2/config.zip -O ~/tdsconfig/config.zip
unzip ~/tdsconfig/config.zip -d ~/tdsconfig/

Cache Clearing

In order to not have the TDS fill up with data, the cache has to be periodically cleared. Edit the ~/tdsconfig/threddsConfig.xml and add:

<DiskCache>
  <alwaysUse>true</alwaysUse>
  <scour>1 hour</scour>
  <maxSize>1 Gb</maxSize>
</DiskCache>

Supply Contact and Host Information in threddsConfig.xml

Edit the ~/tdsconfig/threddsConfig.xml to supply contact and host institution by filling out the contact and hostInstitution XML elements. For example:

<contact>
  <name>THREDDS Support</name>
  <organization>Unidata</organization>
  <email>support-thredds@unidata.ucar.edu</email>
</contact>
<hostInstitution>
  <name>Unidata</name>
  <webSite>http://www.unidata.ucar.edu/</webSite>
  <logoUrl>https://ral.ucar.edu/sites/default/files/public/images/project/Unidata_logo_vertical_400x400_alpha.png</logoUrl>
  <logoAltText>Unidata</logoAltText>
</hostInstitution>

TDS log Directories

Create log Directories

You will need Apache Tomcat and TDS log directories:

mkdir -p /logs/tds-tomcat/
mkdir -p /logs/tds/

SSL Certificate

In the ~/science-gateway/vms/thredds-aws/files/ directory, generate a self-signed certificate with openssl (or better yet, obtain a real certificate from a certificate authority).

openssl req -new -newkey rsa:4096 -days 3650 -nodes -x509 -subj \
  "/C=US/ST=Colorado/L=Boulder/O=Unidata/CN=jetstream.unidata.ucar.edu" \
  -keyout ~/science-gateway/vms/thredds-aws/files/ssl.key \
  -out ~/science-gateway/vms/thredds-aws/files/ssl.crt

Ports 80, 443 and 8443

Open port 80 on the THREDDS VM via OpenStack. Port 80 requests will be forwarded to 8080 inside the THEREDDS Docker container. In addition, open ports 443 and 8443 for SSL and communication from the TDM.

docker-compose.yml

Based on the directory set we have defined, the docker-compose.yml file that looks like:

###
# THREDDS
###
version: '3'

services:
  thredds-production:
    image: unidata/thredds-docker:5.5-SNAPSHOT
    container_name: thredds
    restart: always
    ports:
      - "80:8080"
      - "443:8443"
      - "8443:8443"
    volumes:
      - /logs/tds-tomcat/:/usr/local/tomcat/logs/
      - /logs/tds/:/usr/local/tomcat/content/thredds/logs/
      # ssl certs, keys not in version control, see readme.md
      - ./files/tomcat-users.xml:/usr/local/tomcat/conf/tomcat-users.xml
      - ./files/tdsCat.css:/usr/local/tomcat/webapps/thredds/tdsCat.css
      - ./files/folder.gif:/usr/local/tomcat/webapps/thredds/folder.gif
      - ./files/index.jsp:/usr/local/tomcat/webapps/ROOT/index.jsp
      # HTTPS
      - ./files/keystore.jks:/usr/local/tomcat/conf/keystore.jks
      - ./files/server.xml:/usr/local/tomcat/conf/server.xml
      - ./files/web.xml:/usr/local/tomcat/conf/web.xml
      # AWS TDS Nexrad server
      - ~/tdsconfig/:/usr/local/tomcat/content/thredds/
      - ~/files/credentials:/usr/local/tomcat/.aws/credentials
    env_file:
      - "compose${THREDDS_COMPOSE_ENV_LOCAL}.env"

THREDDS Environment Variable Parameterization

You can provide additional THREDDS parameterization via the compose.env file referenced in the docker-compose.yml file.

### THREDDS related environment variables

# TDS Content root

# Paremeterization of the TDS_CONTENT_ROOT_PATH is probably not needed here
# since paremeterization can already achieved through the docker-compose.yml but
# here it is anyway

TDS_CONTENT_ROOT_PATH=/usr/local/tomcat/content

# The minimum and maximum Java heap space memory to be allocated to the TDS

THREDDS_XMX_SIZE=4G

THREDDS_XMS_SIZE=4G

# See https://github.com/Unidata/tomcat-docker#configurable-tomcat-uid-and-gid

TOMCAT_USER_ID=1000

TOMCAT_GROUP_ID=1000

Start the TDS

Once you have done the work of setting up THREDDS related directories in the way you like,

docker-compose up -d

to start the TDS

Navigate to the TDS

In a web browser, navigate to https://tds-nexrad.scigw.unidata.ucar.edu/thredds/catalog.html to see if is running.

Blocking IPs That Are Filling up the Cache

You will sometimes find that data scraper bots are overloading the TDS Radar server which leads to the cache filling up the disk. One way to mitigate this is to block IPs via iptables. To find the offending IP ranges, navigate to the /logs/tds-tomcat looking for user agents like "spider", "oc4", etc. in the Tomcat access logs.

grep -i -h -E "oc4|rain|spider"  access*  | awk '{print $1}' | sort | uniq | awk -F "." '{print $1 "." $2}' | sort -n | uniq

Work with Unidata system administration staff to block the IP ranges with something like the snippet below. Note that the rule is being inserted into the DOCKER-USER chain with the -I option which is important to get iptables to work with the THREDDS Docker container.

sudo iptables -I DOCKER-USER -s xx.xxx.0.0/16 -j DROP
# etc.

To see how these rules take affect you can:

sudo iptables -L DOCKER-USER -n -v

which will yield something like:

Chain DOCKER-USER (1 references)
 pkts bytes target     prot opt in     out     source               destination
22404 1344K DROP       all  --  *      *       xx.22.0.0/16         0.0.0.0/0
 134K 8044K DROP       all  --  *      *       xx.128.0.0/16        0.0.0.0/0
 270K   16M DROP       all  --  *      *       xx.249.0.0/16        0.0.0.0/0
 265K   16M DROP       all  --  *      *       xx.225.0.0/16        0.0.0.0/0
30892 1606K DROP       all  --  *      *       xx.199.0.0/16        0.0.0.0/0
60216 3613K DROP       all  --  *      *       xx.204.182.0/24      0.0.0.0/0
  50M  107G RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0