Debugging pfcon Containerized Services

Debugging pfcon Containerized Services

Abstract

This page documents a workflow that can be useful in debugging pfcon, pfioh, and pman within their respective containers.

Introduction

Debugging the ancillary services pfcon, pfioh, and pman can be particularly difficult. This workflow documents how to best start the services and then restart pfioh and pman in interactive mode so as to both debug directly if necessary and also to immediately see any generated outputs from those services.

In order to fully debug, the source code of the particular service needs to be volume mapped into its respective container in the correct place. This necessitates small transitory changes to the docker-compose.yml.

Preconditions

Map the service source repo tree into the relevant container

Let's assume we want to debug all the relevant services. Let's also assume that the pman and pfioh repos have been cloned at the same directory level as pfcon.

In the docker-compose.yml of pfcon add the following line to the volumes: section of the pman_service:

pman

      - ../pman/pman/pman.py:/usr/local/lib/python3.6/dist-packages/pman/pman.py

and for the pfioh_service:

pfioh

    - ../pfioh/pfioh/pfioh.py:/usr/lib/python3.6/site-packages/pfioh/pfioh.py

Obviously the paths to pman.py and pfioh.py need to be correct relative to the docker-compose.yml. Note that though these service comprise many files (as witnessed in their respective repos), for the overwhelming majority of debug cases you will be interested in only one file per service as shown above.

Run the dockerized services

From the pfcon repo dir, run (add a -d for debugging verbosity)

./unmake ; ./make

or, if you have local docker images:

./unmake; ./make local

Debug setup

Create an "environment"

The simplest way to debug is to open four terminals, say in a grid. In each terminal cd to the pfcon source repo.

Each terminal will be used to run a specific service.

Terminal 1	Terminal 2
cd <pfconrepo>	cd <pfconrepo>

Terminal 2	Terminal 3
cd <pfconrepo>	cd <pfconrepo>

Stop services and restart in interactive mode

Now, do

Terminal 1	Terminal 2
unmake ;
sudo rm -fr FS ;
rm -fr FS ;
make -d	# ( do nothing )

Terminal 2	Terminal 3
# ( do nothing )	# ( do nothing )

for ease of copy/pasting, in Terminal 1 do

unmake ; sudo rm -fr FS; rm -fr FS ; make -d

Wait for execution in Terminal 1 to stop after pfcon has been restarted in interactive mode by the make script. Now, we can restart pfioh and pman (in that order -- see the next section since host entries in the pfcon container are needed).

Note that the local in the below is only for cases where you have actually built local docker images (see the Dockerfile) of each repo.

In many cases, esp with a source file mapping, the local is not needed.

Restart pman and pfioh in interactive mode

Now, do

export HOST_IP=$(ip route | grep -v docker | awk '{if(NF==11) print $9}')

Create a host entry for pfioh_service and pman_service

The idea behind this debugging approach is to first start services with docker-compose and then restart then in interactive mode. Doing this, however, breaks the management of docker-compose and requires some tweaking. One of these tweaks is to add an entry in the /etc/hosts of the chris_dev_backend container for a host pfcon_service. For example,

    10.23.130.106    pfioh_service
    10.23.130.106    pman_service

where the IP must be the host IP of the machine you are running on. If you are using the standard FNNDSC bash environment with appropriate aliases, you could do (use appropriate IP for your host):

    dkee pfcon "echo '${HOST_IP}   pfioh_service' >> /etc/hosts"
    dkee pfcon "echo '${HOST_IP}   pman_service' >> /etc/hosts"
    dkee pfcon "cat /etc/hosts"

where

alias dkl='docker ps -a'
function dkee {
        NAME=$1
        CMD="$2"
        ID=$(dkl | grep $NAME | head -n 1 | awk '{print $1}')
        docker exec $ID  bash -c "$CMD"
}

and dkee is a function that "executes" and "exits" some command within a container regex.

Terminal 1	Terminal 2
#(leave this as is)	#(leave this as is)

Terminal 2	Terminal 3
# First do this	# And then do this
make -r pfioh local	make -r pman local

NB only use local if you are running locally built images, other wise do not use it!

Call pfcon

hello

Assuming satisfied preconditions, let's say hello to pfcon. It will in turn ask each of pfioh and pman hello and return the response.

export HOST_IP=$(ip route | grep -v docker | awk '{if(NF==11) print $9}')
pfurl --verb POST --raw \
      --http ${HOST_IP}:5005/api/v1/cmd \
      --httpResponseBodyParse \
      --jsonwrapper 'payload' \
      --msg \
'{  "action": "hello",
    "meta": {
                "askAbout":     "sysinfo",
                "echoBack":      "Hi there!",
                "service":       "host"
            }
}' --quiet --jsonpprintindent 4

--30--