Merge pull request #472 from JeffersonLab/nsj_docs

Nsj docs

src/plugins/Calibration/CDC_TimeToDistance/OldFitScripts/README.md

@@ -0,0 +1,38 @@
+# Locating the Data
+At the time of writing, the CDC_TimeToDistance plugin is included in the offline monitoring at JLab. The general location is
+
+/cache/halld/offline_monitoring/RunPeriod-YYYY-MM/verXX/hists/hists_merged
+
+The instructions below assume that the scripts are run at JLab where the offline_monitoring was run with the plugin.
+
+# Setting up the directory
+Use a directory outside of sim-recon for checking the results of the CDC_TimeToDistance plugin. Inside this directory, make a symlink to the merged histograms from the offline monitoring:
+
+ln -s /cache/halld/offline_monitoring/RunPeriod-YYYY-MM/verXX/hists/hists_merged hists_merged
+
+This will make it easy to access the files and will work properly with the script.
+
+Additionally, making symlinks to the scripts will also make life easier.
+
+ln -s $HALLD_RECON_HOME/src/plugins/Calibration/CDC_TimeToDistance/FitScripts/run.py run.py
+
+ln -s $HALLD_RECON_HOME/src/plugins/Calibration/CDC_TimeToDistance/FitScripts/FitTimeToDistance.C FitTimeToDistance.C
+
+ln -s $HALLD_RECON_HOME/src/plugins/Calibration/CDC_TimeToDistance/FitScripts/display.py display.py
+
+# Running the scripts
+With everything prepared as described above, run the python script `run.py`
+
+python run.py
+
+This will create a set of output directories for various PNG images that will be useful to check the calibration. It will also create a directory containing the text files needed for updating CCDB. Once the directories are created, the script finds all of the runs in the hists_merged directory and executes FitTimeToDistance.C on each of the files.
+
+Once this completes, each new directory should be filled with images or text files. To view the relevant plots on a single canvas for all runs, use `display.py`
+
+python display.py
+
+This script assumes that the command `display <image>.png` works, which should be true on JLab machines.
+
+A single canvas will appear with 5 plots. Starting on the left of the top row, this is the result with the initial CCDB constants. Moving to the right, this is what it will look like with the newly calculated constants. The final plot in this row shows the difference between the before and after. The bottom row shows the residuals as a function of drift time with the final plot being a projection onto the residual axis.
+
+To advance to the next run, close the window and hit enter in the terminal.

src/plugins/Calibration/CDC_TimeToDistance/README.md

@@ -1,38 +1,46 @@
-# Locating the Data
-At the time of writing, the CDC_TimeToDistance plugin is included in the offline monitoring at JLab. The general location is
-
-/cache/halld/offline_monitoring/RunPeriod-YYYY-MM/verXX/hists/hists_merged
-
-The instructions below assume that the scripts are run at JLab where the offline_monitoring was run with the plugin.
-
-# Setting up the directory
-Use a directory outside of sim-recon for checking the results of the CDC_TimeToDistance plugin. Inside this directory, make a symlink to the merged histograms from the offline monitoring:
-
-ln -s /cache/halld/offline_monitoring/RunPeriod-YYYY-MM/verXX/hists/hists_merged hists_merged
-
-This will make it easy to access the files and will work properly with the script.
-
-Additionally, making symlinks to the scripts will also make life easier.
-
-ln -s $HALLD_RECON_HOME/src/plugins/Calibration/CDC_TimeToDistance/FitScripts/run.py run.py
-
-ln -s $HALLD_RECON_HOME/src/plugins/Calibration/CDC_TimeToDistance/FitScripts/FitTimeToDistance.C FitTimeToDistance.C
-
-ln -s $HALLD_RECON_HOME/src/plugins/Calibration/CDC_TimeToDistance/FitScripts/display.py display.py
-
-# Running the scripts
-With everything prepared as described above, run the python script `run.py`
-
-python run.py
-
-This will create a set of output directories for various PNG images that will be useful to check the calibration. It will also create a directory containing the text files needed for updating CCDB. Once the directories are created, the script finds all of the runs in the hists_merged directory and executes FitTimeToDistance.C on each of the files.
-
-Once this completes, each new directory should be filled with images or text files. To view the relevant plots on a single canvas for all runs, use `display.py`
-
-python display.py
-
-This script assumes that the command `display <image>.png` works, which should be true on JLab machines.
-
-A single canvas will appear with 5 plots. Starting on the left of the top row, this is the result with the initial CCDB constants. Moving to the right, this is what it will look like with the newly calculated constants. The final plot in this row shows the difference between the before and after. The bottom row shows the residuals as a function of drift time with the final plot being a projection onto the residual axis.
-
-To advance to the next run, close the window and hit enter in the terminal.
+# CDC time to distance calibrations
+
+These use the histograms from the plugin CDC\_TimeToDistance.  
+One evio file is sufficient.
+The scripts referred to here are in the subdirectory scripts.
+
+ttod\_setup.sh copies the scripts into the current directory and creates a link to the calibration histograms
+ttodfit.C fits the time to distance parameters used to estimate the track DOCA from drift time. 
+runcalibs.py runs ttodfit.C over all root files in the subdirectory hists.
+
+gaincalibs.py creates the following directories, containing one file per run:
+- Before - fit function contours for the parameters stored in the root file
+- After - fit function contours for the new parameters obtained from the calibration
+- Combined - overlay of old and new fit function contours
+- ResVsT - 2D plot of track fit residual vs drift time
+- Proj - track fit residuals, fitted with a single Gaussian
+- Monitoring - all of the above, on one canvas 
+- ccdb - file containing new calibration constants to be added to ccdb
+
+It also creates the following files:
+- res.txt  - run number and mean and sigma of the fitted Gaussian
+- res6.txt - as above, for tracks with FOM 0.6+
+- res9.txt - as above, for tracks with FOM 0.9+
+- addtoccdb.sh - bash script to add the data to ccdb.
+- badfiles.txt - list of files for which calibrations failed (may not be present)
+
+plotres.py generates plots of the mean and sigma of the Gaussian fitted to the track residuals
+scanfitresults.py scans the fit results and prints a list of any with mean larger than 8μm 
+
+**To run the calibrations:**
+
+1. Make a new directory and cd into it
+2. Run the script ttod_setup.sh and specify the calibration launch results directory on the command line, eg
+```sh
+$HALLD_RECON_HOME/src/plugins/Calibration/CDC_TimeToDistance/scripts/ttod_setup.sh /work/halld/data_monitoring/RunPeriod-2019-11/mon_ver23/rootfiles/
+```
+3. Run the calibration script:
+```sh
+python runcalibs.py
+```
+4. Inspect the results
+5. Load the new gain constants into ccdb:
+```sh
+chmod +x addtoccdb.sh
+./addtoccdb.sh
+```

src/plugins/Calibration/CDC_TimeToDistance/scripts/plotres.py

@@ -0,0 +1,53 @@
+
+import pandas as pd
+import numpy as np
+import matplotlib.pyplot as plt
+
+df = pd.read_fwf('res.txt',colspecs=[(0,5),(15,22),(29,36)],header=None)
+
+#print df.head()
+print("Statistics:")
+print(df.describe())
+
+run = df[0]
+mean = df[1]
+sigma = df[2]
+
+plt.plot(run,10000*mean, 'bo')
+
+
+plt.title('Mean of single gaussian fitted to track fit residuals')
+plt.xlabel('Run number')
+plt.ylabel(r'Mean of residuals (um)')
+
+
+plt.show()
+
+plt.plot(run,10000*sigma, 'bo')
+
+
+plt.title('Sigma of single gaussian fitted to track fit residuals')
+plt.xlabel('Run number')
+plt.ylabel('Sigma of track fit residuals (um)')
+
+plt.show()
+
+
+#plt.plot(x,y, 'bo', x, poly1d_fn(x), 'r-')
+
+#coef = np.polyfit(x,y,1)
+
+#fpoly1 = np.poly1d(coef)
+
+#plt.plot(x,y,'bo')
+
+
+
+#plt.savefig("resid_mean.png")
+
+
+
+#plt.plot(x,y,'bo',fpoly1(x),'--k')
+#plt.show()
+
+

src/plugins/Calibration/CDC_TimeToDistance/scripts/runcalibs.py

@@ -0,0 +1,84 @@
+import os
+import subprocess
+import glob
+
+testing = 0  # just process one file
+
+
+dirs = ['Before', 'After', 'Combined', 'ccdb', 'Monitoring', 'Proj', 'ResVsT']
+for d in dirs:
+    if not os.path.exists(d):
+        os.makedirs(d)
+
+#rundirlist = subprocess.check_output(["ls", "hists_merged"]).splitlines()
+#for rundir in rundirlist:
+
+#    print(rundir)
+    rundir = ""
+
+#filelist = subprocess.check_output(["ls"]).splitlines()
+
+if os.path.exists("hists"):
+  histdir = "hists"
+elif os.path.exists("hists_merged"):
+  histdir = "hists_merged"
+else:
+  exit("Cannot find hists or hists_merged")
+
+
+
+#filelist = subprocess.check_output(["ls", "hists_merged/" + rundir]).splitlines()
+filelist = subprocess.check_output(["ls", histdir]).splitlines()
+
+filesdone = 0
+
+for file in filelist:
+
+        print(file)
+
+        runfile = histdir + "/" + rundir + "/" + file
+
+
+        if os.path.isdir(runfile):
+
+          if len(os.listdir(runfile)) == 0:
+            fbad = open("emptydir.txt", "a")
+            fbad.write(runfile)
+            fbad.write("\n")
+            fbad.close()
+            continue
+
+          newlist = subprocess.check_output(["ls", runfile]).splitlines()
+
+          runfile = runfile + "/" + newlist[0]
+
+
+        print(runfile)
+
+        scriptname = "ttodfit.C(1,1)"
+
+        print(scriptname)
+
+        # look to see how many calib files have been made already
+        nfiles = len(os.listdir("ccdb"))
+        print(nfiles)
+
+        # root -l -b -q FitTimeToDistance.C(runfile)"
+#        subprocess.call(["root", "-l", "-b", "-q", scriptname])
+        subprocess.call(["root", "-l", "-b", "-q", runfile, scriptname])
+
+        # check whether the script completed & made a new file of calib consts
+
+        print(len(os.listdir("ccdb")))
+
+        if len(os.listdir("ccdb")) == nfiles:
+          fbad = open("badfiles.txt", "a")
+          fbad.write(runfile)
+          fbad.write("\n")
+          fbad.close()
+
+        filesdone = filesdone + 1
+
+        if testing == 1:
+
+           break

src/plugins/Calibration/CDC_TimeToDistance/scripts/scanfitresults.py

@@ -0,0 +1,8 @@
+import math
+
+f = open("res.txt", "r")
+for x in f:
+  resid = x.split()[3]
+  runfile = x.split()[0]
+  if abs(float(resid)) > 0.0008:
+    print(runfile + " " + resid)

src/plugins/Calibration/CDC_TimeToDistance/scripts/ttod_setup.sh

@@ -0,0 +1,37 @@
+#!/bin/sh
+#
+# Script to set up links for CDC_TimeToDistance calibration constant generation
+# in current directory
+#
+# Arg 1 should be the location of the TimeToDistance histograms
+#
+# eg calibfiledir="/cache/halld/RunPeriod-2018-08/calib/ver01/hists"
+
+
+if [ $# -ne 1 ]; then
+  echo
+  echo Usage: ttod_setup.sh calibfiledir
+  echo
+  echo eg ttod_setup.sh /cache/halld/RunPeriod-2018-08/calib/ver01/hists
+  echo
+fi
+
+calibfiledir=$1
+
+
+if [ ! -d $1 ]; then
+  echo Cannot find dir $calibfiledir
+  exit
+fi
+
+echo Linking subdir hists to $1 
+`ln -s $1 hists`
+echo
+
+
+
+cp -iv $HALLD_RECON_HOME/src/plugins/Calibration/CDC_TimeToDistance/scripts/*.py .
+cp -iv $HALLD_RECON_HOME/src/plugins/Calibration/CDC_TimeToDistance/scripts/ttodfit.C .
+
+
+