Skip to content

Salsa

Jump to bottom Edit New page
Thomas Quinn edited this page Mar 16, 2017 · 6 revisions

Also see the older documentation on the Salsa Page

Salsa is a parallel, interactive analysis tool. It's composed of two pieces, a server running in parallel on the back end, and a Java client running on your local machine. Our current implementation uses some script hackery to automagically start and connect to the server when you fire up the client.

Table of Contents

Obtaining Salsa

Salsa can be downloaded from the Nchilada software distribution site.

The lastest version can be checked out of our git repository with 

git clone git://charm.cs.illinois.edu/cosmo/salsa.git
git clone git://charm.cs.illinois.edu/cosmo/utility.git
.

Building Salsa

Also see the README file in the ResolutionServer directory of the salsa distribution.

Prerequisite software

  • Charm++ (version 6.2.0 or later) Or use the binary from the nightly build that matches your system.
  • boost (1.31 or greater)
  • Java JDK 1.5.X or greater. Be sure that the development environment is installed. E.g. the java-1.6.0-openjdk-devel package.
  • Apache Xerces-C++ XML parser
  • Python (2.2 or greater). This is preinstalled on most Linux distributions.

Build Resolution Server

  • Be sure python and java development environments are installed for your distribution. For example, if you are on an RPM based linux system, type "rpm -q python-devel" to check for the python development library.
  • Install Charm++.
Note that you will probably need at least version 6.2.0 of Charm++. If you get a "Charmxi syntax error", during the Resolutionserver build, then you need a later version of Charm++.
  • Compile the "liveViz" module and python modules of Charm++. This is easiest done with
build ChaNGa ...

For example, on a multicore 32 bit workstation, use 

build ChaNGa net-linux -j2 -O2

Or, on a 64 bit workstation, use 

build ChaNGa net-linux-x86_64 -j2
  • Install xerces-c. For Ubuntu systems the package is called libxerces2-dev
  • If it is not available on your system, install and build the Apache Xerces C++ XML Parser version 2.8 in the parent directory. Be careful to follow the build instructions given on the XercesC website. XercesC provides a wrapper script which should generally be used instead of a direct configure. Here is an example procedure for a linux machine, beginning in the xercesc directory. If the correct build instructions are not followed, then XercesC will not be properly installed and Salsa will not work.
cd src/xercesc; ./runConfigure -p linux; make

If you had to build xerces-c yourself, set the XERCESCROOT environment variable to point at the root of the XercesC directory. E.g. for bash

export XERCESCROOT=/path_to_salsa/salsa/xercesc

  • Install (but you don't need to build) the Boost C++ library (http://www.boost.org/) in the parent directory. The Ubuntu package is called libboost-dev.
  • Set the environment variable "CHARM_DIR" to point at the directory in which the source of charm is located.
  • Type ./configure in the salsa/ResolutionServer directory. This should find charmc (if you had set CHARM_DIR correctly), a version of python, and xerces-c if all of your environment variable were set correctly.






  • Type make in the ResolutionServer directory.



Build Resolution Server on Blue Waters






Setting up a python build on a CrayXE is tricky.  Here are the steps on Blue Waters:







    

  • Load the python module with module load bwpy.
    • 

  • Perform a charm++ build, pointing at the python directory:
    • 


./build ChaNGa gni-crayxe hugepages --basedir=$BWPY_DIR/usr -j4 -O2










    

  • Load the boost module with module load boost

    • 

  • Configure ResolutionServer in the ResolutionServer subdirectory with CPPFLAGS=-I$BWPY_DIR/usr/include LDFLAGS=-L$BWPY_DIR/usr/lib ./configure

    • 

  • Now make ResolutionServer.  There are a few issues.  One is that PYTHON_INCLUDE_DIR might not be set correctly in the Makefile.  It should be set to $(BWPY_DIR)/usr/include/python2.7.  Then linking needs to be dynamic to avoid an odd shared library bug: add "-Wl,-rpath,$(BWPY_DIR)/usr/lib -dynamic" to LDFLAGS.  Also, the link step may still fail because of a missing library -lutil and -ldl.  Add these to the LIBS macro in the Makefile.
    • 




Build Interactive Client






The java client is in the separate directory InteractiveClient.  Make sure the "CHARM_DIR" environment variable points to the charm source directory (there are some java class libraries that are needed there) and type make in the InteractiveClient directory.  This should build the Salsa.jar Java archive.



The make will download .jar files and .so libraries for the GL rendering (jogl). These need to be installed in a directory pointed at by "LD_LIBRARY_PATH" so they can be loaded when running the java client.





Build Command Line Client






A command line client, taco is located in the CommandClient directory.  After running "configure", edit the Makefile to be sure that the "CHARM_DIR" variable is pointing at the charm source directory.  The build will need the readline development library for the command line editing features.  Type make to build the client.





Starting Salsa on a Workstation






Although Salsa is designed as a parallel analysis application, it can be run on a workstation, or a group of workstations on a network.  In order to do this, one of the "net" versions of charm++ needs to be used for the build.  For example, for a linux workstation, build charm++ with

build charm++ net-linux


in the charm directory.  When ResolutionServer is built, a charmrun executable is produced which does the job of starting the server up on multiple (or just one) workstations.  This executable assumes that the ssh server is running on the target workstation.  For example, on a Redhat/Fedora workstation the openssh-server package must be installed.  It is also useful to have your ssh keys set up in order to avoid typing your password all the time.  See SSH Keys and the SSH with Keys HOWTO for information on how to do this.  Or, if you're just running on your own desktop and don't want to mess with this, add "++local" after the charmrun command below.


An example command line is

charmrun ResolutionServer ++server +p 1 simlist


which would start ResolutionServer on one processor on the machine specified in the nodelist or ~/.nodelist.  An example of this file is

group main ++shell /usr/bin/ssh -X
host mymachine


The charmrun command will report which port it is listening on.  E.g.

ccs: Server IP = 127.0.0.1, Server port = 32991


This information can now be used to start the Java interactive client:

java -jar Salsa.jar localhost 32991 tipsyfile


All this is somewhat automated in the Run_Salsa.sh script in the ResolutionServer directory. Alternatively, Salsa can be run directly by 

Salsa.sh <hostname> <number of CPUs> <simulationfile> 






Python execution






The server is actually running a python interpreter into which the client is feeding commands.  If you want to access it directly, use the Write and Execute Code menu item.  Be aware that for any complicated python, the stack size on the server needs to be increased from its default.  The ResolutionServer will need to be started with

charmrun ++server ResolutionServer +p 1 +stacksize 1000000 simlist






Starting Salsa on a Cluster








Starting ResolutionServer






ResolutionServer is a parallel program, and so starting it depends on the nature of your parallel machine.  Here is a script to run ResolutionServer as an MPI job on the Student Cluster (Robert).  This script is available in ~trq/bin.x86_64/salsa_bsub on robert.

#!/bin/sh
# Usage:
# salsa_bsub nproc
NPROC=$1
echo set up tunnel with \" ssh -v -L16512:XXX:16512 robert\" replacing XXX
echo with IP address given below.
bsub -q short -n $NPROC -I << EOF
#!/bin/sh -x
rm -f nodelist
for i in \$LSB_HOSTS ; do
        echo  \$i >> nodelist
done
NPROC=\`wc nodelist | awk '{print $1;}'\`
mpirun_ssh -hostfile nodelist -np $NPROC /home/trq/bin.x86_64/ResolutionServer ++server ++server-port 16512 simlist
EOF

 
The crux of the script is the penultimate line where an MPI job started using all the nodes in nodelist.  The ++server ++server-port 16512 options tell it to start listening for commands on the local internet port 16512.   simlist can be a file listing available simulations, but if it doesn't exist, it is ignored.




Tunnelling





Unfortunately, the internet port that ResolutionServer is listening to is buried deep within the cluster and is inaccessible to the outside world.  Hence one needs to set up a "tunnel" to access it.  The concept is that a separate port is setup (say, on your workstation) such that any connections to this port are forwarded to the port inside the cluster via an ssh connection to the cluster front end.  The command to set this up would be, e.g.,

ssh -v -L16512:10.127.1.235:16512 robert


If this is run on your workstation, then ssh listens for connections on port 16512 on your workstation, and forwards any connections through robert to port 16512 on the host "10.127.1.235" (an internal node on robert).  The host number is reported by ResolutionServer when it starts so that you know which internal node to connect to.


That's the hard part.  Any ideas for automating this process would be greatly appreciated.





Interactive Client





With the tunnelling established, the java client can be started on your workstation:

java -jar Salsa.jar localhost 16512 /path-on-robert/simulation_file


This connects to the 16512 port on your local machine, which gets forwarded by the tunnelling to the parallel server.  The simulation file (tipsy or NChilada format) needs to be specified with respect to where ResolutionServer is run on robert.




Starting Salsa on a Supercomputer






See Research:SalsaBuildInstructions for details on particular supercomputers.





Using the Interactive Client






The java program will open a window and request an image from the server.  If all goes well, an image will be displayed.  The image can be rotated and zoomed as in Tipsy.  New colorings can be defined in the Coloring item of the Manage drop-down menu.  Ranges of individual attributes can be examined in the Attributes entry of the Manage menu. Subsets can be selected using the shift button and dragging, similar to Tipsy. It is helpful to monitor the computer you are using (type 'top' in the command line), as some functions such as zooming in Salsa can take a few seconds. Do one thing at a time.




Using the Graphics Window






Once the simulation is loaded, you can rotate, zoom in, zoom out, select sphere, select box, use the ruler, and refresh by clicking on the buttons along the right of the image of the simulation. After a button has been selected, click on the image to zoom, rotate, etc. 



To zoom, click on the area of the image you want to center the zooming in or out. To rotate, click and drag the image. Note that unlike in tipsy, rotations will not reset the zoom scale.  If the potential attribute is defined, the view will rotate around the particle in the image with the lowest potential. Alternatively, rotations can be more precisely done using the three sliders along the bottom, which will rotate the simulations along each of the three axes independently.



To select box, click and drag on the image, creating a 2D box to be selected. Upon release, the simulation will be rotated by 90 degrees so that the third dimension of the box can be specified, again with a left mouse click and drag. Similarly, to create a sphere, click and drag to create a circle on the image.  The initial mouse position will define a center of the sphere, and the release point will set the boundary.  The third dimension of the center of the sphere will be defined by the center of rotation.



To use the ruler, click on the two places you want to measure the distance between. The distance will be printed out in the terminal where Salsa was launched.



To highlight any groups, click on the groups in the Marked groups list in the Groups menu (press control during click to select multiple groups for highlighting), and click the Aplly mark button beside the list. The image will refresh automatically, but may take a few seconds. To unmark a group, you can deselect from the Marked groups list by clicking while holding the control button, and then clicking the Apply mark button again.





Using Python






Under the Write and execute code menu item a window can be popped up into which python can be entered and executed on the server.  The charm module contains methods for operating on the data.
print dir(charm) will give a list of these methods.  print charm.__doc__ will give brief documentation.  More complete documentation can be found in Research:Salsa Python



A particularly interesting method is charm.runLocalParticleCode(string).  This will cause the method localparticle to be executed with each particle as an argument.  The string argument should be python code containing a definition of localparticle that will manipulate the particle attributes.  E.g.

charm.runLocalParticleCode("def localparticle(p):\n   p.mass = 2*p.mass")


will double the mass of every particle.


This method can also be accessed through the Write and execute local code menu.  In that case, the def statement is implied, and the particle data can be address through the variable p.  E.g., the above can be accomplished by typing p.mass = 2*p.mass in the LocalParticle Code text box.





Taco: a command line client






The CommandClient directory of the Salsa distribution contains taco, a command line "shell" client that allows you to type python at the ResolutionServer.  This may make it easier to perform scripting operations with the ResolutionServer.  Note that you can connect to the same ResolutionServer that you started with Salsa, so you can visualize and do scripting at the same time.  The usage is:

taco hostname port




Note that the PYTHONPATH environment variable needs to be specified to import modules and run python scripts to do analysis in taco. For example, in tcsh:



setenv PYTHONPATH .:/astro/users/trq/nchil/src/salsa/ResolutionServer/lib






Tipsy Python Module






Some of the familiar commands from tipsy are implemented in the tipsy module.  Access these with

import tipsy


The functions can be called as (e.g.) tipsy.boxstat(...).  Docstring documentation is available via (e.g.) help(tipsy.boxstat).  Available functions include:











    

  • boxstat
    • 

  • markbox
    • 

  • markgal
    • 

  • profile
    • 

  • rotcur
    • 

  • setbox
    • 

  • setsphere
    • 

  • writemark
    • 




Queso Python Module






Queso is a module that was developed by Eric Carlson to perform some of the analysis needed by the Aquila code comparison project.  Access these functions with

from queso import *


This provides the following packages:

























    

  • quesoConfig:This is the configuration module, which should be checked any time a new simulation is being analyzed.  In particular, set quesoConfig.kpcunit and quesoConfig.msolunit to match your simulation.
    • 

  • vectormath: Provides basic vector operations in salsa.
    • 

  • glob:       Provides functions related to global quantities (as opposed to profiles).  Currently implemented methods include:  
      

    • Finding a group center via center of mass or potential min
      • 

    • Determine the virialRadius and create a virial sphere group
      • 

    • Get the specific angular momentum vector of a group and the first associated Euler angles.
      • 

    • Transform position and velocity into a galactic coordinate system aligned with an arbitrary groups angular momentum vector.
      • 

    • Find the stellar half-mass-radius of a group.
      • 

    • Find the mean mass weighted metallicity of a group
      • 

    • Find the moment of inertia tensor of a group
      • 

    • Calculate the distribution of jz/jcirc for a group
      • 

    

    • 

  • profile:  Provides some additional profiling capabilities that are not included in the tipsy profile package.  Currently implemented methods include:  
      

    • Compute cylindrical SFR profile aligned with the galaxy.
      • 

    • Compute star formation history (age vs formation rate)
      • 

    • Compute spherical metallicity profile.
      • 

    

    • 

  • export:   Provides functionality to export salsa groups to tipsy-ascii files.  Currently implemented methods include:  
      

    • Write group to tipsy ascii file
      • 

    • Export a cube of specified radius that is aligned with the galactic coordinate system (for face-on/edge-on projections etc...).  Can be used in conjunction with tipsy's 'vista' command to generate density projections.
      • 

    

    • 




Getting Help






Bugs and feature requests can be submitted to the issue tracker on our Github site.



Also check out our list of Research:Salsa Issues for common errors when running Salsa.





Development






Salsa is being actively worked on.  See Research:Salsa Development for details.

              


              

                
                   Add a custom footer
              

          



  
          

            

              

                
                   Add a custom sidebar
              


            
Clone this wiki locally