Documentation for PHENIX Simulation Software at Vanderbilt

Contact Person: Charles F. Maguire
WWW location http://www.hep.vanderbilt.edu/~maguirc/simulation/simulation.html
Creation Date: June 14, 2004
Last Revision by Maguire: September 8, 2008: Note that ACCRE is no longer available

Introduction

The purpose of this WWW site is to document the PHENIX simulation software system available at Vanderbilt, specifically on the two CPU farms VUPAC and ACCRE. The ACCRE name stands for Advanced Computing Center for Research and Education (ACCRE) , and was formerly called the VAMPIRE system. It is with these two CPU farms that the Vanderbilt group has performed simulations for the PHENIX collaboration in the past.

The smaller VUPAC system [link is defunct, will be replaced in October], which has about 30 dual CPU nodes, can be thought as the Departmental computer farm and is maintained since 2006 by a structural biology group. This farm has its operating system at RHEL 4.0 , a close cousin of the Scientific Linux 4 which became the default OS at RACF in September 2007. All of the desktop machines used by the Vanderbilt PHENIX group are part of the VUPAC network.

The larger ACCRE system, has about 300 dual CPU Opteron nodes, and a comparable number of IBM PPC nodes. ACCRE is using the SL5 OS since March 2008. The binary files and libraries produced under SL4 are not interchangeable with such files produced on the SL5. Hence, for the time being, we at Vanderbilt would have to rebuild the entire PHENIX library set independently on the ACCRE platform. We used to do this for the SL3 OS when ACCRE was at SL3 and RCF was at SL4. However, we have not yet started to do SL5 builds at ACCRE. The details on how the rebuild was done for SL3 are contained elsewhere.

The PHENIX simulation process can be divided into four distinct steps

1. Events generation as specified by an analysis group or a PHENIX working group.
   
   
2. Tracking of generated events through the PISA simulation package which produces an output "hits" file.
   
   
3. Event reconstruction of the PISA output file into physics analysis files called "data summary tapes" (DST). The DSTs have various generations, so to speak, the most commonly used being the nanoDSTs. For the Central Arm spectrometer these nanoDSTs are called CNTs.
   
   
4. Embedding of the simulated nanoDST events into real nanoDST events for the purpose of checking the collision centrality dependence of the event reconstruction software.

Present Status of PHENIX Simulation Software at Vanderbilt

For work at Vanderbilt on the VUPAC cluster, we use copies of the SL4 software libraries built at RACF and copied down to Vanderbilt. As indicated previously, we can not now use the ACCRE system which is at the SL5 build. The possibility of doing an SL5 build at ACCRE will be explored.

Notes on the porting of PHENIX software from RACF to Vanderbilt

One thing which makes the porting of PHENIX binaries from RACF particularly difficult is that much of the PHENIX software system is infested with references to a logical volume /opt/phenix, which points to an AFS area at RACF. For both the VUPAC and the ACCRE clusters, their respective system administrators have vetoed the idea of putting in a special /opt/phenix logical volume. This is a reasonable approach since the administrators would not want to maintain a raft of special logical volumes for many different research groups. At RACF there are only three research groups: ATLAS, PHENIX, and STAR. It is difficult enough to maintain cyber security without each group asking for special variations on the general system.

For PHENIX software the /opt/phenix means that the building of personal analysis libraries with the make system uses setup scripts in /opt/phenix/bin which include that logical volume as a standard path. So at Vanderbilt we have to have our own copy of this logical volume, which for SL4 on VUPAC is at /home/phenix/phenixSL4/opt/bin. In turn there are many softlinks pointing to other scripts in other directories which also contain the non-portable /opt/phenix path. All those scripts need to be re-edited at Vanderbilt to use the /home/phenix/phenixSL4/opt path instead. This includes many levels of softlinks, for example the /home/phenix/SL4/opt/stow/autoconf-2.54/share/autoconf scripts.

Status of the four main components of PHENIX software for simulation analysis at Vanderbilt

1. The events generation software is the least dependent on operating system changes and external libraries. In fact, some of the events generation can be done internally inside of PISA itself. The main issue here is recompilation on the new OSes when necessary, and documentation. There are two main external event generators for PHENIX use. One is a global event generator called HIJING which is used as a background. The second is a single particle or single meson generator called EXODUS, developed by Ralf Averbeck. With either one of these event generators, one produces large size files (hundreds of MBytes) with hundreds or more HIJING events, or millions of EXODUS events, which files will then be read by PISA as input.
   
   
2. a) The PISA program works on the VUPAC system. The CONDOR batch processing system, which is used at RACF, has recently been installed on VUPAC. This CONDOR system replaces a previous, "home-made" batch queuing PERL script system which had worked on the VUPAC system. This PERL system required that you use the ssh-agent facility to forward your ssh passphrase. The notes on how to do that are contained at this location. However, this particular ssh-forwarding was disabled on the VUPAC. Hence, the CONDOR system, which we have not yet documented for our group's use, will have to be used for batch procession on VUPAC.
   
   b) PISA is not working at ACCRE, until further notice.
   
   
3. The PHENIX event reconstruction software works on the the VUPAC cluster, but not now on the ACCRE farms. The last major fix (September 1, 2008) was to get the PostgreSQL database system upgraded to 8.2 and working again at VUPAC. Ron Belmont, and Dillon Roach are the local experts on the PostgreSQL database [link is needed to describe database maintenance and updates at Vanderbilt].
   
   
4. The embedding software is a special component originally developed by a person named Jiangyong Jia who is at Columbia. Presently, Ron Belmont is the PHENIX expert on the embedding software. This embedding component is used to determine centrality dependent efficiency corrections for the analysis of heavy ion data taken with PHENIX. This component has not yet been imported to Vanderbilt.

Standard PHENIX software environment script at Vanderbilt

Just as with your computer accounts at RACF, your computer account at Vanderbilt should execute a software environment script which points to the local Vanderbilt disks which contain the PHENIX software libraries. In this way we are not dependent on any Internet connection to RACF. This standard environment script is called phnxVandy_setup.csh on both the VUPAC and the ACCRE computer farms. On the VUPAC system in your .cshrc login shell the following line

The important result of these environment setup scripts that will cause you to use the latest pro version of the PHENIX software libraries. The latest pro version is pro.82 which was copied down from RACF on August 12, 2008. Therefore, you can can test that you have the standard environment installed by giving the following two commands:

1) echo $LD_LIBRARY_PATH
On the VUPAC system this command should return for you (on one line)

./usr/lib:/home/phenix/phenixSL4/opt/lib:/usr/local/lib:/home/phenix/pro82-12Aug2008/lib:
/home/phenix/root-5.17.01/lib:/home/maguirc/cern/pro/lib

/home/phenix/pro82-12Aug2008/bin/pisa

Running Events Generation Software at Vanderbilt

Input events for a PISA simulation can be generated either internally, or externally as output files from stand alone programs run before PISA. One of the most popular of these stand alone programs is called EXODUS, written by Ralf Averbeck at Stony Brook. The EXODUS binaries and shared object library files are now part of the rebuild system, and should appear in the paths set up by the phnxVandy_setup.csh script.

The following are the steps needed to create an EXODUS output file suitable for input to PISA.

1. Create a working directory in your name on a scratch area (e.g. /scratch/YourLastName/ on ACCRE or /rhic1/YourLastName on VUPAC). Then create an exodusEvents subdirectory under your name directory. Change directory to the exodusEvents subdirectory.
   
   
2. In this subdirectory make the following softlink commands, where you use phenix on the VUPAC system and phnxreco on the ACCRE system:
   
   
   • ln -fs /home/phenix/cvs/offline/analysis/exodus/defined_decays.txt .
   • ln -fs /home/phenix/cvs/offline/analysis/exodus/defined_particles.txt .
     
     
3. Start the EXODUS program by typing exodus_generate
   
   
4. The EXODUS program will ask for the type of event to generate. Typically you will want "Single-particle generator" which is option 5).
   
   
5. After that, the EXODUS program will request the number of events.
   
   
6. The next request will be for the name of an ASCII text file which will be the output file from the program. You can type oscar.txt for this name.
   
   
7. The program will than ask for the type of transverse momentum (pt) distribution. You can have either a flat (uniform) distribution, or a distribution which is enhanced at low pt. Typically people like the enhanced low pt distribution since this gives greater statistical accuracy to the lower pt particle, and the real data are concentrated at lower pt too. The low pt enhancement is option 2).
   
   
8. The program will then give you a list of particle types. For example, 211 is a pi+. Type in which particle you want.
   
   
9. The program will then ask for the minimum pt value.
   
   
10. The program will next ask for the maximum pt value.
    
    
11. Similarly, the program will ask for the minimum and maximum rapidity values in succession. The central arm of the PHENIX detector has range of approximately -0.35 to +0.35 units of rapidity, and it is typical to use -0.60 to +0.60 in simulation to account for event vertex-dependent changes to the PHENIX rapidity acceptance. The event vertex range is given in the following input.
    
    
12. After this the program will ask for the vertex range, meaning the Z extent of the data. For example, 30 would mean +/- 30 cm which is typical.
    
    
13. Finally, the program will ask for the minimum and maximum azimuthal angle ranges ("phi") where you give the values in radians. You can use 6.283185 as the number for 2pi radians, which is typically what is wanted for the full azimuthal range.
    
    

Running PISA at Vanderbilt

An older documentation site for running the PISA software in PHENIX is at http://www.phenix.bnl.gov/phenix/WWW/simulation/pisaHead.html . The PHENIX wiki system has since superseded these static web server page areas.

This older PHENIX WWW site encompasses not only the running of PISA but also the production of simulated DSTs. The emphasis is on the use of canned shell scripts which will launch individual simulation jobs for a first time user. These first time user scripts have been adapted to work at Vanderbilt. New users of PISA should be familiar with the directions contained at the main simulation documentation site, especially in the PISA hits file subsection.

Single PISA jobs on VUPAC

The VUPAC farm has the RHEL 4 operating system. This means, among other things, that the C++ compiler is gcc 3.4.6. The compiler version in turn determines that a particular ROOT software build be used, one which was compiled with that specific C++ compiler. At present we use the ROOT-5.17/01 version, as built by PHENIX at RACF. That ROOT directory tree has been copied from RACF to the Vanderbilt computer farms.

You can obtain a standard set of PISA control files on VUPAC similarly to how it is done on RACF. You first would make a working directory, on a scratch disk not your home disk but rather a directory which you make in your area under the /rhic1 volume, i.e. a subdirectory such as /rhic1/yourName/pisaTest. In that working directory you would execute the script

closerootout_ - nbytes: 180272
closerootout_ - closing PISA TFile
PISA Running Times: Real Time =   5.47 seconds Cpu Time =   4.92 seconds
VertexOffset::~VertexOffset

Single PISA jobs on ACCRE (This section is obsolete until further notice.)

You can obtain a standard set of PISA control files on VUPAC similarly to how it is done on RACF. You first would make a working directory, on a scratch disk not your home disk. In the ACCRE system there is also a scratch areas called /scratch , which is somewhat analogous to the /rhic1 scratch areas at VUPAC. (A big difference is that the /scratch areas are simply visible from all nodes at ACCRE. If you have a /rhic1 area on VUPAC node vpac22, then to see that area on vpac23 you must use the full path /net/vpac22/rhic1.)

At ACCRE you should first make a directory with your name on it in the /scratch area, and then make subdirectories in that named directory. That way, it is easy to determine who is the owner of the files. In the PISA test working subdirectory you would execute the script

PISAEvent.root close: NBYTES = 89668
PISA Running Times: Real Time =  11.41 seconds Cpu Time =  11.10 seconds

PERL Scripts for PISA hits files production in simulation projects on ACCRE (This section is obsolete until further notice.)

Simulation projects involve the generation of perhaps hundreds of output files. This is accomplished on computer farms which consist of independent nodes, typically having 2 CPUs per node. For PISA hits file generation in simulation projects we have adopted the following strategy. We assume that a certain number of jobs will be run simultaneously, say 25. Each job will be run in a different sub-directory so that its input and output files do not conflict with the names of simultaneously running jobs. When a particular job is over its generically named output file (PISAEvent.root) will be renamed to a unique, well-defined name, and transferred into a separate ("done") directory. Then this particular directory is used again with for a different PISA job. The PISA jobs are different from each other because they use different random number seeds to start the Monte Carlo process. So even though the overall input event request is the same, the GEANT tracking of the events will always be different because of the different random number seed. For this reason, the different directories are named rndm01 , rndm02 , and up to rndmNN where NN represents the number of jobs which will be running in parallel.

For a given simulation project, say project 99 as an example, one will set up a project99 top directory in a scratch disk disk area. For example, on the ACCRE system the scratch area is /scratch/yourAccount where yourAccount is your account name. The steps to start generation PISA hits files for a project area as follows:

1. In the /scratch/yourName/projectXX create the subdirectories named exodusEvents, pisaEvents, and dstEvents. For the most part the input events into PISA will be EXODUS event files stored in the exodusEvents subdirectory. The instructions above told about generating EXODUS events. It may be convenient for you have extra subdirectories in the pisaEvents and dstEvents subdirectory, such as ParticleType/FieldKind for particle particles/antiparticles and magnetic field configurations. This will depend on the project which you are doing.
   
   
2. Change the directory to the /scratch/yourName/projectXX/pisaEvents subdirectory plus whatever extra subdirectories might be needed for this project. After you have changed directories copy over the example /home/phnxreco/perlScripts/makeRndm.pl PERL script to this subdirectory. You must look inside the makeRndm.pl to learn the three control inputs needed for this PERL script. You will also have to edit the makeRndm.pl script to point to the correct top directory area, /scratch/yourName/projectXX/pisaEvents/ plus subdirectories. When you have a correct version of makeRndm.pl , then run the script with the correct set of three input arguments, as documented inside the script. The purpose of this script is to set up the set of subdirectories in which a set number of PISA simulation jobs will be running simultaneously. The script also does softlinks for needed input files. In particular, on the ACCRE farm, the compute nodes do not have access to the wide area AFS network served out of the Brookhaven lab. So all the needed input files have to have local copies or softlinks. At ACCRE the master copies of these needed files are stored in the /home/phnxreco/macroTest/ subdiectory.
   
   The makeRndm.pl script will copy over the needed PISA input files. In particular, you should check that the pisa.kumac file has the correct detector subsystems in place. For simulations before Run7 there was no TOF-West, for example. Also, for Run6 or earlier, the MAGF control line in the pisa.kumac file is used to set the polarity of the field, either ++ or --

   MAGF '3D++' 1.00 0001  0.0 0.0
   

   The 1.00 is a scale factor which indicates the use of the ++ field, while a -1.00 would indicate the use of the -- field.
   
   For Run7, the two Central Arm magnetic field coils had currents in opposite directions, in order to have straight line trajectories through the HBD (which did not work well in Run7 as it happened). Hence, the makeRndm.pl script was modified to accept a run7a or a run7b input which will set the scale factor to 1.00 or -1.00, respectively, for use with the 3D+- magnetic field specification. [NOTE: the makeRndm.pl script has to be modified, and checked to work for Run6 or earlier because of changes in PISA during 2007.]
   
   As mentioned before, PISA simulation projects typically use OSCAR event files in ROOT format as input. The OSCAR event file may have in it say 5 million events, and the PISA program is set up to process say 50 thousand event In the top working directory of the project there will be an empty oscar.root file made by the makeRndm.pl script. You have to manually make a softlink from the actual OSCAR ROOT file, which you produced in the EXODUS steps above, to this oscar.root file before proceeding.
   
   
3. Next copy the job submission script:
   
   
   • If you are running on the ACCRE system, which uses the PBS batch submission system, then in the same top directory area copy over the example script /home/phnxreco/perlScripts/project99PISAPiPlus+-.pl . The name of this script indicates that it for running positive pions with the +- magnetic field (coil currents in opposite direction, normal polarity). Simulation projects often involve running several different particles and antiparticles in both the normal and the reversed polarity configurations.
     
     
   • The batch submission process is not yet established on the VUPAC system, which will be using CONDOR instead of PBS.
   
   On ACCRE, the control files for PISA are copied into the compute node's local /tmp directory. The PISA hits output file is generated first in this /tmp directory. When the PISA job is finished the output file would normally be copied into a /scratch directory areas which you set up with the makeRndm.pl script. Then the local /tmp directory is deleted. For this particular example project99PISAPiPlust+-.pl script which was used as a prototype for QM'08 simulation projects, the /gpfs2 area is used instead of the /scratch area.

   You should examine the PBS script to see how all this is done. Also, one should always set the checkMode variable in the PERL script to be equal to 1, which only creates but does not submit the jobs scripts to PBS. You can manually submit one of the jobs to see that all is working properly and that the allocated CPU and wall clock times are sufficient.
   
   Overall the job submission script has two parts. The first part assembles individual job scripts for a total number of jobs, and then submits the first cycles for those jobs for batch processing. For example, it may be that 50 jobs are required in total for a given particle. If 25 jobs can be run simultaneously, where the number 25 would have been specified in the makeRndm.pl script execution, then 50/25 = 2 cycles of processing will be required. That is, 2 jobs will be run in succession in each subdirectory. After each job is completed, the output will be moved to a "done" subdirectory and a following job will be started. The monitoring of when each job completes, the output movement, and the start of the following job in a given subdirectory are all controlled by the second part of the job submission script.
   
   Read the documentation on the first page of the scripts for how to make appropriate changes in what to look for in the log file. Unfortunately there is a longstanding GEANT bug which will sometimes cause a FORTRAN STOP statement to be executed in PISA. So the PISA job will stop before the requested number of events has been completed. There will be a PISAEvent.root file produced, and moved to the "done" area, but the re-submission logic of the PERL script will be affected. There are ways to get around this problem, which can be automated in the future.

Single Job Event Reconstruction Software at Vanderbilt ("PISA-to-DST")

After the PISA hits file generation, the next stage in the simulation sequence is typically to process this file with the detector response software and then the event reconstruction software. These two processing steps are combined into one stage which is called PISA-to-DST. You can obtain a standard set of PISA-to-DST control files on VUPAC similarly to how you obtained the input files for the PISA hits file generation. You first would make a working directory, on a scratch disk not your home disk but rather a directory which you make in your area under the /rhic1 volume on the VPAC system, or in your areas under the /scratch volume the ACCRE system. In that working directory on VUPAC you would execute the script

eventLoop : Real Time = 134.80 seconds Cpu Time = 129.54 seconds
Completed reconstruction.

vpac13.phy.vanderbilt.edu

The pisaToDSTLinker.csh script sets up for Run7 200 GeV Au+Au simulations. In the past one has been able to setup for other run simulations with a more complex script called pisaToDSTLinkerNoAFS.csh which requires one input parameter. This feature needs to be confirmed for the pro.82 software.

PERL Scripts Production of Simulated nanoDST Files (Needs to be modified for VUPAC system which uses CONDOR )

Event reconstruction of the simulation project files at Vanderbilt started in 2005. The overall design of these reconstruction job PERL scripts is much the same as for the PISA hits file production scripts, however, and so is the procedure:

1. First set up a working project top directory for reconstruction. On the VUPAC systems, this would typically be in your /rhic1 area. On the ACCRE system, this would be in one of your scratch area, for example /scratch/YourLastName/YourProject plus any needed subdirectories. As an example, on the VUPAC system, where batch production will not work until CONDOR is in place, you would copy the /home/phenix/perlScripts/nanoMakeRndm.pl script to this directory and edit it as necessary for your project. On ACCRE, where the PBS production method is working, the script location is /home/phnxreco/perlScripts/nanoMakeRndm.pl. This is the script which sets up the group of subdirectories, say 25, in which reconstructions will be independently processed.
   
   
2. On ACCRE you will next copy the example /home/phnxreco/perlScripts/project99DSTPiPlus+-.pl script and edit as necessary. As the name implies, this script is being developed to reconstruct the positive pions in the +- field for the example TOF-West Project 99. You should look carefully in this script in order to determine what you neede to edit for a different project.

Checking the simulated simDST.root output file (working only at VUPAC)

A simple analysis package for the simulated DST output file exists in the simDSTCheck CVS area. There is a single C++ Fun4All module function simDSTCheck.C which will process one or more simulated DST input files into a single output NTUPLE file. The NTUPLE parameters contain various data members of the PHCentralTrack data structure. As such, this global central track (cgl) NTUPLE file is useful for understanding the response and acceptance of the various central arm subsystems to hadronic, charged particles. There are also parameters from the original GEANT information which are matched to the reconstructed central track. This GEANT information is being recovered with the use of the McEvalSingleList class.

A ROOT macro analyze_simDST.C does the actual processing of one or more simulated input DST files. As indicated above, the processed information is put into a single NTUPLE contained in a ROOT output file which has the default name simDSTCheck.root. In the same CVS area there are other ROOT macros, e.g. plotTFW_PhiResiduals_LowSector.C, which show how to use the output NTUPLE for a particular detector subsystem analysis.

A second ROOT macro named analyze_simDSTPairs.C scans the cgl NTUPLE in the simDSTCheck.root file event-by-event with the purpose of combining pairs of oppositely charged, identified particles into a parent invariant mass. This assumes that the oppositely charged particles in the same event were produced by the decay of a neutral meson. The hadron mass identification is accomplished according to the momentum dependent capabilities of the various outer detector subsystems in the central arm of PHENIX: TOF-East, EMCal East (PbSc only), TOF-West (simulation is not yet completely working), Aerogel, and EMCal West. The cuts on the hadron mass identification used by the analyze_simDSTPairs.C macro can be adjusted by the user according to the needs of the analysis. The analyze_simDSTPairs.C macro does not contain anything specific about a particlar meson decay.

Building the simDSTCheck library

The following instructions review the steps for building the simDSTCheck library for use at Vanderbilt. These are the standard steps for building any existing library from the PHENIX CVS area.

1. First check out the software from the PHENIX CVS area. You would typically have a CVS top directory (e.g. /home/maguirc/cvs). In that directory you would give a CVS check out command: cvs checkout offline/analysis/simDSTCheck After this command is given you should see a number of files being copied from the PHENIX CVS area at RACF into your local directory.
   
   
2. Next, go to your Build area (e.g. /home/maguirc/build). If you don't have such an area then create it. Similarly, you should have an Install area (e.g. /home/maguirc/install), and you should create such an area too. In the Build directory, make a new subdirctory called simDSTCheck.
   
   
3. In the simDSTCheck subdirectory of your Build area, give the autogen.sh command similar to the following: /home/maguirc/cvs/offline/analysis/simDSTCheck/autogen.sh --prefix=/home/maguirc/install Of course, use your own directory paths instead of mine. A number of lines of output should scroll by the terminal, ending with:

   
   configure: creating ./config.status
   config.status: creating Makefile
   config.status: executing depfiles commands
   
   

   If you don't see these lines, then the autogen.sh process has failed.
   
   
4. After the autogen.sh process has completed, give the following command which compiles the source code for the simDSTCheck library, and then puts the shared object libraries in your Install area gmake install The compilation and install processes should take only a few seconds. At the end you should see something like

   
   bin/sh /home/maguirc/cvs/offline/analysis/simDSTCheck/mkinstalldirs /home/maguirc/install/include
    /usr/bin/install -c -m 644 /home/maguirc/cvs/offline/analysis/simDSTCheck/simDSTCheck.h /home/maguirc/install/include/simDSTCheck.h
   gmake[1]: Leaving directory `/home/maguirc/build/simDSTCheck'
   
   

   Again, your paths will appear in these output lines instead of mine. You can check that the libraries have been installed by listing them with a command like ls -al /home/maguirc/install/lib You should get an output similar to the following with your account name instead of mine

   
   -rwxr-xr-x  1 maguirc rhic   5876 Sep  6 17:38 libsimDSTCheck.la*
   lrwxrwxrwx  1 maguirc rhic     23 Sep  6 17:38 libsimDSTCheck.so -> libsimDSTCheck.so.0.0.0*
   lrwxrwxrwx  1 maguirc rhic     23 Sep  6 17:38 libsimDSTCheck.so.0 -> libsimDSTCheck.so.0.0.0*
   -rwxr-xr-x  1 maguirc rhic 160523 Sep  6 17:38 libsimDSTCheck.so.0.0.0*
   
   

Running the simDSTCheck library

In order to run the simDSTCheck library you must first modify your LD_LIBRARY_PATH environment variable such that it points to your Install area first. This will enable the library loading command in the ROOT macro to find the libsimDSTCheck.so library. The path modification command is like

/home/maguirc/install/lib:.:/usr/lib:/home/phenix/phenixSL4/opt/lib:
/usr/local/lib:/home/phenix/pro82-12Aug2008/lib:
/home/phenix/root-5.17.01/lib:/home/maguirc/cern/pro/lib

After you have done the library modification command, then do the following steps

1. Change your working directory to the macro area in your simDSTCheck source code area which you created previously. My area is
   /home/maguirc/cvs/offline/analysis/simDSTCheck/macro
   
   
2. In this area you will see a analyze_simDST.C ROOT macro. You should look at the comment lines in this macro, and then learn how the macro works. There are a number of options available as input parameters when this macro is executed.
   
   
3. By default the macro reads a text file simDST.txt which contains a list of one or more individually named simulation DST files. The version of the simDST.txt file in CVS contains the name of a simulation DST file in my /net/vpac23/rhic1/maguire/lambdaTests07Sep2008/pisaToDST area. This simulation DST file was produced by processing 25K single Lambda events produced by EXODUS with PISA and pisaToDST.
   
   
4. Start the ROOT program with the command root If you are running from a non-graphics terminal, you would have use the -b option to prevent a display not found error.
   
   
5. When you get the root prompt then you should execute the ROOT macro with the default options by using the following command .x analyze_simDST.C The ROOT processing of the default input file will take a few minutes. You will see warning messages such as

   
   Warning in : no dictionary for class TABLE_HEAD_ST is available
   Warning in : no dictionary for class DCGHIT_ST is available
   Warning in : no dictionary for class DDCHTRACKS_ST is available
   

   but these are not serious. The ROOT processing should take about two minutes, and at the end of the processing you should see lines such as

   
   Reached end of input file list 
   eventLoop : Real Time = 133.84 seconds Cpu Time = 133.71 seconds
   Total events processed 25000
   Writing NTUPLE output file;  number of entries = 7875
   

   
   
   
6. When the ROOT processing is finished, the output will be a ROOT NTUPLE file called simDSTCheck.root. This file can be processed by plotting macros according to your own analysis purposes. The macros currently in the simDSTCheck CVS area, such as plotTFW_PhiResiduals.C, were developed in 2007 to assist with the debugging of the TOF-West simulation code.

Documentation for GridFTP transfer of files from Vanderbilt to RACF (still in progress)

A simulation project can produce hundreds of GBytes of output. These files need to be transported to the RACF computer system, and then stored into the archival storage system called HPSS. The file information should also be stored in the PHENIX file database system, although that procedure is not yet documented. GridFTP is the preferred way of transferring files from Vanderbilt to RACF, replacing the older BBFTP method which requires a password for every copy session.

On the ACCRE farm, there is a designated GridFTP server node called firebird, which is isolated from the main ACCRE farm and disks because of security reasons. Hence files have to be transferred first from the disk areas (/scratch or /gpfs2) visible on the main ACCRE farm to buffer disks on the firebird server. A java software product called FDT will perform this first step. When the FDT step is completed, then a GridFTP copy job will be automatically launched on the firebird server to copy to a disk area in the maguire account at RACF. There are three sample PERL scripts for using FDT+GridFTP in the /home/phenix/perlScripts directory.

• gridCopyExodusProject99.pl for copying EXODUS event input files
• gridCopyPISAProject99.pl for copying PISA hits output files
• gridCopyDSTProject99.pl for copying DST and CNT output files

Documentation for BBFTP transfer of files from Vanderbilt to RACF (still in progress)

An older utility program to transfer files from the ACCRE or VUPAC disks to the RACF disks is called bbftp . Typically all the files to be transferred from an ACCRE disk will reside in a particular directory on one of the /scratch volumes. Since there typically many files to transfer one usually sets up an input "control.txt" file for use by bbftp. This control.txt file contains the file names which are being transferred as well as the disk directory at RACF which is the destination of the transfer.

1. There are three sample PERL scripts in the /home/phenix/perlScripts directory.
   
   
   • gridCopyExodusProject99.pl for copying EXODUS event input files
   • gridCopyPISAProject99.pl for copying PISA hits output files
   • gridCopyDSTProject99.pl for copying DST and CNT output files
   
   These PERL scripts will generated both C-shell scripts for use by GridFTP, and ASCII control text files for use by bbftp. You should read and understand how these scripts work. The most important feature is that you have the correct source (at ACCRE or VUPAC) and destination (at RACF) directory locations. At RACF the directory locations will be typically on the /phenix/data11 volume, in subdirectories exodusEvents, pisaEvents, and dstEvents under your own name. Because the file names typically include information about the particle type and the magnetic field, it is not normally necessary to add extra subdirectories to the RACF names.
   
   
2. Once you have generated a given bbftp control text file, then the command to make the transfer to RACF is something like
   
   bbftp -i bbftpCopyPISA20March2007.txt -p 10 -u yourName rftpexp.rhic.bnl.gov
   You will be asked for your password at RACF. After you give your password, the file transfers should begin automatically. The above command asks that the bbftp system use 10 parallel streams for the file transfer. You should examine the control text file to see that the bbftp commands have the proper source and destination file directories. Also, it would be wise for you to have a window opened to an rcas node at RACF where you can monitor how the transfers are progressing

Embedding Software at Vanderbilt (to be documented)