Jonathan Poisson
Table of Contents
Q: I just deleted some of my files, how do I rcecover them?
Q: When exactly are backups done on the cluster?
Q: Is there any way to configure my access to the cluster so I no longer need to type my password?
Q: How can I increase the walltime of my jobs?
Q: How can I know the amount of disk space I am using without using “du”?
Q: How can I delete files in one directory which are older than 24 hours?
Q: How do I copy my files to localscratch in my job submission script?
Q: How can I use “find” to locate a specific file?
Q: What are the best compilers for C++ and Fortran77?
Q: Why can't you tell you which compiler is the best?
Q: Is there any way to speed up a large file transfer?
Q: Are there any mandatory rules for running parallel (multi-core/multi-node) programs?
Q: How do I check where a program is spending most of its time?
Q: How do I setup or change my password for the labonly section of the webserver?
Q: I just deleted some of my files, how do I recover them?
A: The umms-amino storage system has several levels of backup in the event of data loss. There exist weekly backups of the whole storage system ( home directory ) up to six months ago. As well as daily backups for the immediate past two weeks.
Currently to access backups you need to contact myself to mount the appropriate backup for recovery.
Q: When exactly are backups done on the cluster?
A: All the time.
There are two systems performing backups of your primary storage space (/home).
1. The storage system itself keeps short term snapshots of recent activity. It keeps one snapshot every day (11 PM) for 14 days.
2.We have the thumper system which performs a complete copy every week (over the weekend starting Saturday AM), it keeps approximately six months worth of these backups.
A: When dealing with I/O (Input/Output) its important to clarify your intentions and meaning as there are many ways to quantify input and output and many different optimization goals. For the purpose of the umms-amino cluster the optimization goal is to reduce I/O operations on the central storage system, i.e. /home.
Given this goal, it is almost always ideal to first copy input to the local disk device (/tmp) as we have no optimization goals for I/O on local disks. You may think that it makes little difference as you are reading the entire file, either to copy it, or to input it to your program. But when making this assumption you may not realize you are quantifying the I/O in a manner different from the system.
The system quantifies I/O as a number of “operations” which do not always have a fixed correlation to real file size. An operation may read a single line of your file or it may read a thousand lines of your file. Knowing that operations may be of variable size, the key thing to discover and realize is that (in general) the amount of time the operation takes also may not correlate to the amount data that operation contains. To simplify for our context: when you read a thousand lines from a file it can take the exact same amount of time as if you read a single line.
Naturally there are many exceptions and additional concepts and considerations in an I/O calculation, so this cannot be stated to be true for all cases. It does make clear, that when you read a file, one line at a time, one operation at a time; it can take much longer, and be much less efficient than a copy which automatically optimizes the size of the operations to complete the copy as fast as possible.
Q: Is there any way to configure my access to the cluster so I no longer need to type my password?
A: Yes.
Before saying more, I should tell you that this isn't the most “secure” thing to do. If your desktop gets hacked, suddenly they have free access to your cluster account. If you walk away for a minute without locking your screen, somebody can quickly run over and have cluster access in seconds. It opens up a lot more possibilities, so just be careful with it.
The simple way to configure this is:
On your desktop, run ssh-keygen. This will ask you a few questions: where to store the key, and if you would like to use a passphrase (a passphrase is essentially a password on your key that you have to type before using it).
Once these questions are answered, you need to copy the contents of your id_rsa.pub (do not actually copy and overwrite your id_rsa.pub file, as that will cause problems for your cluster account) and paste them into your authorized_keys file on the cluster (usually ~/.ssh/authorized_keys).
If everything checks out, then the next time you log in it should not ask you for a password.
A: Your jobs can be deleted with the “qdel” command. For “qdel” to work you must supply it with a job number to be deleted. If you want to delete ALL your jobs, you can try this command:
/opt/moab/bin/mjobctl -c -w user=${USER}
Q: How can I increase the walltime of my jobs?
A: In general, you cannot. In most systems the walltime is used to determine job priority and start time, therefore, the scheduling software disables the ability for regular users to change their walltime after the job has started as this would effectively allow you to exploit the system in favor of your jobs.
It should be known that the current configuration of umms-amino does not weigh walltime into account when scheduling. So there is effectively no reason why you cannot demand a thousand hours for every job.
In the case of parent jobs, I can easily and quickly setup a limit (hard and soft) to the number of jobs you can have running at any given time. This would alleviate problems with parent jobs overrunning due to excessive queue times.
Q: How can I know the amount of disk space I am using without using “du”?
A: Try using the “quota” command. This lists out relevant information regarding your quota on the storage system, including the amount of space used in blocks. On umms-amino, each block is sized at 1024 bytes.
Q: How can I delete files in one directory which are older than 24 hours?
A: The best way to do this is through creative usage of the (gnu) “find” command. For example, to list all files and directories older than 24 hours in the /tmp directory you can use:
find /tmp -mtime +1
This will list all files in the /tmp directory and its subdirectories older than one day. If you want to exclude the directories themselves, then you can use:
find /tmp -mtime +1 -type f
And if you want to limit yourself to the current directory without venturing into subdirectories you could do this:
find /tmp -maxdepth 1 -mtime +1 -type f
(Note: maxdepth is finicky and should be placed before the -mtime argument)
Now if you want to delete those files in that list you just have to feed the output to the rm command. There are a few ways to do this.
Within find:
find /tmp -maxdepth=1 -mtime +1 -type f -exec rm '{}' \;
Or piping to xargs:
find /tmp -maxdepth=1 -mtime +1 -type f | xargs rm
A: You should use the “-s” option. This limits the du output to one line, which is the total size of the specified directory and all it's subdirectories. So to learn the size of all the files in /tmp and its subdirectories you would use “du -s /tmp”
Q: How do I copy my files to localscratch in my job submission script?
A: This depends a lot on how you are submitting your job, where your data is, and what scripting language you are using.
For example if you are submitting a bash script from your data directory, then it you can see the technique from an example.
Bash Example
#!/bin/bash
#go to your submission directory
cd ${PBS_O_WORKDIR}
#define scratch location
LOCAL_WORKDIR=/tmp/${USER}/${PBS_JOBID}
# make scratch directory
mkdir -p ${LOCAL_WORKDIR}
# copy all data in current directory (the directory you submitted from)
# to the scratch directory
cp -r * ${LOCAL_WORKDIR}
# change directories to scratch
cd ${LOCAL_WORKDIR}
# run your program
<Your stuff goes here>
# copy back data
cp -r * ${PBS_O_WORKDIR}
# change directory back
cd ${PBS_O_WORKDIR}
#cleanup
rm -rf ${LOCAL_WORKDIR}
Naturally, you will have to change the script depending on where your data is located, but the concept is the same. Create a scratch space, copy your data, change directories to that space, and then execute your program.
Since many of you use perl here is an example using perl.
#!/usr/bin/perl
use File::Path;
use File::Copy;
# go to submission directory
chdir($ENV{'PBS_O_WORKDIR'}) or die "Failed to change to PBS_O_WORKDIR";
# define scratch location
$LOCAL_WORKDIR='/tmp/'.$ENV{'USER'}.'/'.$ENV{'PBS_JOBID'};
# make scratch directory
mkpath($LOCAL_WORKDIR, 1, 0775);
# copy all data in current directory (the directory you submitted from)
# to the scratch directory
for my $file (<*>) {
copy($file, $LOCAL_WORKDIR) or warn "Failed to copy $file";
}
# change directories to scratch
chdir($LOCAL_WORKDIR);
# run your program
<Your stuff goes here>
# copy back data
for my $file (<*>) {
copy ($file, $LOCAL_WORKDIR) or warn "Failed to copy $file";
}
# change directory back
chdir($ENV{'PBS_O_WORKDIR'});
# cleanup
rmtree($LOCAL_WORKDIR, 1, 1);
Again, that basic template needs to be modified to suit your particular needs following the same basic concept. Also note that my example does not utilize backticks, but backticks may be substituted for many operations, such as the copy operation.
Q: How can I use “find” to locate a specific file?
A: To locate a specific file you need to use the “-name” parameter which breaks down results by name. Say you're looking for a file named “test.html” which could be anywhere.
Example:
find / -name test.html
This will search every directory on the computer for files named test.html, but you may notice a lot of permission denied. To suppress those, redirect the error channel to /dev/null
Example:
find / -name test.html 2>/dev/null
/tmp/test.html
/usr/share/doc/db4-devel-4.7.25/ref/build_win/test.html
/usr/share/doc/db4-devel-4.7.25/ref/build_unix/test.html
As you can see in this example a few test.html files were located, hopefully one of them is the one you were looking for. If not you need to expand your search. Say you remember it was not exactly test.html but had test.html in it. A slight alteration to the find command makes a difference to as to the search output.
Example:
find / -name *test.html 2>/dev/null
/home/jdpoisso/Downloads/gcc/gcc-4.2.4/INSTALL/test.html
/tmp/test.html
/usr/share/doc/doxygen-1.6.1/examples/overload/html/class_test.html
/usr/share/doc/doxygen-1.6.1/examples/jdstyle/html/class_test.html
/usr/share/doc/doxygen-1.6.1/examples/include/html/class_test.html
/usr/share/doc/doxygen-1.6.1/examples/memgrp/html/class_test.html
Adding the “star” prior to the test.html opens the search so that “find” will return any file that ends with test.html. The “star” is a wildcard operator which matches to any set of elements greater than or equal to zero in length (See Kleene star). The star may be positioned anywhere in the “-name” parameter to account for a potential and unknown set.
You can infer from the prior examples that the “-name” parameter seeks exact matches, and only matches to directory names and not their contents. In order to locate a file with this method you need to have a reasonable idea of what you are looking for. Given a situation where you have very little idea what you are looking for it may be better to use a “shotgun” method to look for a file. I recommend this method.
Example:
find / 2>/dev/null | grep pdb
...
/home/jdpoisso/rasmol-2.7.5-23Jul09/data/1ldb.pdb.gz
/home/jdpoisso/rasmol-2.7.5-23Jul09/data/1crn.pdb
/home/jdpoisso/Downloads/model1.pdb
/home/jdpoisso/TM-align/pdb/1a7tA.pdb
/opt/sun/javadb/docs/html/adminguide/cadminbckupdb.html
/home/jdpoisso/TM-align/pdb/1a8i_.pdb
...
As you can see you get a much wider variety of matches, only a fraction of which are listed here, with this method. We have found: a gzipped pdb file, contents of a pdb directory, and an html file that has nothing to do with what you're looking for. Most of your results will be unhelpful but you can narrow it down further by changing your search path, or piping an additional grep to the command until you get it right.
Q: What are the best compilers for C++ and Fortran77?
A: I can't tell which compilers are the “best.”
In general, you should use gcc and gfortran, they are the most frequently updated and developed.
Q: Why can't you tell you which compiler is the best?
A: As a gross simplification it is because the results of a compiler are only an approximation.
In Computer Science a lot of research is directed towards developing, refining, and attempting to prove better algorithms and means of compilation. Definitively answering the question of the “best” compiler is the subject of much research.
Your experience may show that you get better results with a certain compilers, but your experience is in no way definitive. It constitutes only a few data points under marginally variable inputs (your programs).
Like with any heuristic method your looking for the method which returns the best results for the most inputs across the range of all inputs. Discovering and implementing the most efficient method for each individual input is impractical.
In general you should use compilers which are updated frequently and are commonly used, as they are most likely to take advantage of the latest research and methodologies in compiler construction. Compilers developed by institutions or corporations that fund compiler research are also likely to be very good (such as Intel or PGI).
Q: Is there any way to speed up a large file transfer?
A:When simply copying a file from a single source to a single destination.
Generally, No.
Raw speed is entirely limited by the intervening network capacity, which is a physical limitation. There exist varied software and hardware tricks to increase “speed.” As well as best practices such as compressing your data before transfer.
Also be aware of the negative consequences of some of these tricks. Programs such as download accelerators that claim to speed up the transfer of files. In most cases these programs work by establishing multiple connections to facilitate the transfer of the file. Since most sites share their physical bandwidth between multiple connections this often provides some speed boost due to multiple connections. However, most sites try to limit the number of connections and will block IPs with too many connections.
Q: Are there any mandatory rules for running parallel (multi-core/multi-node) programs?
A: As of right now, only the rules which are true for all jobs apply. However, I should point out that the cluster scheduler as currently configured will not effectively or efficiently schedule parallel jobs.
In our current setup the scheduler will assign each core request to separate nodes, unless specifically told not to. If told not to, the job will wait until it meets the criteria you specified. Normally you would ask for as many cores in as few nodes possible, such as 16 cores across 2 nodes, or 32 cores across 4 nodes; 8 cores per node. Since the scheduler distributes all jobs evenly across all nodes. Seldom does any node have all 8 cores available for parallel computation.
From a best practices perspective jobs should restrict themselves to one node unless they require resources not available to a single node. This is entirely for efficiency reasons, as once a parallel program crosses between nodes it introduces the software limitations and physical constraints of the network. These liabilities can greatly slow your program, to the point of being slower than if run with fewer cores on a single node. It is up to the programmer to anticipate and mitigate these liabilities as much as possible. Even with superior hardware and networking nothing will completely eliminate the additional variables caused by crossing node boundaries.
Q: How do I check where a program is spending most of its time?
A: This is a very complicated question.
When profiling a program there are many tools available. In the interests of simplicity; the most handy are the GNU “time” command, and those you program yourself.
In general most utilities take the program in its entirety and measure its time, and not the time for specific sections of your program. This is why most programming languages include libraries so you can program this yourself, perl for example has “Benchmark” and “Time::HiRes”.
In general you can measure specific sections of your code by checking the time prior to entering that section, and then checking the time when you exit the section. The difference is how much time the computer spent in that section. In this way you can measure the length of a particular IO access, or a complex set of formulas.
When taking the program as a whole the GNU time utility is good for getting information on how long the program took and what it did. The GNU time utility is at “/usr/bin/time” (Note: running plain “time” is not the same thing), you use it by giving it a format string and your program name.
So to get the same information as the POSIX “time” command you would just run :
“/usr/bin/time –format=”%E %S %U” program”
This prints out the real time, the system time, and the user time in that order. Now if I wanted different information, such as the amount of memory used during the program runtime then I'd run this:
“/usr/bin/time -format=”%M %K”
This outputs the maximum amount of memory used by the program during its lifetime, and the average amount used by the program during its lifetime.
You can also use GNU time to find esoteric information regarding your program. Such as the number of page faults, the number of times your program was context-switched (stopped by the operating system to run another program) as well as I/O information regarding signals, inputs, outputs and messages.
A: This answer to this question depends greatly on what files are being deposited in your home directory and how your script is running.
First, you should realize that the scheduler only generates two files. An output file which captures the standard output of the job, and an error file which captures the standard error of the job. By default these files are copied to the directory you submitted your job from. You can even set arbitrary locations for these files in your job script with PBS directives.
Example:
#PBS -o /tmp/myjobs/output
#PBS -e /tmp/myjobs/error
This example illustrates two PBS directives, when placed at the beginning of the file will send these files to arbitrary locations. Namely the output to /tmp/myjobs/output and the error to /tmp/myjobs/error
Now say you don't want two files instead you just want it bundled together in one. You can join the two files with another PBS directive.
Example:
#PBS -j oe
#PBS -o /tmp/myjobs/output
This set of directives, merges the standard error into the standard output and places that file in /tmp/myjobs/output when the job completes. Rather than two separate files, you now have a single file which contains both the output and the errors which occurred during the job.
Now that I've explained what contribution the scheduler may make to any files copied to your home directory I can explain how other files from your job might end up in your home directory.
Since the scheduler generates only a few files then the only remaining culprit for files being copied back is in fact your own job script. Here is a typical scenario.
Procedure:
1. Make directory for job in scratch location
2. Copy all files to scratch location
3. Change directory to scratch location
4. Run job
5. Copy all files back from scratch location
6. Clean up scratch location
Now consider what happens when your job starts. The scheduler logs you into the assigned node, into your home directory. Your job script is then executed and the procedure listed above is set in motion.
Now consider what might happen if some of these steps were to fail...
Procedure:
1. Make directory for job in scratch location
What if the scratch location (/tmp) is full? Or is read only? The directory isn't created then. There's an error and nothing happens.
2. Copy all files to scratch location
3. Change directory to scratch location
These two can happen interchangeably, and order and method is important. Did you “cd” to the scratch location first? If the directory isn't there, then this will fail, and where will you be? Still in your home directory.
How did you copy your files, did you copy the files to your current directory (i.e. “cp /library/yzhang/some_file .”) or did you copy them arbitrarily to the scratch location? If your current directory then suddenly all that data that was supposed to go to scratch is now in your home directory.
4. Run job
Does your job generate files? Creating and writing information out to the current directory? If your directory never changed, then these files are generated in your home directory, not the scratch location (/tmp) that you originally intended.
5. Copy all files back from scratch location
6. Clean up scratch location
If your scratch location was never established and created, these operations will do nothing to clean up the mess in your home directory that was created accidentally.
This is only one possible scenario.
There are many other possibilities why your job could be copying files to your home directory, and invariably it is caused by your job script. Unintentionally, and triggered by something unexpected (such as /tmp being full or read-only, notify your sys admin to fix this) but the cause is the job script.
Ideally your script should perform “sanity checks” to verify that certain parts of the procedure occurred correctly such as checking error codes or comparing your present directory to to the scratch location before copying files. It is the burden of the programmer to anticipate and compensate for unusual states or to accept the side effects of those unusual states.
Q: How do I setup or change my password for the labonly section of the webserver?
A: There is a special command you need to run on the webserver (zhanglab.dcmb.med.umich.edu) itself in order to set this password. If you are a member of the lab you should be able to log into zhanglab.dcmb.med.umich.edu via secure shell (ssh). Once logged in, run this command "sudo /usr/bin/htdigest /www/auth/.htpasswd labonly {INSERT_YOUR_USERNAME_HERE}". You should then be asked to set a new password for yourself, that is the password you must use to log into the labonly section of the web page.
Example:
[jdpoisso@umms-amino ~]$ ssh zhanglab.dcmb.med.umich.edu
jdpoisso@zhanglab.dcmb.med.umich.edu's password:
[jdpoisso@zhanglab ~]$ sudo /usr/bin/htdigest /www/auth/.htpasswd labonly jdpoisso
Changing password for user jdpoisso in realm labonly
New password:
Re-type new password:
[jdpoisso@zhanglab ~]$