ECON Knowledge Base

This page is read only. You can view the source, but not change it. Ask your administrator if you think this is wrong.
====== Running Batch Jobs ======

Fundamentally, the Econ cluster will be used to run relatively resource-intensive computations.  Linux has a number of utilities that are installed by default which make it easy to run such jobs.  Some are explained briefly below.

===== Nohup =====
The [[http://en.wikipedia.org/wiki/Nohup|nohup]] command is another way to get your process to run in the background.  For the standard mathematical software, use it like:
 <code> nohup stata-mp -b do myjob.do &
 nohup matlab -nojvm -nodisplay <myjob.m> myjob.out &
 nohup sas myjob.sas & </code>
In general, you can use it like:
 <code>nohup <your command> &</code>
The "nohup" command is primarily helpful in that it allows you to stop your current terminal session without interrupting the process.  That is, running a command through nohup allows you to disconnect from your terminal session while the command continues to run.

See also the Stata FAQ page dedicated to UNIX batch jobs:
[[http://www.stata.com/support/faqs/unix/batch.html|How do I run Stata in batch mode?]]

If you forgot to <code>nohup</code> a process and would like it to continue if the terminal is closed then use the command <code>disown</code>

=====The "&" and "fg" Commands=====
A very common use case might be that you want to run a large job, but want to tweak some aspects of your job process after it has started.  To do this, finish your initial command with "&", i.e. instead of running 
 <code>matlab < program.m > output.log</code>  
Instead, run
 <code>matlab < program.m > output.log & </code>

This starts your Matlab program.m in the background, so you can run some more commands.  Perhaps, you want to renice your Matlab command so that it doesn't use all of the cluster CPU.  After renicing, you want to check on your Matlab command.  To do this, use the "fg" command.  Type 
<code> fg matlab </code>
This will bring you back to viewing the output from your original command!

Note that the "&" command is the same as that used with <code>nohup</code> above.  The function of "nohup" is that it suppresses "hangup" signals that the terminal produces when running a command and it redirects output to a text file (nohup.out by default) instead of the terminal window.

=====Screen=====
The screen command allows you to start a shell that is detachable.  This is useful if you are executing commands in a shell and do not want to lose your spot when you close your shell.  Compared to the & and fg commands, you can close your true terminal session and get back to what you are doing on another computer or in another terminal on the same computer.

To use screen, type
 <code>screen</code>
and press enter.  You will be shown some introductions to screen and asked to press enter.  After doing so, you will be connected to a virtual terminal.  Pressing (depending on your terminal configuration) Ctrl-A and then D will "detach" the screen session.  You will go back to your normal terminal, but the screen session will remain active and executing it's processes.  To re-attach the session, type 
 <code>screen -r</code> 
and then press <TAB>.  Pressing <TAB> will auto-complete from among your available screen sessions.  Pick one of your sessions and press return - you are brought back to that session!

For more info on screen, see [[http://www.rackaid.com/resources/linux-screen-tutorial-and-how-to/|this tutorial]] or [[http://en.wikipedia.org/wiki/GNU_Screen|the Wikipedia article]].

Also, note that if you log out of the cluster and then log back in, you might be on a different node where your screen process was not running.  To move between nodes, use the <code>ssh <node></code> command.  See more on how to deal with multiple nodes at [[Managing_Jobs]]

Inside of screen, if you want to scroll up throw previous terminal output, type "Ctrl-A [" to enter scrollback mode which enables Pg-Up Pg-Down. Use Esc to leave scrollback mode.

=====At=====
The "at" command is useful to schedule jobs at a future date or time.  If you know, for example, that you want to run a job at 1 AM tomorrow morning, you could run
 <code>your commands | at 1am tomorrow</code> 
You can also run the command
<source lang="bash">at 1am tomorrow
> your commands
> another line of your commands
> Ctrl + D
</source>
The command "at 1am tomorrow" lets you start entering shell code to execute.  Pressing Ctrl+D lets the terminal know that you are done entering code, and it schedules the code to execute at the appropriate time.  For more on "at" see the [[http://en.wikipedia.org/wiki/Batch_(Unix)| at Wikipedia page]] or [[http://linux.die.net/man/1/at |the online main page]].

=====Viewing Output Files=====
When you run <code>nohup stata-mp -b do myjob.do &</code>, Stata will write a log file to myjob.log (unless you've specified something else within the code).  You might want to view the log file live as it is written.  You can do this with the command
 <code>tail -f myjob.log</code>
To stop viewing the output file, type <code>Control+C</code>

=====E-mail notification of process completion=====
If you want to get an e-mail notification of when something completes, make file (e.g. run_and_email.sh) that contains
 <code>stata -b do estimation.do</code>
 <code>echo "Alright it finally finished. -Me" | mail -s "Process done" youremail@gmail.com</code>
Then execute the script with
 <code>bash run_and_email.sh</code>
You can then use nohup on the above
 <code>nohup bash run_and_email.sh &</code>
ECON Knowledge Base

User Tools

Site Tools

Page Tools
