This fall, we are bringing you a major refresh of our website. Visit the beta now:

Try the beta
Indiana University

Quarry at Indiana University

Note: After seven years of production service, Indiana University's Quarry research computing cluster will be decommissioned on January 30, 2015. New accounts on Quarry will no longer be available after Thursday, December 18, 2014. IU students, faculty, and staff can request accounts on Karst, IU's newest high-throughput computing cluster; for instructions, see Requesting an account. User data on Quarry will not be removed; you can access your home directory data on Quarry from your account on any other IU research computing resource (see Accessing one home directory from another). All software modules available on Quarry will be available on Karst. If you have questions or concerns about Quarry's retirement, or need help, contact the High Performance Systems group.

On this page:


Quarry (quarry.uits.indiana.edu) is a Linux cluster computing environment for research and research instruction use at Indiana University.

Quarry nodes run Red Hat Enterprise Linux (RHEL 6.5). Job management is provided by the TORQUE resource manager and the Moab job scheduler. The Modules system is used to simplify application access and environment configuration.

Accounts on Quarry are available to all IU students, faculty, staff, and sponsored affiliated researchers. For more, see Requesting an account below.

Note: With Quarry retiring January 30, 2015, Research Technologies now hosts its "condominium computing" service on Karst. For more, see At IU, what options does UITS offer for purchasing computing resources in the UITS secure Data Centers?

Back to top

System information

Note: The scheduled monthly maintenance window for Quarry is the first Tuesday of each month, starting at 7am and ending the next morning (Wednesday) at 7am.

System summary
Machine type Cluster computing, high-throughput computing, and condominium computing for research and research instruction
Operating system RHEL 6.5
Memory model Distributed and shared
Local storage 42.06 TB aggregate
Rmax 26.1 TFLOPS

Back to top

File storage options

Note: Before storing data on this system, make sure you understand the information in the Working with ePHI research data section (below).

You can store files on your home directory or in scratch space:

  • Home directory: Your Quarry home directory disk space is allocated on a network-attached storage (NAS) device. You have a 100 GB disk quota, which is shared (if applicable) with your accounts on Big Red II, Karst, Mason, and the Research Database Complex (RDC).

    The path to your home directory is (replace username with your IU Network ID username):


    IU graduate students, faculty, and staff who need more than 100 GB of permanent storage can apply for accounts on the Research File System (RFS) and the Scholarly Data Archive (SDA). See At IU, how can I apply for an account on the SDA or RFS?

  • Local scratch space: Local serial-attached SCSI (SAS) scratch space is available on every node at: /tmp

    Local scratch space is not intended for permanent storage of data, and is not backed up. Files in local scratch space are automatically deleted once they are 14 days old.

  • Shared scratch space: Once you have an account on one of the UITS research computing systems, you also have access to 3.5 PB of shared scratch space.

    Shared scratch space is hosted on the Data Capacitor II (DC2) file system. The DC2 scratch directory is a temporary workspace. Scratch space is not allocated, and its total capacity fluctuates based on project space requirements. The DC2 file system is mounted on IU research systems as /N/dc2/scratch and behaves like any other disk device. If you have an account on an IU research system, you can access /N/dc2/scratch/username (replace username with your IU Network ID username). Access to /N/dc2/projects requires an allocation. For details, see The Data Capacitor II and DC-WAN high-speed file systems at Indiana University. Files in shared scratch space more than 60 days old are periodically purged, following user notification.

For more, see At IU, how much disk space is available to me on the research computing systems?

Back to top

Working with ePHI research data

The Health Insurance Portability and Accountability Act of 1996 (HIPAA) established rules protecting the privacy and security of personal health data. The HIPAA Security Rule set national standards specifically for the security of protected health information (PHI) that is created, stored, transmitted, or received electronically (i.e., electronic protected health information, or ePHI). To ensure the confidentiality, integrity, and availability of ePHI data, the HIPAA Security Rule requires organizations and individuals to implement a series of administrative, physical, and technical safeguards when working with ePHI data.

Although you can use this system for processing or storing electronic protected health information (ePHI) related to official IU research:

  • You and/or the project's principal investigator (PI) are responsible for ensuring the privacy and security of that data, and complying with applicable federal and state laws/regulations and institutional policies. IU's policies regarding HIPAA compliance require the appropriate Institutional Review Board (IRB) approvals and a data management plan.

  • You and/or the project's PI are responsible for implementing HIPAA-required administrative, physical, and technical safeguards to any person, process, application, or service used to collect, process, manage, analyze, or store ePHI data.

The UITS Advanced Biomedical IT Core provides consulting and online help for Indiana University researchers who need help securely processing, storing, and sharing ePHI research data. If you need help or have questions about managing HIPAA-regulated data at IU, contact the ABITC. For additional details about HIPAA compliance at IU, see HIPAA & ABITC and the Office of Vice President and General Counsel (OVPGC) HIPAA Privacy & Security page.

Important: Although UITS HIPAA-aligned resources are managed using standards surpassing official standards for managing institutional data at IU and are appropriate for storing HIPAA-regulated ePHI research data, they are not recognized by the IU Committee of Data Stewards as appropriate for storing institutional data elements classified as Critical that are not ePHI data. For help determining which institutional data elements classified as Critical are considered ePHI, see Which data elements in the classifications of institutional data are considered protected health information (PHI)?

The IU Committee of Data Stewards and the University Information Policy Office (UIPO) set official classification levels and data management standards for institutional data in accordance with the university's Management of Institutional Data policy. If you have questions about the classifications of institutional data, contact the appropriate Data Steward. To determine the most sensitive classification of institutional data you can store on any given UITS service, see the "Choosing an appropriate storage solution" section of At IU, which dedicated file storage services and IT services with storage components are appropriate for sensitive institutional data, including ePHI research data?

Note: In accordance with standards for access control mandated by the HIPAA Security Rule, you are not permitted to access ePHI data using a group (or departmental) account. To ensure accountability and enable only authorized users to access ePHI data, IU researchers must use their personal Network ID credentials for all work involving ePHI data.

Back to top

Available software

Software installed on Quarry is made available to users via Modules, an environment management system that lets you easily and dynamically add software packages to your user environment. For a list of software modules available on Quarry, see Quarry Modules in the IU Cyberinfrastructure Gateway.

For more on Modules, see the Using Modules to set up your software environment below. For more on the IU Cyberinfrastructure Gateway, see What is the IU Cyberinfrastructure Gateway?

Quarry users can request software using the Research Systems Software Request form.

Back to top

Requesting an account

To request an account on Quarry, IU students, faculty, staff, and sponsored affiliated researchers can use the IU Account Management Service (AMS):

Note: Accounts remain valid only while the account holder is a registered IU student, or an IU faculty or staff member. On Big Red II, Karst, Quarry, Mason, and the Research Database Complex, accounts are disabled during the semester following the account holder's departure from IU, and then are purged within six months. To exempt a research systems account from disabling, email a request to the Accounts Administration team. If the request is approved, the account will remain activated for one calendar year beyond the user's departure from IU. Then, at the end of the year, the account will be purged. Extensions beyond one year for research accounts are granted only for accounts involved in funded research and having an IU faculty sponsor, or with approval of the Dean or Director of Research and Academic Computing.

Back to top

Accessing Quarry via SSH

Access Quarry using your favorite SSH client; see At IU, what SSH/SFTP clients are supported and where can I get them?

ssh quarry.uits.indiana.edu

Log in with your IU Network ID username and passphrase.

Make sure to read the message of the day (MOTD); it contains news and information regarding the status of the cluster.

Note the following:

  • Windows users: If you use an SSH client in Windows, you cannot open tools that need a graphical user interface (GUI), like the TotalView debugger. You'll need X Window emulation software, such as Cygwin. UITS recommends using XLiveCD created by the Research Technologies division of UITS.

  • X applications: To use the TotalView Debugger/Intel Trace Analyzer, or any other graphical application, from Quarry compute nodes, you must disable X forwarding by specifying the -x flag: ssh -x your_username@quarry.uits.indiana.edu

    To use common graphical applications, such as Emacs, from the head node, do not use the -x option; the default X forwarding must remain enabled.

  • Intra-cluster logins: When you log into your Quarry account for the first time, passphrase-less SSH keys will be automatically created in your home directory. Those keys should enable you to log into compute nodes that you have gained access to through TORQUE without entering a passphrase. Thus, parallel jobs should run seamlessly on multiple compute nodes without any manual intervention.

    However, you may see the following error message when you try to access assigned compute nodes:

    Permission denied (publickey,password,keyboard-interactive)

    This indicates that the intra-cluster RSA key pair in your home directory is either not present or corrupted. If this happens, enter gensshkeys (in /opt/xcat/sbin) from any login node. That will generate a passphrase-less key pair, and allow you seamless intra-cluster logins between any nodes in the cluster assigned for your use by TORQUE.

Back to top

Shell information

On Quarry, the default shell is bash. When you log in for the first time, you will be prompted to select your preferred login shell by the changeshell program:

jdoe@Quarry:~> changeshell This program will assist you in changing your login shell on all nodes of the Quarry cluster. . . . 1) bash 2) tcsh . . . 5) quit Select 1-5: 2 Changing login shell for jdoe Password: Shell changed. Your shell has been changed to the Cornell tcsh shell This will take effect on all nodes within 15 minutes
  • If you choose the Bash, Bourne, or Korn shell, the system will automatically read and execute commands from the /etc/profile and ~/.profile files (plus, for Bash, ~/.bash_profile).

  • If you choose the csh or tcsh shell, the system will automatically read and execute commands from the .login and .cshrc (or .tcshrc) files.

For details, see In Unix, what is the shell? or the Unix man pages.

Back to top

Login, interactive, and compute nodes

Nodes on Quarry are labeled using qXXXX designations (e.g., q0141):

  • Login nodes: When you log into Quarry, you will be placed on one of the system's login (or head) nodes; the node you are assigned is determined on a round-robin basis.

    User processes on login nodes are limited to 20 minutes of CPU time. Processes exceeding this limit are automatically terminated without warning.

  • Interactive nodes: Quarry's interactive nodes are for running interactive jobs that require more than 20 minutes (but less than 24 hours) of processor time. To use an interactive node, log into Quarry, and then SSH from the login node to q0145, q0146, q0147, or q0148; for example: [pamidala@q0143 ~]$ ssh q0145

    If your job requires more than 24 hours of CPU time, prepare a job script, and submit it to run as a batch job on the compute nodes; see Compiling programs and submitting batch jobs to TORQUE below.

  • Compute nodes: Quarry's compute nodes are for running batch jobs. They are accessible by the batch queuing system (and from each other) only; you cannot SSH directly to a compute node.

Back to top

Queue information

The properties of this system's batch job queues are defined to maximize job throughput and minimize the amount of time jobs are queued (i.e., waiting to start). This system employs a TORQUE routing queue to funnel jobs from the default BATCH queue into either the SERIAL, NORMAL, or LONG queue, depending on each job's resource requirements. Your job will run in the SERIAL, NORMAL, or LONG queue unless you specifically submit it to the DEBUG queue by adding either the #PBS -q debug TORQUE directive to your job script or the -q debug switch to your qsub command.

Note: To best meet the needs of all research projects affiliated with Indiana University, the High Performance Systems (HPS) team administers the batch job queues on UITS Research Technologies supercomputers using resource management and job scheduling policies that optimize the overall efficiency and performance of workloads on those systems. If the structure or configuration of the batch queues on any of IU's supercomputing systems does not meet the needs of your research project, fill out and submit the Research Technologies Ask RT for Help form (for "Help Needed", select High Performance Systems job or queue help).

Quarry's batch queues have the following properties:

Maximum walltime: 12 hours
Maximum nodes per job: 1
Maximum cores per job: 8
Maximum number of jobs per queue: 2,000
Maximum number of jobs per user: 500
Direct submission: No
Maximum walltime: 7 days
Maximum nodes per job: 6
Maximum cores per job: 48
Maximum number of jobs per queue: 1,500
Maximum number of jobs per user: 500
Direct submission: No
Maximum walltime: 14 days
Maximum nodes per job: 117
Maximum cores per job: 936
Maximum number of jobs per queue: 550
Maximum number of jobs per user: 75
Direct submission: No
Note: If your job requires more than two weeks of walltime, email the High Performance Systems group for assistance.
Maximum walltime: 30 minutes
Maximum nodes per job: 2
Maximum cores per job: 16
Maximum number of jobs per queue: None
Maximum number of jobs per user: 2
Direct submission: Yes (qsub -q debug)
Note: The DEBUG queue is intended for short, quick-turnaround test jobs requiring less than 30 minutes of wall clock time.

Back to top

Requesting single user time

Although UITS Research Technologies cannot provide dedicated access to an entire compute system during the course of normal operations, "single user time" is made available by request one day a month during each system's regularly scheduled maintenance window to accommodate IU researchers with tasks requiring dedicated access to an entire compute system. To request single user time on one of IU's research computing systems, fill out and submit the Research Technologies Ask RT for Help form (for "Help Needed", select Request to run jobs in single user time on HPS systems). If you have questions about single user time on IU research computing systems, email the HPS team.

Back to top

Managing local and remote files

Like other Unix systems, Linux uses a hierarchical, tree-structured directory system to organize files. Data on local disks reside in journaled file systems that contain a root directory from which associated files and subdirectories branch. Directories are catalogs of files that associate names with files, and are used to segregate files into related groups. To display information about currently mounted file systems, at the shell prompt, enter:


Each file is described by an inode. An inode contains critical information about the file, such as file type (directory, ordinary, character special, block special, or pipe), ownership, access permissions, group ID, file size, file creation, modification, and access times, and a pointer to its data blocks. To list information about the files within your current working directory, at the shell prompt, enter ls -al. Following is a sample of ls output:

$ ls -al drwxr-xr-x 5 jdoe chem 1024 Feb 21 16:06 . drwxr-xr-x 12 bin bin 512 Feb 05 10:56 .. -rw-r--r-- 1 jdoe chem 13250 Feb 14 11:29 .Xdefaults -rw-r--r-- 1 jdoe chem 120 Dec 19 14:41 .aliases.ksh -rwxr----- 1 jdoe chem 24 Jan 13 11:50 .exrc -rw-r--r-- 1 jdoe chem 471 Jan 29 16:37 .profile -rw-r--r-- 1 jdoe chem 255 Jan 29 16:18 .rhosts drwxr-xr-x 3 jdoe chem 512 Dec 19 14:48 bin -rw------- 1 jdoe chem 2054 Feb 21 16:07 .sh_history -rw-r--r-- 1 jdoe chem 37479 Feb 21 14:51 whitepaper.pdf -rwxr-x--- 1 jdoe chem 45 Jan 16 15:22 submit.sh -rwxr-x--- 1 jdoe chem 79 Feb 19 13:02 myjob.c -rwxr-x--- 1 jdoe chem 53 Feb 19 16:45 myjob.c.old -rwxr-xr-x 1 jdoe chem 93 Jan 29 16:10 myjob.o drwxr-xr-x 2 jdoe chem 512 Dec 20 08:41 project1 drwxr-xr-x 2 jdoe chem 512 Dec 20 08:41 project2

The first character in the first field indicates the file type (e.g., - for ordinary file, d for directory). The next nine characters in the first field indicate the file access permissions: r for read, w for write, and x for execute; a hyphen (-) indicates that the corresponding read, write, or execute permission is denied. Characters 2, 3, and 4 indicate the file owner's permissions, characters 5, 6, and 7 indicate the group's permissions, and the last three characters indicate all others' permissions. Execute permission for a directory allows users to search through and list its contents.

Field 2 indicates the number of links to the file. The next two fields show the owner and the group associated with the file. Field 5 gives the size of the file in bytes. Field 6 is the time the file was created or last modified. The last field is the name of the file. The files named . and .. indicate the current and the parent directory, respectively. For more about ls and available options, enter man ls.

To change directories, use the cd command. You can use an absolute or relative path name with the cd command. For example, assume you are currently in the /usr/local directory. To get to the /usr/local/bin directory, you could enter either cd /usr/local/bin (using the absolute path name) or cd bin (using the path name relative to /usr/local). To get back to /usr/local, you could enter cd /usr/local or just cd .. (to go up to the parent directory). Entering cd without an argument will put you into your home directory. To determine what your current working directory is, enter pwd.

Following are some common commands for working with files:

  • To view a file on your screen, enter cat. To view one screen at a time, enter cat | more.

  • To copy a file to the same or a different directory, enter cp. This command overwrites any existing file of the same name. For more information, enter man cp.

  • To rename a file or to move it to a new directory, enter mv. For more information, enter man mv.

  • To delete a file, enter rm. For more about deleting single or multiple files, enter man rm.

  • To create a directory, use the mkdir command. To delete an empty directory, use the rmdir command. For more information, enter man mkdir or man rmdir.

You can also use the commands for managing local files to manage remote NFS files, provided that the access permissions allow it. For example, you cannot use rm on a file in a remote file system that's mounted as read-only.

For access to remote files other than those already accessible to Quarry, you may access a remote host using the Secure Shell scp or sftp commands to copy files to or from the remote system.

To see how many 1 KB blocks of disk space you are using, enter du -k. To see the number of 1 KB blocks in your disk quota, enter quota.

You may exceed your soft quota up to the limit of your hard quota. If you exceed your soft quota, you will receive a message to that effect. You will then have seven days to reduce your use below your soft quota. If you do not reduce your disk usage below your soft quota, you will receive a grace period of seven days, after which you will not be able to write to your disk until you reduce your disk usage below your soft quota.

Back to top

Editing files

Using vi

Before entering vi, make sure your environment variable TERM is set correctly. When vi starts, it looks at TERM to see what kind of terminal you are using. It then loads the proper terminal control information from /etc/termcap (the terminal capabilities file). Enter the appropriate commands below depending on your shell:

  • If you use the Bourne (sh) or Korn (ksh) shell: $ TERM=vt100 $ export TERM $ vi filename
  • If you use the C shell (csh): % setenv TERM vt100 % vi filename

For information about common vi commands, see How do I use the vi text editor?, A quick reference list of vi editor commands, or In vi, how can I access deleted text?

You can find more information in Learning the vi Editor by Linda Lamb (O'Reilly and Associates, Inc., 1992).

Using Emacs

GNU Emacs is a version of Emacs written by the author of the original (PDP-10) Emacs, Richard Stallman. GNU Emacs retains all the functionality of other Emacs editors, but it is also customizable and extensible by modifying its Lisp editing commands. GNU Emacs is designed to work well with X Window systems, but it also functions with ASCII terminals.

The account creation process on Quarry copies a .emacs file into the newly created home directory:

[jdoe@Quarry] cat .emacs ;; .emacs ;;; uncomment this line to disable loading of "default.el" at startup ;; (setq inhibit-default-init t) ;; turn on font-lock mode (when (fboundp 'global-font-lock-mode) (global-font-lock-mode t)) ;; enable visual feedback on selections ;(setq transient-mark-mode t) ;; default to better frame titles (setq frame-title-format (concat "%b - emacs@" system-name))

Because GNU Emacs uses the Ctrl-q and Ctrl-s key sequences as commands, disable the use of Ctrl-q and Ctrl-s as terminal output start/stop signals before entering GNU Emacs. To do this, enter stty -ixon. After completing your GNU Emacs session, you may re-enable terminal start/stop signals by entering stty ixon.

GNU Emacs has an extensive interactive help facility, but to use it you must know how to manipulate Emacs windows and buffers. (For help with the Emacs notation that follows, see In Emacs, how are keystrokes denoted?) To enter Emacs help, with an Emacs window open, press C-h. To access an interactive tutorial that teaches the fundamentals of Emacs, press C-h t. To find a command given its functionality, press C-h a. The Help Character command (C-h c) describes a given character's effect, and the Help Function command (C-h f) describes a given Lisp function specified by name.

For information about common Emacs commands, see GNU Emacs Quick Reference Guide.

Back to top

Command files (shell scripts)

Shell scripts are command files interpreted by a shell. These scripts may contain any statements valid in the shell in which they are interpreted. You can use them to perform repetitive tasks, such as setting up your environment, compiling and executing programs, submitting batch jobs, and performing system management tasks. Locally written shell script utilities are contained in /usr/local/bin.

Shell programming constructs include variable assignment, argument handling, evaluation of integer expressions, conditional execution, flow control, file status checking, string and integer comparisons, reading of input, interrupt signal traps, and menu screen generation.

The following example of a C shell script illustrates some basic shell programming constructs.

C shell script

# Scriptname: showuser # Purpose: display information about a user set USAGE = "Usage: showuser userid" # Test for argument if ($#argv > 1) then echo $USAGE exit 1 endif if ($#argv == 1) then set USERNAME = $1 else if ($#argv == 0) then echo "Enter the user's login name: " set USERNAME = $< endif set IFS = ":" # internal field separator # Use sed to edit /etc/passwd, putting blank in empty field sed "s/$IFS/ /g" < /etc/passwd > $HOME/passwd echo "_________________________________________________________" echo "Userid UID GID Full Name Home Directory Login Shell" echo " " # Use awk to find the correct line and print it out awk "/$USERNAME/" $HOME/passwd echo "_________________________________________________________" rm $HOME/passwd exit

The following reference guides may be of interest:

Back to top

Using Modules to set up your software environment

On the research computing resources at Indiana University, the Modules environment management system provides a convenient method for dynamically customizing your software environment.

Some common Modules commands include:

Command Action
module avail List all software packages available on the system.
module avail package List all versions of package available on the system, for example:

module avail openmpi
module list List all packages currently loaded in your environment.
module load package/version Add the specified version of the package to your environment, for example:

module load intel/11.1

To load the default version of the package, use:

module load package
module unload package Remove the specified package from your environment.
module swap package_A package_B Swap the loaded package (package_A) with another package (package_B).

This is synonymous with:

module switch package_A package_B
module show package Shows what changes will be made to your environment (e.g., paths to libraries and executables) by loading the specified package.

This is synonymous with:

module display package

To make permanent changes to your environment, edit your ~/.modules file. For more, see In Modules, how do I save my environment with a .modules file?

For more about the Modules package, see the module manual page and the modulefile manual page. Additionally, see On Big Red II, Karst, Quarry, Mason, and Rockhopper at IU, how do I use Modules to manage my software environment?

Back to top

Parallel computing and MPI

The Quarry cluster is equipped with a proprietary low-latency interconnect. Parallel jobs can be run over gigabit Ethernet; the cluster is connected to a Force10 E1200 switch.

To see which versions of Open MPI and MPICH are currently available, use the appropriate module avail command:

module avail mpich module avail openmpi

Note: Open MPI and MPICH compiled with the 64-bit Intel v10 compilers are available. The MPICH package is currently available compiled only with the Intel compiler suite.

To add either package to your environment:

  1. Add the necessary compiler suite to your environment:

    • GNU Compiler Collection (GCC): module load gcc
    • Intel Compiler Suite: module load intel
  2. Load the module for the appropriate package, compiler suite (suite), and version (<version>):

    • MPICH: module load mpich/<suite>/<version>
    • Open MPI: module load openmpi/<suite>/<version>

Briefly, use mpicc or mpif77 to compile and link MPI-enabled software. Then, use mpirun to execute the job on multiple nodes of the cluster. In practice, you must submit your job through the TORQUE resource manager to ensure fair and efficient use of the cluster.

Note: On Quarry, Moab serves as the job scheduler for the TORQUE resource manager. For information on Moab, see:

Back to top

Compiling programs and submitting batch jobs to TORQUE

Two simple programs are listed below for your convenience:

C program

/* * Copyright 2005, The Trustees of Indiana University. * Original author: Arvind Gopu (UITS-RAC-HPC group)... * . . . [ snip ] . . . */ #include<stdio.h> #include<math.h> int main () { double PI2=3.141592654/2.0, theta, sintheta; int i, N=4; for (i=0; i<=N; i++) { theta = i * (PI2/N); sintheta = sin (theta); printf (" Sin (%8.6lf) = %8.6lf \n", theta, sintheta); } return 0; }

Fortran program

C C Copyright 2005, The Trustees of Indiana University. C Original author: Don Berry (UITS-RAC-HPC group); C . . . [ snip ] . . . C program sine real :: PI2=3.141592654/2.0 integer :: N=4 real x,s integer i do i=0,N x=i*(PI2/N) s=sin(x) write(6,"(f11.6 f11.6)") x,s end do end

These commands use the GNU compilers (gcc and g77 for C and Fortran programs, respectively) to compile the two example programs:

jdoe@Quarry: ~> gcc -o sine_c sine.c -lm jdoe@Quarry: ~> g77 -o sine_f sine.f

Submit job(s) to TORQUE

It's best to write your job in its own script file, and tell TORQUE to execute that. Following is a script that runs the example sine_c program:

[jdoe@Quarry]$ cat submit_sine_c.sh #PBS -l nodes=1:ppn=1,walltime=5:00 #PBS -m ae #PBS -N job_sine_c ${HOME}/simple_quarry_jobs/sine_c

Use the script to submit your job to TORQUE:

[jdoe@Quarry]$ qsub submit_sine_c.sh 95.m2.quarry.teragrid.iu.edu

Check the job status with the qstat command:

[jdoe@Quarry]$ qstat | grep jdoe [jdoe@Quarry]$ qstat -u jdoe Job ID Username Queue Jobname SessID NDS TSK Memory Time S Time --------------- -------- -------- ---------- ------ --- --- ------ ----- - ----- 95.m2.quarry.teragri jdoe batch job_sine_c -- 1 1 -- 00:05 R -- 96.m2.quarry.teragri jdoe batch job_sine_f -- 1 1 -- 00:05 Q --

Note: Quarry will email job-related messages to the address specified in the ~/.forward file in your home directory. (Note the period [.] preceding the filename.) By default, this is the email address you provided when you requested your account. To change this email address, on the command line, enter (replace username@host.com with your email address):

echo "username@host.com" > ~/.forward

For more about submitting and tracking jobs, see What is TORQUE, and how do I use it to submit and manage jobs on high-performance computing systems?

Output and error files

Assuming your job runs to completion, you can find messages it tried to print on the console in an output file. You can specify the directories where the output file and the error file go using the -o and -e flags, respectively. The default file name of these files is of the format job_name.osequence_number and job_name.esequence_number, where job_name is the name of the job (check out the -N option) and sequence_number is the job number assigned when the job is submitted.

For example, if TORQUE assigned job id 95.m2.quarry.teragrid.iu.edu to your job, with jobname job_sine_c, the output file would be named job_sine_c.o95 and the error file would be named job_sine_c.e95. (Also see the use of the -j TORQUE directive as explained in the section above.)

[jdoe@Quarry]$ ls job_sine_* job_sine_c.e348921 job_sine_c.o348921 job_sine_f.e348922 job_sine_f.o348922

Check the output files:

[jdoe@Quarry]$ cat job_sine_c.o348921 Sin (0.000000) = 0.000000 Sin (0.392699) = 0.382683 Sin (0.785398) = 0.707107 Sin (1.178097) = 0.923880 Sin (1.570796) = 1.000000 [jdoe@Quarry]$ cat job_sine_f.o348922 0.000000 0.000000 0.392699 0.382683 0.785398 0.707107 1.178097 0.923880 1.570796 1.000000

Back to top

Compiling and submitting parallel jobs

Here is a simple C program you can submit to use multiple processors:

#include <stdio.h> #include <mpi.h> /* #include "VT.h" */ /* only needed for Vampir Trace */ int main( int argc, char *argv[] ) { /* Remember: We are programming according to the SPMD model! */ /* This means that each and every processor executes this program. */ /* It is convenient to reference the typical processor in the */ /* first person, using "I" and "me" and "my." In other words, */ /* read through this program as if you were one of the processors. */ /* My ID number or my "rank": */ int myrank, length; char myname[BUFSIZ]; /* I initialize for myself the MPI API: */ MPI_Init(&argc, &argv); /* Who am I? I request my ID number: */ MPI_Comm_rank(MPI_COMM_WORLD, &myrank); MPI_Get_processor_name (myname, &length); //VT_symdef( 200, "for-loop", "Calculation" ); /* I print the standard greeting along with my ID: */ printf( "Hello, parallel worlds! This is processor %s and my rank is %d!\n", myname, myrank ); /* Finally I close the MPI API: */ MPI_Finalize(); return 0; }

Here is the script file to submit the job:

#! /bin/bash #PBS -l nodes=1:ppn=4,walltime=1:00 #PBS -m ea #PBS -N My_mpi_job # Get the number of processors NP=`wc -l $PBS_NODEFILE | awk '{print $1}'` # print out the nodes where the job runs echo "Execute node list:" sort -u $PBS_NODEFILE # change to the working directory cd simple_quarry_jobs # run an mpi job: mpirun -np $NP -machinefile $PBS_NODEFILE helloWorlds

Compile and submit:

[jdoe@Quarry]$ mpicc -o helloWorlds helloWorlds.c [jdoe@Quarry]$ qsub submit_parallel.sh 97.m2.quarry.teragrid.iu.edu

Look at the output file:

[jdoe@Quarry]$ cat job_helloWorlds.o348923 LAM 7.1.1/MPI 2 C++/ROMIO - Indiana University Hello, parallel worlds! This is processor bc27 and my rank is 1! Hello, parallel worlds! This is processor bc25 and my rank is 2! Hello, parallel worlds! This is processor bc27 and my rank is 0! Hello, parallel worlds! This is processor bc25 and my rank is 3! LAM 7.1.1/MPI 2 C++/ROMIO - Indiana University

Back to top

Usage policies

For information about your responsibilities as a computer user at IU, see:

Back to top

Acknowledging grant support

The Indiana University cyberinfrastructure managed by the Research Technologies division of UITS is supported by funding from several grants, each of which requires you to acknowledge its support in all presentations and published works stemming from research it has helped to fund. Conscientious acknowledgment of support from past grants also enhances the chances of IU's research community securing funding from grants in the future. For the acknowledgment statement(s) required for scholarly printed works, web pages, talks, online publications, and other presentations that make use of this and/or other grant-funded systems at IU, see If I use IU's research cyberinfrastructure, what sources of funding do I need to acknowledge in my published work?

Back to top


Support for research computing systems at Indiana University is provided by various units within the Systems area of the Research Technologies division of UITS:

To ask any other question about Research Technologies systems and services, use the Request help or information form.

Back to top