Access and Usage


In the following sections, john is used as an example username. Be sure to replace this with your own Michigan Tech ISO username.

If superior-login1.research.mtu.edu is unavailable, please use
superior-login2.research.mtu.edu.

Superior is a shared computational facility with near-zero GUI support. As a result, analysis, visualization and manuscript preparation, etc., will need to be done in a local workstation. If not yet familiar with using Linux OS command line, please contact Dr. Gowtham to enroll in the following free online course.

FOSS 101: Essentials of Free and Open Source Tools



SSH access

SSH from a terminal is the only allowed protocol to log in - other insecure protocols (telnet, rlogin, rsh, ftp, etc.) are unsupported and disabled by default.


ssh john@superior-login1.research.mtu.edu


Supply ISO password when prompted. Use VPN if connecting from an off-campus network.

Sharing login credentials (username and/or password) with other and/or unauthorized users is a violation of Acceptable Use of Information Technologies. Using this infrastructure in a manner that violates the above and/or other provisions described in the aforementioned policy will lead to immediate suspension of account to protect the integrity of the system and to curtail abuse. To the extent computer usage is believed to be a violation of federal, state, or local laws, Michigan Technological University will turn the matter over to the appropriate authorities.

Physical access to the computing infrastructure is by request only and all users will be accompanied by an authorized staff member of Information Technology.

Two-factor authentication (optional)

This optional feature, Two-Factor Authentication, expects that the user has a smart phone (Android or iPhone) with Google Authenticator app installed. The procedure is detailed below:


  1. Install the Google Authenticator (it's free) using an appropriate link below:

    Android | iPhone

  2. SSH into superior-login1.research.mtu.edu.
  3. Run google-authenticator.


  4. Use Scan Barcode in Google Authenticator app to capture the QR code.



  5. Make a note of the secret key, verification code and emergency scratch codes in a secure place.

  6. Choose y as an answer for all subsequent questions.


  7. Inform Information Technology so that the administrators can complete the necessary steps.

Transferring files to and from Superior

Suppose that all necessary files (and/or folders) to run a (set of) simulation(s) are located in the campus home directory, in ${HOME}/project_X. Then, the following command - when run from any linux machine that is maintained by Information Technology (e.g., colossus.it.mtu.edu, guardian.it.mtu.edu or other machine in a lab) - will securely transfer them, as is, to Superior:

export SUPERIOR="superior-login1.research.mtu.edu"
rsync -ave ssh -hPz \
${HOME}/project_X/ john@${SUPERIOR}:/research/john/project_X/

When the (set of) simulation(s) have been successfully completed, the following command - when run from any linux machine that is maintained by Information Technology (e.g., colossus.it.mtu.edu, guardian.it.mtu.edu or other machine in a lab) - will securely transfer them, as is, from Superior:

export SUPERIOR="superior-login1.research.mtu.edu"
rsync -ave ssh -hPz \
john@${SUPERIOR}:/research/john/project_X/ ${HOME}/project_X/

Software suites

Free and open source research software suites as well as the ones that are campus-licensed will be available for all researchers. Commercial software suites that are paid for by specific research groups will be available for exclusive use by respective research groups.

Information Technology will integrate the pre-compiled binaries and/or software suites for which there is only one way to compile/install with the queuing system. If the compilation requires research/project-specific options, flags, etc., then researchers will need to share their approach and detailed notes in order for the binary/suite to be integrated with the queuing system.


# Name Version Compiler Serial Parallel GPU
01 Abaqus 6.14-3 --
02 ANSYS EM 16.2 -- --
03 ANSYS FLUENT 16.2 -- --
04 ANSYS FLUENT 17.1 -- --
05 ANSYS HFSS 16.2 -- --
06 CALYPSO 3.6 Intel 2013.0.028 --
07 CMake 3.8.0 -- -- -- --
08 COMSOL 5.2 --
09 CONVERGE 2.2.0 --
10 CONVERGE 2.2.0b --
11 CORSIKA 7.4xxx Intel 2013.0.028
12 CRYSTAL 14 1.0.3 Intel 2016.1
13 CST 2015 --
14 FEBio 2.4.2 -- -- --
15 Field II 3.24 -- -- --
16 Gaussian 09 D.01 --
17 GCC 4.4.6 -- -- -- --
18 GCC 5.4.0 -- -- -- --
19 GCC 6.3.0 -- -- -- --
20 GEOS-CHEM 9.0.x Intel 2013.0.028
21 Gnuplot 5.0.5 GCC 4.4.6 -- --
22 GULP 4.4 Intel 2016.1
23 IDL 8.2 SP1 -- -- -- --
24 Intel Cluster Studio XE 2013.0.028 -- -- -- --
25 Intel Parallel Studio XE 2016.1 -- -- -- --
26 Intel Parallel Studio XE 2017.3 -- -- -- --
27 k-Wave (MATLAB Toolbox) 1.1 -- --
28 LAMMPS 2015.10.20 Intel 2013.0.028
29 LAMMPS 2016.02.29 Intel 2016.1
30 LS-DYNA 9.1.0 (SMP) --
31 Maple 2015.1 --
32 Mathematica 11.0.1.0 --
33 MATLAB R2014b --
34 MATLAB R2015b --
35 MATLAB R2016b --
36 MATLAB R2017a --
37 ModelE -- Intel 2013.0.028
38 MOLPRO 2012.1 p3 --
39 NAMD 2.11 Intel 2016.1
40 NVIDIA CUDA 6.5 -- -- -- --
41 OpenFOAM 1.6-ext Intel 2013.0.028 --
42 OpenFOAM 2.2.1 Intel 2013.0.028 --
43 PERL 5.10.1 -- -- -- --
44 Python 2.7.11 GCC 4.4.6 -- -- --
45 Python 3.5.1 GCC 4.4.6 -- -- --
46 Quantum ESPRESSO 5.3.0 Intel 2016.1
47 R 3.2.3 Intel 2016.1
48 R 3.2.5 Intel 2016.1
49 SeaDAS/BEAM 7.3.2/5.0 --
50 SIESTA/TranSIESTA 3.2-pl3 Intel 2013.0.028
51 SIESTA/TranSIESTA trunk-444 Intel 2013.0.028
52 SOFI2D -- Intel 2016.1 --
53 SOFI2D SH -- Intel 2016.1 --
54 SOFI3D -- Intel 2016.1 --
55 Towhee 7.1.0 Intel 2016.1
56 VASP 5.3.3 Intel 2013.0.028
57 VASP 5.3.5 Intel 2013.0.028
58 VASP 5.4.4 Intel 2016.1

Queues and scheduling policy

  1. short.q (compute-0-N; N: 0-1)

    While there is no limit on how many processors any given simulation can use, the wall time is restricted to 24 hours. This queue has the shortest turnaround time and is expected to serve as a sand box for researchers to test the maturity of their program.


  2. long.q (compute-0-N; N: 2-67)

    There is neither a limit on how long a given simulation can run nor how many processors it can use. Researchers will have access to this queue by default.


  3. gmodegar.q (compute-0-N; N: 72-75)
    lvalenza.q (compute-0-N; N: 68-71)
    ongbw.q (compute-0-N; 92-95)
    pexue.q (compute-0-N; 76-91)

    Specific researchers/institutes have paid for acquisition of compute nodes in these queues. Unless explicitly indicated, researchers are expected to submit short term (e.g., ~4 hours or less) simulations to thsese queues and these simulations will generally have longer wait times. Exclusive compute node feature is also disabled in these queues.


  4. gpu.q (compute-0-N; N: 96-98)

    This queue, with each node having four NVIDIA Tesla M2090s, is exclusively for running software suites that can exploit the power of GPU. While there are no limits on how long a given computation can run and how many CUDA cores a given computation can use, only one computation can be run per GPU at any given time. Researchers will have access to this queue by default.


  5. qlogin.q (compute-0-N; N: 99)

    This queue has four NVIDIA Tesla M2090 GPUs, is exclusively for compiling programs and testing scripts/utilities, etc. from within a qlogin session.


A researcher group's priority for subsequent simulations in Superior is computed periodically taking the group's production (e.g., usage etiquette, total CPU time used, number of publications that cite this facility, number of advanced degrees that have resulted from use of this facility, external funding brought in to support this facility, etc.) into account. As such, each research group is encouraged to designate a point of contact (e.g., a graduate student) to update Dr. Perger with this information as detailed here.

Pre-emption (the act of temporarily interrupting/suspending a task, with the intention of resuming it at a later point in time), and advanced reservation are disabled on Superior. In other words, any simulation that starts will complete, unless it has reached the wall time limit (for e.g., running on short.q) or it has been stopped [terminated by the researcher, terminated by the administrator with researcher's approval, power cycling of node(s), bug in the source code/compiler, etc.].

${HOME}/.bash* files

${HOME}/.bash_profile and ${HOME}/.bashrc have been designed to ensure all researchers have required access to necessary software suites. As such, editing them under any circumstance is very highly discouraged. However, researchers may put their own shell customizations (e.g., aliases, functions, variables, etc.) in

${HOME}/.bash_${USER}

Running set or env to see a list of aliases, functions, variables already in use will help minimize the chance of altering their definition. This, in turn, will minimize the chance of software suites not working as intended.

Running programs in login node(s)

When a researcher runs a program on Front End/Login Node(s), either inadvertently or intentionally, the system sends a warning email. When the same researcher is found to be running a program on Front End/Login Node(s) even after the first warning, the system logs the researcher out without further warning and blocks access until further notice. A meeting with the researcher (and/or her/his advisor) will be necessary before the researcher can access Superior again.

Researchers are strongly encouraged to use less, more, grep or qlogin, etc. to analyze (larger) files as using vi (or vim) can use a large percentage of CPU and memory, and in turn can result in a violation.

Running programs on Front End/Login Node(s) can cause serious problems such as drastically reduced performance for everyone, corrupted file system, missing features, prevent researchers from logging in and shutting the cluster down. In the event that Information Technology has to rebuild the cluster from scratch to repair these symptoms, it will take considerable amount of time and researchers using this cluster might not be able to meet their project deadlines.

Job/Batch submission scripts

qgenscript is a fairly extensive interactive utility that provides necessary script for all computational science/engineering suites available in Superior and should be called from the folder where all the necessary files (or folders) to run the simulation are located. For e.g.,

cd /research/john/project_X
qgenscript

Apart from generating a script based on the researcher-supplied information, qgenscript also displays the wait time statistics and the command necessary to submit that script to the queue. For every simulation, researchers can opt to receive status notification emails (begin, end, abort/kill and suspend). Submitted simulations can often stay in qw mode for longer than normal periods of time, depending on the availability of resources (processors, memory, number of licenses, etc.).



Array jobs
Suppose that a project involves running a large number of simulations, say 1000, and suppose that all these simulations use the same software suite & command but run on different sets of data, named input_1.txt, input_2.txt, input_3.txt, ..., input_1000.txt.

Using qgenscript a thousand times to generate the required job/batch submission script for each simulation becomes a very tedious, time consuming and inefficient process. It also makes monitoring (or deleting) them a hassle.

To facilitate an efficient handling of such a scenario, qgenscript offers an optional feature with which researchers may specify that a given simulation is an array job and then specify the range (i.e., 1-1000; 1, 2, 3, ..., 1000 are then referred to as task ID to identify a specific data set).

The result is just one batch submission script that not only makes it easier to submit and monitor the simulation (or terminate some/all tasks, if need be) but also puts significantly less burden on the queuing system. As many of the tasks will run concurrently as possible, provided all requested resources are available. Output and other files will also be tagged with respective task ID for unique identification. Researchers must note that the queuing system will send email notificaitons for each task.

Submission script modification
It is critical that researchers do not edit submission scripts generated by the qgenscript utility and/or re-use/re-purpose an old one for a newer simulation as an inadvertent mistake can cause unnecessary delays.

Should there be a need to edit the submission script OR create a custom submission script to meet a specific need (i.e., array jobs scenario is more complex than the one described above; e.g., run iff the output file for a given task ID is not present, select jth line from a file as a parameter for jth task, using task ID itself as a parameter for a function call, and so on) OR generate a large number of submission scripts (e.g., parameter sweep scenario that doesn't lend itself as an array job), researchers should contact Dr. Gowtham.

qlogin: Compiling programs & minimal analysis

When there is a need to compile programs and/or utilities, and/or run scripts/tools/utilities to create the input files and/or analyze the output files, researchers are expected to do so in one of the compute nodes. Using the following command

qlogin

will not only grant legitimate access to an otherwise inaccessible compute node but also protects the researcher from potential violations, as described in Running programs in login node(s) section.

Depending on the availability of resources, the number of concurrent qlogin sessions (2), the duration of each such session (4 hours) and/or the number of processors per session (2) might be limited. Researchers may refer to the Modules section to learn more about the default set of compilers, libraries and software suites available in a qlogin session.

Modules (to manage the user environment)

Modules system lets researchers easily specify one or more software that needs to be used. A naive task of loading the module will implictly set all necessary environment variables (e.g., PATH, LD_LIBRARY_PATH, MANPATH, etc.). By the same taken, unloading the module will implicity unset the environment variables with ease.

Modules are particularly useful to researchers (frequently) compiling their own (or third party) software prior to running simulations. Researchers may note that submission scripts generated by the qgenscript utility (except for Custom serial program, Custom parallel program, Custom BASH script and Custom Python script) already include the module implementation and no additional work on researchers' part is necessary.

By design, modules are not available for use in the login nodes -- a proactive measure to ensure researchers do not inadvertantly compile or run programs (which, in turn, may lead to account lockouts). Once within a qlogin session, the following modules are loaded for all researchers by default.

  gnuplot/5.0.6
  intel/2016.1
  mathematica/11.0.1.0
  matlab/R2017a
  r/3.2.5


GCC 4.4.6, PERL 5.10.1 and Python 2.6.6 -- installed as part of the Operating System -- are also available by default. Researchers may load/unload additional modules as necessary. Following are some of the commonly used module commands/actions.



  1. module avail

    Displays all available packages and versions. Default version of a specific software, in case of multiple available versions, is marked with D. A package that is loaded is indicated by L


  2. module list

    Displays all loaded packages


  3. module avail PACKAGE

    Displays all available versions of PACKAGE. For e.g., module avail matlab


  4. module load PACKAGE/VERSION

    Loads the VERSION of PACKAGE. If the VERSION is omitted, then the default version of the PACKAGE (as indicated by module avail) will be loaded.

    For e.g., module load matlab/R2014b will load MATLAB R2014b whereas module load matlab will load MATLAB R2017a. As such, it is recommended that researchers explicitly specify the version to avoid potential mistakes/errors in results


  5. module unload PACKAGE

    Unloads the PACKAGE. For e.g., module unload matlab. Though not necessary, it may still be a good practice to specify the VERSION while unloading a package. For e.g., module unload matlab/R2017a


  6. module purge

    Unloads all loaded packages


  7. module save PROJECT_NAME

    Saves currently loaded packages in PROJECT_NAME environment for later re-use. PROJECT_NAME is optional but using it will help for easier classification of modules as required by different projects. This action survives logging out and logging back into a qlogin session.

    For e.g., suppose that intel/2016.1, r/3.2.3 and matlab/R2014b are required for a given project, say Secret Project. Instead of having to unload the default modules and load these three every single time, the following commands will save them in the PROJECT_NAME environment for easier later re-use.

      module list
      module purge
      module load intel/2016.1
      module load r/3.2.3
      module load matlab/R2014b
      module list
      module save PROJECT_NAME


  8. module restore PROJECT_NAME

    Restores the PROJECT_NAME environment and loads all its packages.

    For e.g., suppose that intel/2016.1, r/3.2.3 and matlab/R2014b are required for a given project, say Secret Project. Further suppose that these packages were saved in PROJECT_NAME environment. The following commands will restore the environment and list loaded packages.

      module restore PROJECT_NAME
      module list


    One may modify the list of loaded packages and save environment using the same name to modify it


  9. module help

    Displays all module commands


  10. module help PACKAGE/VERSION

    Displays additional information, if any, about VERSION of PACKAGE. If the VERSION is omitted, then additional information is displayed for the default version of the PACKAGE (as indicated by module avail).

    For e.g., module help matlab/R2014b will show information related to MATLAB R2014b whereas module help matlab will show that about MATLAB R2017a. As such, it is recommended that researchers explicitly specify the version to avoid potential mistakes/errors


Researchers may also create and manage their own private modules, and may refer to TACC's Advanced User Guide and More About Writing Module Files for additional information.

Adding missing R packages/libraries

Researchers using R, a free and open source software for statistical computing, may use the following workflow to add any missing packages/libraries within a qlogin session. Replace intel/2016.1 and r/3.2.5 with desired versions of Intel compiler and R respectively. Also, replace PKG with name of the desired package/library.

  mkdir -p /research/${USER}/r/3.2.5/library
  cat <<EOL >> ${HOME}/.bash_${USER}
  export R_LIBS_USER="/research/${USER}/r/3.2.5/library"
  EOL
  source ${HOME}/.bashrc

  module purge
  module load intel/2016.1
  module load r/3.2.5
  module list


After ensuring that intel/2016.1 and r/3.2.5 are indeed loaded, run the following command (all in one line).

  Rscript -e "install.packages('PKG', lib='${R_LIBS_USER}', contriburl=contrib.url('http://cran.mtu.edu/'))"

If the above command terminated successfully, the package/library PKG should have been installed. The following command can be used as a further check.

  Rscript -e "library('PKG')"

As a test of this entire workflow, replace PKG with one of the following and ensure that it actually works: gstat, ncf, pROC, splancs or yacca.

Disk usage and data backp

Superior is not backed up and researchers are responsible for their data. Owing to limited storage space, Superior is set up in such that every researcher, barring none, will get an email once the respective HOME (10 MB) and RESEARCH (600 GB) folders exceeds a pre-set limit - and will continue to get such emails few times a day until disk usage gets below the set limit. A sample notification is given below:



From DoNotReply@mtu.edu Sun Jan 12 12:24:24 2014
Date: Sun, 12 Jan 2014 12:22:24
From: "Admin @ superior.research.mtu.edu" 
To: john@mtu.edu
Cc: YOUR-ADVISOR@mtu.edu
Subject: superior.research: Research Disk Usage Reminder #1


  Reminder #      : 1
  Date/Time       : Sun, 12 Jan 2014 12:22:24
  Hostname        : superior.research.mtu.edu
  User            : john
  Research folder : /research/john
  Limit           : 600 GB
  Current usage   : 660 GB (+10 %)

You have about 4 hours before the next reminder. Please make 
time to clean up your research directory.

IMPORTANT NOTE: 
At reminder #5, you will be logged out, your account will be 
locked and all your simulations will be terminated without 
further notice.

Do not reply to this email OR forward this email to 
it-help@mtu.edu (or anyone else in IT).


After the 3rd consecutive reminder for home partition and 5th consecutive reminder for research partition, the researcher's account will be temporarily suspended and researcher's simulations will be terminated. A request from the researcher's advisor will be necessary to re-enable the account.



From DoNotReply@mtu.edu Wed Jan 15 12:21:18 2014
Date: Wed, 15 Jan 2014 12:21:18
From: "Admin @ superior.research.mtu.edu" 
To: john@mtu.edu
Cc: YOUR-ADVISOR@mtu.edu
Subject: superior.research: Research Disk Usage Reminder #5


  Reminder #      : 5
  Date/Time       : Wed, 15 Jan 2014 12:21:18
  Hostname        : superior.research.mtu.edu
  User            : john
  Research folder : /research/john
  Limit           : 600 GB
  Current usage   : 660 GB (+10 %)


IMPORTANT NOTE: 
At reminder #5, you will be logged out, your account will be 
locked and all your simulations will be terminated without 
further notice.

 
This is reminder #5. You have been logged out and your account 
has been locked. Also, all your simulations have been terminated. 
Request your advisor to open a ticket in ServiceDesk Plus portal 
to have your account enabled.
 
  1. Open http://superior.research.mtu.edu/help in a web browser
  2. Log in with ISO credentials
  3. Update 'Subject:' as 
     'superior.research: Re-activate access for john'
 
Do not reply to this email OR forward this email to
it-help@mtu.edu (or anyone else in IT).

Useful commands

 
Built-in
qdel SIM_ID Delete a simulation from the queue
qhold SIM_ID Hold/Pause a running/waiting simulation
qlogin Log into a compute node (for compilation, quick analysis, etc.)
qrls SIM_ID Release a simulation from hold
qstat -j SIM_ID More information about a running/waiting simulation
qsub SCRIPT Submit SCRIPT (generated by qgenscript) to the queue