Computing Resources

From ComputingWiki

Jump to: navigation, search

Computing Resources

Contents

[edit]

Headnode

  • headnode.stat.iastate.edu is a Linux machine used for submitting computationally intensive jobs to computation nodes.
  • Every faculty and graduate student has a folder in the /Users folder.
  1. /Users is hosted on a 1.1 TB disk partition
  2. Files are backed up to another 1.4 TB disk partition, accessible via the backups link in your /Users folder.
  3. You can access your /Users folder from Windows Explorer using \\headnode.stat.iastate.edu\NET-ID where NET-ID is your ISU netid
  • From a Mac you would use smb://headnode.stat.iastate.edu/NET-ID
  • You can SSH into the machine using port 323 and your ISU net-id and password.
  • The headnode is not to be used to run your programs. You can compile your programs but testing should be done on another of our Linux machines.
[edit]

Subversion

  • Subversion is a Revision Control System that runs on headnode.stat.iastate.edu
    • A Revision Control System (RCS) allows users to ....
      • Collaborate on programming projects
      • Save revisions of files/projects
      • Remote checkout/update of files
    • Example 1: Thesis
      • You use TexnicCenter on Windows to edit your thesis but you use R code to generate plots that you want to include.
        • Create thesis project on headnode
        • Check in your R source, any plots, and any other accompanying files (except large data sets).
        • Modify, and run your R code on the Linux machine.
        • Update the project on headnode
        • Update the same project on the Windows machine and you now have all the plots and updated R source.
    • Example 2: Programming project
      • You are working with 3 other students on code for your major professor.
        • Contact Ted Peterson who will...
          • Create a repository for the project and set group permissions allowing read/write access to the group.
      • Now you can work independently on the project and let Subversion deal with updating files with mods from more than one user
  • How to use:
  1. SSH to headnbode.stat.iastate.edu and make a directory named svn in you /Users/NET-ID directory
    1. ssh -p 323 NET-ID@headnode.stat.iastate.edu
    2. cd /Users/NET-ID
    3. mkdir svn
  2. Create a project
    1. svnadmin create --fs-type fsfs /Users/NET-ID/svn/PROJECT-NAME
  3. If you already have a directory where the files for PROJECT-NAME live then import them.
    1. svn import "Directory where PROJECT-NAME lives" file:///Users/NET-ID/svn/PROJECT-NAME -m "initial version"
  4. Now create a working directory for PROJECT-NAME
    1. mkdir -p /Users/NET-ID/working/PROJECT-NAME
  5. Now checkout the project
    1. cd /Users/NET-ID/working/PROJECT-NAME
    2. svn checkout file:///Users/NWET-ID/svn/PROJECT-NAME
  6. Now let's schedule a file to be added to the project
    1. cat date > date.txt
    2. svn add date.txt
    3. If you add a directory to the project all items underneath it will also be added
  7. To delete an item
    1. svn delete date.txt
  8. Add a new, empty directory to the project ( same as: mkdir floo; svn add floo )
    1. svn mkdir floo
  9. Commit changes
    1. svn commit -m "A few changes to date.txt and new floo"
  10. Get changes
    1. svn update
  • What if I am on another machine?
  1. We need to tell Subversion that our machines do not listen to the default SSH port
    1. bash users: export SVN_SSH="/usr/bin/ssh -p 323"
    2. csh/tsh users: setenv SVN_SSH="/usr/bin/ssh -p 323"
  2. Change to the working directory for the project
    1. cd WORKING-DIR
  3. Checkout the project
    1. svn checkout svn+ssh://NET-ID@headnode.stat.iastate.edu/Users/NET-ID/svn/PROJECT-NAME
  4. The commands to update, commit, etc are the same, just change the file:/// to svn+ssh://NET-ID@headnode.stat.iastate.edu/
  • NOTES
  1. Don't have more than one working copy of the project unless you make sure you always commit your changes
  2. There are Windows clients available. A demo of TortoiseSVN can ne found here
[edit]

Computation Cluster

  1. There are currently 32 CPU's available for use. They are shared amongst the primary investigators (PIs), their students and the rest of the department.
  2. The PIs and their students will not necessarily be using the scheduling software to run jobs. Every attempt is being made to accommodate both sets of users, please contact Ted Peterson with questions.
  3. The number of jobs a user may submit are bound by the following rules:
    1. No more than 8 jobs at a time, and
    2. No more than 1/2 of the remaining empty slots
    3. If your number of jobs exceeds the above then the script will go to sleep for 10 minutes and then try to submit more of your jobs
  4. To run a job on the computation cluster do the following:
    1. Copy the source files (R,C,C++,Fortran) to your Users folder on the headnode. You may want to use sub folders to keep your work separate.
    2. Create a file, say 'jobs' with the following content (separated by '|')
      1. Field1: Type of job (currently only needs an R if you are running an R job, empty for everything else)
      2. Field2: Name of the executable or main R file
      3. Field3: List of any switches that need to be passed
      4. Field4: Locations of any data files necessary (separated by commas)
        Note that if you have data that is always used, hard code its location into your program.
      5. Example might look like this: R|bench.R|iterations=100|
    3. Once the file of jobs is created you can do a trial run like this: submitGrid -test jobs
    4. You can see how many free slots are available like this:submitGrid -usage
    5. you can submit your jobs like this: submitGrid -submit jobs
    6. If you have more jobs than allowed then submit your jobs like this nohup submitGrid -submit jobs &. Go ahead and log out.
  5. You can check on your job status like this: qstat
  6. Your jobs will also appear on the Linux "jobs" page: here
  7. You can kill a queued or running job like this: qdel job_no where job_no comes from the qstat command
  8. Because the jobs are run locally on the NFS share it is easy to overwrite output if you run multiple jobs from the same directory. To prevent this the submitGrid command does....
    1. Creates a unique directory
    2. Copies the program/R source into the above directory
    3. Copes any local data necessary into the above directory
    4. Executes the job from the local directory
  • Video of submitting R jobs located here
[edit]

File Server

Overview:

  1. The server (SAN) is implemented on fully redundant hardware.
  2. Your files are backed up by Windows Shadow Copies (which are available to the user), snapshots, and to tape.
  3. You have the ability to create group space and control access.
  4. You can scan directly to the SAN from our copier.
  5. Your files are also available to you on a Mac and Linux platforms.

Policies:

  1. Do not store personal files on the server.
  2. The SAN was designed to store only source files, thesis material, etc. Do not store data (put it in the SCRATCH folder if you must) or other large files.
  3. There is currently no quota policy but a systems admin will periodically scan home directories to check on usage.
  4. The scratch directory is to be used for larger files that do not need to be backed up.
  5. Although there are currently no quotas enforced please try to keep your usage to less than 2 GB per user. If you need more space use the scratch folder.

Howto:

  1. You can permanently mount your SAN folder to your Windows machine by doing the following:
    1. Open Windows Explorer (or double click on the My Computer icon).
    2. Select Tools->Map Network Drive
    3. Pick P: for the drive letter
    4. in the folder field enter \\kemp.stat.iastate.edu\students$\NET-ID
    5. Select the option to restore the connection each time you log on
    6. Select OK
  2. All other times you will need to access your files using the following methods:
  3. Windows:
    1. Open Windows Explorer (or double click on the My Computer icon).
    2. In the address window type \\kemp\students$\NET-ID (NET-ID) is your isu net-id).
  4. Macintosh:
    1. Open the Finder and type <APPLE> K
    2. Enter "smb:\\kemp\students$\NET-ID
  5. Linux (X86-64):
    1. Log into one of our Linux machines (4,5,6, or 10).
    2. Now execute mount.pl san
    3. The mount.pl script will create a mount folder in /Users/NET-ID/mnt/san if it does not already exist and mount your SAN home folder.
    4. When finished don't forget to disconnect the mount, "unmount.pl san"
  6. Linux (PPC)
    1. Once you log in just execute mount.pl san and your SAN folder will be available via ~/mnt/san
    2. When finished don't forget to disconnect the mount, "umount.pl san"
[edit]

Public Labs

Overview:

The department operates several public labs in Pearson and Carver Hall.

  1. Pearson 0113 (SAS/JMP/TA Office Hour Room)
  2. Carver 205 (Main teaching lab with SPSS)

The labs all have the same software, except for 205 Carver as noted above.


Schedules:

Pearson 0113 Schedule

Carver 205 Schedule


Policies:

  • These labs are open to the public when not in use by classes.
  • Pearson 0113 are open M-F from 8:00 AM to 5:00 PM.
  • Snedecor 307 is open M-R from 7:00 AM to 7:00 PM and 7:00 AM to 5:00 PM on Friday.
  • Statistics Grad students have access to the labs during off hours.
  • Machines that have a user logged in during the open hours can be restarted if a student needs it for a class.
  • Every hour each machine is shut down if no user is logged in.

Back to Stat Department Facilities

[edit]

Grad Student Labs

  1. 3rd floor Wilson Hall has about 24 Windows XP machines for student use.


[edit]

Servers

  1. linux4.stat.iastate.edu (Dell Precision 390, 4 GB Ram, (4) 2.67 GHz Intel 5150).
  2. linux5.stat.iastate.edu (Same as linux4)
  3. linux6.stat.iastate.edu (Same as linux4 but with 16 GB of Ram)
  4. linux10.stat.iastate.edu (8 processor machine with 32 GB of Ram)


Surf over to Linux usage page to see how busy these machines are.

[edit]

Notes

[edit]

Grad Student Printing

  • The department maintains several printers throughout the building that are mainly used for students.
    • The public labs, Carver 205 and Pearson 0113 each have one public (UBill) printer.
      • These printers use the ISU UBILL Pclpr software installed on all of our lab machines.
  • The department charges students $0.05/page to print to the UBILL printers.
    • TA's need to use the printer/copier on the 3rd floor of Wilson Hall or the TA printer in 0113 Pearson for printing information necessary for the course the instructor is teaching. A log is kept of every job that has been printed and is checked in a regular basis so please do not print material NOT related to your class!!
      • Open "Printers and Faxes"
      • Type "\\stat\" and press <ENTER>
      • Pick the Wilson-Printer-Copier-TA printer and press <ENTER>
      • A few seconds later you should be able to print to the printer
[edit]

Staff and Faculty Printing

  • The departnment operates several printers which are available for use
  • The printers are made available to your computer by ...
    • Open "Printers and Faxes"
    • In the address bar type "\\stat" and press <ENTER>.
    • Pick the printer you would like to use and press <ENTER> again.
    • If you are authorized to use the printer you will see the printer queue appear.
  • Our public printers are ...
    • 1st Floor Wilson Duplexing Color
      • Our only duplexing color printer, This is the most expensive printer we have so please use only when color is absolutely necessary.
    • 1st Floor Wilson single sided black and white printer
      • 3rd Floor Wilson is where the high volume printer/copier is located
[edit]

Printing Problems Students

  1. If you are trying to print to a UBILL printer here are some problems and solutions
    1. I get one page and it says something about me having no subsidy, what gives?
-The university "gives" you $5 at the beginning of each semester to use for printing (called your subsidy). When that runs out you must select to use your ubill as the source for printing funds when given the choice by PCLPR.
  1. Nothing happens
Retrieved from "http://streaming.stat.iastate.edu/wiki/index.php/Computing_Resources"
Personal tools