/scratch

Description of File System:

/scratch is mounted on all worker nodes. The file system is intended to provide extremely high I/O throughput to or from disk. The files are subject to deletion once their quota\lifetime is exceeded. Users are granted a default quota of 100GB in /scratch and are requested to apply for additional storage by e-mailing the HPC administrators. Quotas are implemented for file system administration. This is the preferred place to put any intermediate files required while a job is executing. Users are expected to move their data from /scratch to long term storage as part of the clean up process in their job script. /scratch is purposely built as a computational work space area and is not intended for long term storage. This file system is not backed up.

General Usage:

Users are requested to follow the proposed work-flow to ensure no data are lost. As indicated, this file system is designed to provide high throughput to disk. We encourage users who have large datasets which need to be read into memory, to copy their datasets to the /scratch file system, execute the required application and redirect the output onto the /scratch file system. Data that you wish to keep must be removed from the /scratch file system. Please note that you may only store 20GB of data on /home.

File System Backup:

The /scratch file system is not backed up. Any data which is lost cannot be recovered. Ensure that your data is moved off the file system when jobs complete.

File System Rebuild:

The /scratch file system has been purpose built for high throughput. In order to keep expanding the file system we would need to schedule a maintenance window in order to expand and re-balance the file system capacity. Users will be informed well in advance of the scheduled maintenance window.

Operational and Security Level Maintenance:

The eResearch HPC team is required to keep systems patched regularly according to the Scheduled ICTS Maintenance Plan.

The servers that present the /scratch file system need to be patched regularly. Users will be informed of expected maintenance should it be deemed necessary. Not every ICTS maintenance window will result in HPC downtime.

Recommendations:

  • Do not store important files in /scratch

  • Code, scripts, libraries, application executables etc should not be stored in /scratch

  • /scratch is designed to work with large files. Smaller files such as scripts or source code should be placed on /home.

File System Quota Check:

Users are encouraged to check their quotas by executing the following command:

myquota

This will produce two results, your quota in /home and /scratch for example:

=====================================================================================
QUOTA IN /home/jsmith

Quota information for storage pool Default (ID: 1):

user/group || size || chunk files
name | id || used | hard || used | hard
--------------|------||------------|------------||---------|---------
jsmith| 1000|| 5.82 GiB| 20.00 GiB|| 74685| 1000000

=====================================================================================

QUOTA IN /scratch/jsmith

Quota information for storage pool Default (ID: 1):

user/group || size || chunk files
name | id || used | hard || used | hard
--------------|------||------------|------------||---------|---------
jsmith| 1000|| 29.69 GiB| 100.00 TiB|| 22647| 10000000

=====================================================================================

You should pay attention to the chunk files (inodes) parameter. This indicates the number of files that you may create, this is also a limit. If you anticipate creating very large numbers of files please inform the system administrator.

Identifying old files:

To determine which files have reached their age limit on the /scratch volume you can run the following:

find /scratch/myusername -mtime +100

Where username is your login name and the value after the + is your data lifespan in days on the /scratch volume. Replace 100 with your appropriate data lifespan from the table below. By reducing the life span value by a few days you can also determine which files are reaching their age limit.

File System Cleanup Policy:

The /scratch volume is shared space and is intended for temporary processing, not long term storage. Should the /scratch volume run out of space automatic scripts will delete the oldest data on the system. It is your responsibility to move processed data off the cluster. There are no backups of /scratch

QuotaLifespan (days)
100GB (default)100
250GB80
500GB60
1TB50
5TB40
10TB30
20TBDepends on usage

Quota:

Storage allocation is applied to a user on the /scratch file system. These are not directory quotas, they are user quotas applied to the entire volume.

Delete:

Files which have not received action after the allocated “ delete “ time period, will be automatically removed. should the /scratch volume start to run out of space.

Inode Limits:

We have implemented a default chunk file (inode) quotas. This means you can effectively create 100,000 files / folders in /home and 10,000,000 files / folders in /scratch. If you need to create more files please let us know.

Warning:

Using the touch command to modify files in the /scratch file system will result in your account being suspended.

Long term storage:

Users that require reliable long term central storage should contact eresearch@uct.ac.za

Menu