BGFS (formerly FHGFS)

Description of File System:

/researchdata/fhgfs is mounted on all worker nodes. The file system is intended to provide extremely high I/O throughput to or from disk. The files are subject to deletion once their quota\lifetime is exceeded. Users are requested to apply for storage by e-mailing the HPC administrators. Quotas are implemented for file system administration. This is the preferred place to put any intermediate files required while a job is executing. Users are expected to move their data from /researchdata/fhgfs to long term storage as part of the clean up process in their job script. /researchdata/fhgfs is purposely built as a computational workspace area.

General Usage:

Users are requested to follow the proposed work-flow to ensure no data are lost. As indicated, this file system is designed to provide high throughput to disk. We encourage users who have large datasets which need to be read into memory, to copy their datasets to the FhGFS file system, execute the required application and redirect the output onto the FhGFS store. Data that you wish to keep must be removed from the FhGFS file system. Please note that you may only store 40GB of data on /home.

File System Backup:

/researchdata/fhgfs file system is not backed up. Any data which is lost cannot be recovered. Ensure that your data is moved off the file system when jobs complete.

File System Rebuild:

The /researchdata/fhgfs file system has been purpose built for high throughput. In order to keep expanding the file system we would need to schedule a maintenance window in order to expand and re-balance the file system capacity. Users will be informed well in advance of the scheduled maintenance window.

Operational and Security Level Maintenance:

The eResearch HPC team is required to keep systems patched regularly according to Schedule Maintenance Plan – http://www.icts.uct.ac.za/images/Scheduled_maintenance_plan.pdf

FhGFS as well as the underlying operating system needs to be patched regularly. Users will be informed of expected maintenance should it be deemed necessary. Not every ICTS maintenance window will result in HPC downtime.

Recommendations:

  • Do not store important files in /researchdata/fhgfs

  • Code, libraries, application executables etc should not be stored in /researchdata/fhgfs

  • FhGFS is designed to work with large files. Smaller files such as scripts or source code should be placed on /home.

File System Quota Check:

1. Users are encouraged to check their quotas by executing the following command:

  beegfs-ctl --getquota --uid $USER
or
  beegfs-quota

2. Users who are engaged in group work on /researchdata/fhgfs workspace must execute the following command. Please specify the group name as you could belong to more than one group:

  beegfs-ctl --getquota --gid

Identifying old files:

To determine which files have reached their age limit on the fhgfs volume you can run the following:

find /researchdata/fhgfs/username -mtime +100

Where username is your login name and the value after the + is your data lifespan in days on the fhgfs volume. Replace 100 with your appropriate data lifespan from the table below. By reducing the life span value by a few days you can also determine which files are reaching their age limit.

Filesystem Cleanup Policy:

The Hex HPC cluster has scripts which are executed weekly to determine which files have not been accessed in a certain amount of time. The /researchdata/fhgfs file system has the following cleanup procedure in place. The eResearch Support Team retain the right to improve system performance and clean up files on /researchdata/fhgfs at any time if needed.

QuotaLifespan (days)
100GB (default)100
250GB80
500GB60
1TB50
5TB40
10TB30
20TBDepends on usage

Quota:

Storage allocation is applied to a user or group on the FhGFS file system. These are not directory quotas, they are user / group quotas.

Delete:

Files which have not received action after the allocated “ delete “ time period, will be automatically removed.

Inode Limits:

We have implemented a default inode quota of 50000. This means you can affectively create 50000 files / folders on your allocated quota. If you need to create more files please let us know.

Warning:

Using the touch command to modify files in the fhgfs storage area will result in your account being suspended.

Long term storage:

Users that require reliable long term central storage should contact eresearch@uct.ac.za

Menu