ufs18 File Systems
The ufs18 file system is a 2018 IBM General Parallel File System (GPFS) installed in HPCC for storing our home or research spaces. While it is faster and more stable, users need to learn its differences from other file systems and understand how to use it.
The only way to get quota information of ufs18 file space is to
run the command
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19
where all file spaces accessible to the user are listed, including home, research, and scratch. In each space, the information of quota, usage and availablility on space size and number of files can be found. If "Free" or "Available" is a negative value (such as "Space Available" in home directory of the above example), the usage is over the quota. Please remove, transfer or compress some files so the used space or the file count can be lower than the "Quota" value. Since GPFS uses a different compression algorithm, you may notice higher space size after files are copied to the ufs18 file system.
Actual disk usage different from quota results
The new file system has a smallest file block size of 64K. This means that files between 2K and 64K will occupy 64K of space. This causes space usage inflated greatly for users with large amounts of small files. One suggested solution would be to compress many small files into one large file (at least larger than 64K). If you still have any difficulty, a temporarily larger quota can be requested by a user if their quota is at 1T with many small files.
Limit on number of files
Besides the quota on the size of space, users are also limited to 1 million files in their home or research directory. We need to set this limit because, with a great number of files, the file system will spend too much time on back-up to be able to function normally. If possible, users should compress many files into one to reduce the number of files.
If users do not wish to have the limit, they can request an extra
there is no limit on the file count yet no back-up on the files either.
Users will be responsible for their own back-up on the
By default, one half of the allowed quota of a space size will be
assigned to the requested
nodr space. The quota of the original space
/mnt/research/ is then downsized to half so
the total disk space quota remains the same. A different proportion of
their sizes can also be requested. Once this
space is created, the path and the quota information can be found from
the results of the
quota command mentioned above.
Quota setting on research space
The quota setting on research space is based on the group
ownership of the files. Any files with the group ownership the same as
the research space are followed by the
quota command. However, any
files (larger than 8 MB) with a group ownership different from the
research space are not allowed to exist.
Due to this reason, even though there is no over-quota issue from
the results of
quota command, users might still get an error message
such as “failed to ... ... Disk quota exceeded” when they create, copy
or write a file to his research space. To resolve this "Disk quota
exceeded" problem, users may do the following:
Make sure the directory to which files are copied has the same group ownership as the research space and has the set-group-ID bit.
For example, you get the error message when trying to transfer files from your local computer to a directory
Drctryin your research space
ls -ldcommand to check the group ownership and the access permission of the directory
Drctryand the research space
1 2 3
$ ls -ld /mnt/research/Group/Drctry /mnt/research/Group drwxrwx--- 2 UserName Prmry 5464 Feb 27 11:34 /mnt/research/Group/Drctry drwxrws--- 9 UserName Group 8192 Jul 10 15:34 /mnt/research/Group
where you can see their differences. For the group ownership,
Prmryof the directory
Drctryis different from
Groupof the research space
/mnt/research/Group. For the permission,
rwxrwx---of the directory does not have the set-group-ID bit
rwxrws---as the research space. The owner
UserNameof the directory can run the commands:
$ chgrp -R Group /mnt/research/Group/Drctry # Change the group ownership to Group $ chmod g+s /mnt/research/Group/Drctry # Set up set-group-ID bit
to change the two attributes of the directory. (A further instruction about file permission can be reviewed from the wiki page File Permissions on HPCC.) Once the settings are corrected:
$ ls -ld /mnt/research/Group/Drctry drwxrws--- 2 UserName Group 5464 Feb 27 11:34 /mnt/research/Group/Drctry
the file transfer can proceed to the directory.
If the file already exists, its group ownership needs to be changed to the group of the research space.
For example, you try running a command to copy, transfer or write a file
footo a directory
Drctryof your research space. However, a file with the same file name
fooalready exists in the directory
Drctry. In order to make the command work,
fooin the directory
Drctrymust have the same group ownership as the research space. Otherwise, the owner of the file can use
chgrpcommand mentioned above to correct the group ownership or you have to rename or remove
fooin the directory
If the file is going to be created, user's primary group may need to be set to the group of the research space.
Users can use
newgrpcommand to reset his primary group temporarily. For more information, please refer to Change Primary Group page.
Samba mapping path for local computer
Users mounting their home directories or research spaces will need to update their SMB/Samba/Windows File Sharing paths in their clients. To determine the mount path, log into HPCC, ssh to a development node and run:
To determine the mount point for your research space, use the following command:
You may also use the powertools command to see all paths of your home and research spaces:
ACL for GPFS
If you are using access control list (ACL), you will need to update them to NFSv4 ACLs. You will need to use the mmgetacl, mmputacl, and mmeditacl commands. Please refer to the GPFS Commands page for more details.