Skip to content

MUSTFS Storage Service

MUSTFS is a distributed filesytem based on Ceph accessible from all the computing plateforms servers (interactives servers included).

MUSTFS service provides disk based storage which can be seen as semi-permanent . It is designed to host large data volumes, like non-critical experimental data that have to be accessed regularly by several colleagues. This service is not supposed to provide back-ups.

The usage of this storage service and the allocation of additional capacity above the default quota are subject to financial conditions which must be accepted by laboratory direction, group or project manager.

Group storage areas

Group data areas are shared by all users belonging to the same UNIX group (corresponding to a laboratory/project/experiment or collaboration) and accessible from interactive servers (lappui/lapthui or lappusmb) and from all the compute nodes of the batch farm. Once created, this space can be managed indifferently by all the users belonging to the same unix group. Users can also organize and delegate authorizations to other users with Linux Access Control Lists (ACLs).

New users are invited to check the existence of a Group storage data area for their laboratory, experience or project group :

  • /lapp_data/mygroup for LAPP and LAPTh users
  • /uds_data/mygroup for USMB users

The default quota is 1 TiB.

Synthetic view of MUSTFS areas

The diagram below shows the various storage areas and capacities available for local usages and provided by MUSTFS service (code colour blue).

It is worth noting that HOME directories of LAPP/LAPth members (/home1 and /home3) are not provided by MUSTFS service but hosted on the Network-Attached Storage (NAS) filer which is part of LAPP/LAPTh infrastructure (code colour green).

Storage overview

MUSTFS Quotas

MUSTFS allows quotas to be set on any directory in the file system. The default quota is 1 TiB per group (laboratory or project).

All the GROUP data areas available under /lapp-data and /uds_data have a GROUP quota set which is shared among users of the same group, project or laboratory. Processes that are writing to the GROUP area will be stopped a short time after the GROUP quota limit is reached. They will inevitably be allowed to write some amount of data over the configured limit. How far over the quota they are able to go depends primarily on the amount of time, not the amount of data. Generally speaking writers will be stopped within 10s of seconds of crossing the configured limit.

To view quota limit in bytes :

$ getfattr -n ceph.quota.max_bytes /lapp_data/some_dir`
# file: lapp_data/some_dir
ceph.quota.max_bytes="1099511627776"`

Additional space may be allocated upon request to must-support AT lapp.in2p3.fr.

How to recover a recently deleted file

All MUSTFS group storage areas are snapshoted on a regular basis. This could come handy in case you want to recover a recently deleted file or want to revert to an older version.

Snapshots are only kept for 4 days, you WILL NOT be able to recover your data after this delay.
In case you need assistance, please get in touch with us.

Snapshots are available through the .snap hidden folder inside every directory of your group area. Note that even an ls -a WILL NOT show the .snap folder, and thus, you'll need to access it by directly mentionning it, like : 

cd /<group_area>/<your_dir>/.snap

You are then presented with a list of folders reprensenting the available snapshots for this directory :

# ls -alh
total 0
drwxr-xr-x 1 username group    0 Jun 30 09:32 .
drwxr-xr-x 1 username group 208K Jul  9 15:58 ..
drwxr-xr-x 1 username group 207K Jun 20 11:58 _daily_07_01_<id>
drwxr-xr-x 1 username group 207K Jun 20 11:58 _daily_07_02_<id>
drwxr-xr-x 1 username group 207K Jun 20 11:58 _daily_07_03_<id>
drwxr-xr-x 1 username group 207K Jun 20 11:58 _daily_07_04_<id>

You can then naviguate into them and copy your data as usual.