Filesystem Target¶

The following section describes how to setup a Storage Target to a POSIX-compatible mounted file system.

The target file system must be mounted on all nodes which will migrate or recall files.

Example Configuration File¶

[General]
RetrieveObjectBasePath=/mnt/targetfs
RetrieveObjectName=$1
StoreObjectBasePath=/mnt/targetfs
StoreObjectName=$1
RemoteLocationXAttr=targetfs1:$1
DeleteOnRecall=False
ObjectXAttrManipulationMode=auto
EnsureMountPoint=/mnt/targetfs

Configuration Keywords¶

Required keywords

EnsureMountPoint

RetrieveObjectName

StoreObjectName

Optional keywords

Keyword	Default Value
ACLSave	False
DeleteOnRecall	False
MaxCopyThreadCount	MaxUploadThreadCount
MaxDownloadThreadCount	Number of cores / 2
MaxUploadThreadCount	Number of cores / 2
ObjectXAttrManipulationMode	"auto"
RemoteLocationXAttr	"$0"
RetrieveObjectBasePath	""
ShadowMetadataStore	"samedir"
StoreObjectBasePath	""

EnsureMountPoint¶

Specifies the mount point that ngmigrate and ngrecall processes will validate prior to migrating or recalling a file.

Should the specified location not be the root of a mount point, the migration or recall will abort with an error.

This is an important safety feature to ensure that the local disk is not used in the event of a target mount failing.

It is typically set to the root of the mount point.

Syntax

EnsureMountPoint=/mountpoint

Example

EnsureMountPoint=/mnt/targetfs

RetrieveObjectName¶

Defines a mapping from the value of a remote location (APXrmtXX) DMAPI extended attribute of a local file to the name suffix of an associated file on the remote storage target. The value of RetrieveObjectBasePath parameter is prepended to that suffix to obtain the full path of a file to retrieve (copy from).

In a value of this parameter, the reference $0 represents the entire value of a remote location (APXrmtXX) DMAPI extended attribute. The references $1, $2, ..., $9 represent parts of that entire value matched to parts of a regular expression specified by RemoteLocationXAttrRegex for this storage target in the Master Configuration File. The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.

It is recommended to strip a directory prefix from the value of this parameter and specify the prefix in the RetrieveObjectBasePath parameter. Otherwise, ngscan will ignore that directory prefix when listing files on the remote storage target.

The default value for this parameter is $0.

The value of this parameter typically matches the value of StoreObjectName parameter.

Example

RetrieveObjectName=$1

StoreObjectName¶

Defines a mapping from the name of a local file to the name suffix of an associated file on the remote storage target. The value of StoreObjectBasePath parameter is prepended to that suffix to obtain the full path of a file to store (copy to).

In a value of StoreObjectName parameter, the reference $0 represents the entire absolute name of a local file. The references $1, $2, ..., $9 represent parts of that entire name matched to parts of a regular expression specified by LocalFileRegex for this storage target in the Master Configuration File. The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.

It is recommended to strip a directory prefix from the value of this parameter and specify the prefix in the StoreObjectBasePath parameter. Otherwise, ngscan will ignore that directory prefix when listing files on the remote storage target.

The default value for this parameter is $0.

Example

StoreObjectName=$1

ACLSave¶

Defines whether ngmigrate copies the ACLs of local files and directories to remote files directories or whether ACLs are saved metadata remote files and directories.

If the filesystem of a storage endpoint is of type GPFS, ngmigrate copies the ACLs of local files and directories to the ACLs of files and directories in the storage endpoint
If the filesystem of a storage endpoint is not of type GPFS and metadata is stored in native xattrs, ngmigrate saves the ACLs of local files and directories in native xattrs of files and directories in the storage endpoint
If the filesystem of a storage endpoint is not of type GPFS and metadata is stored in shadow metadata files, ngmigrate saves the ACLs of local files and directories in shadow metadata files for data files and directories in the storage endpoint

Refer to ObjectXAttrManipulationMode to specify where the metadata of files and directories is saved.

If this parameter is False, ngrecall does not attempt to restore the ACLs of local files and created local directories on reverse stubbing/premigration.

The default value is False.

Syntax

ACLSave=<True|False>

Example

ACLSave=True

DeleteOnRecall¶

Defines whether to immediately delete the data from the POSIX mount target when successfully recalled back to the file system.

Setting this to False will result in files entering a "premigrated" state on successful recall.

Syntax

DeleteOnRecall=<True|False>

Example

DeleteOnRecall=True

MaxCopyThreadCount¶

Defines the maximum number of threads an ngcopy command will use when copying or moving files to the filesystem target. Each thread copies an individual file. Moving a file from a source location to a target location is implemented as copying the file to the target location with subsequent deleting the file at the source location.

The default value of this parameter is equal to the value of MaxUploadThreadCount parameter.

Syntax

MaxCopyThreadCount=<# of threads>

Example

MaxCopyThreadCount=4

MaxDownloadThreadCount¶

Defines the maximum number of threads an ngrecall command will use when copying files from the filesystem target. Each thread copies an individual file.

Defaults to the number of CPU cores in the system divided by 2 (but not less than 1).

Syntax

MaxDownloadThreadCount=<# of threads>

Example

MaxDownloadThreadCount=4

MaxUploadThreadCount¶

Defines the maximum number of threads an ngmigrate command will use when copying files to the filesystem target. Each thread copies an individual file.

Defaults to the number of CPU cores in the system divided by 2 (but not less than 1).

Syntax

MaxUploadThreadCount=<# of threads>

Example

MaxUploadThreadCount=4

ObjectXAttrManipulationMode¶

Defines the behaviour for storing and retrieving the metadata of files and directories migrated to and recalled from the POSIX mount target.

When native mode is selected, metadata is stored and retrieved as extended attributes on migrated files and directories. If the target POSIX mount does not support extended attributes, then migration and recall will abort with an error.

When shadow mode is selected, metadata is stored to and retrieved from a file stored alongside the data file or directory.

When disable mode is selected, metadata will not be used.

Setting this parameter to auto (the default) results in the ngmigrate/ngrecall process attempting to store metadata to or retrieve metadata from extended attributes, and if the storing or retrieving operation fails, ngmigrate/ngrecall uses an accompanying file for storing or retrieving metadata.

Syntax

ObjectXAttrManipulationMode=<auto|shadow|native|disable>

Example

ObjectXAttrManipulationMode=disable

RemoteLocationXAttr¶

Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the Master Configuration File.

Syntax

RemoteLocationXAttr=targetref:<LocalFileRegexReference>

Example

RemoteLocationXAttr=targetfs1:$1

RetrieveObjectBasePath¶

Defines a base path for copying files from the remote storage target. This base path is prepended to a path composed using the RetrieveObjectName parameter to obtain the full path of a file to copy. By default, this base path also acts as a base path for listing objects on the remote storage target by ngscan.

The empty base path is interpreted as /. If a base path does not end with /, then / is implicitly appended to the base path.

The base path ordinarily specifies a mount point or a directory on the mount point. Such a base path is the root of a filesystem sub-tree representing the remote storage target. When listing objects on the remote storage target, ngscan lists files in this sub-tree by default.

The default value for this parameter is /. This value is inconvenient for using with ngscan: by default, it will list files starting at the root of the entire filesystem.

The value of this parameter typically matches the value of StoreObjectBasePath parameter.

Example

RetrieveObjectBasePath=/mnt/targetfs

ShadowMetadataStore¶

Defines a location to store shadow metadata files.

If this parameter is equal to "samedir", shadow metadata files for remote objects and folders are stored in the same directory as the remote objects and folders.

If this parameter is equal to "subdir", shadow metadata files for remote objects and folders are stored in the subfolder .ngenea_metadata of a remote folder containing the remote objects and folders.

If this parameter is equal to "altdir:PATH", shadow metadata files for remote objects and folders are stored in the directory PATH, where PATH should be absolute one (i.e., start with /). Ngenea HSM tools automatically create missing directories for placing the shadow metadata file of a remote object or folder. Directories created under PATH correspond to parent folders of the remote object or folder. Parent folders are replicated until reaching the longest common ancestor path calculated for PATH and the absolute path of the remote object or folder.

E.g., if PATH is "/mnt/fs_storage/ngenea_metadata", and the path of a remote object is "/mnt/fs_storage/path/to/file", the longest common ancestor path is "/mnt/fs_storage". Therefore Ngenea HSM tools will create the shadow metadata file "/mnt/fs_storage/ngenea_metadata/path/to/.file.xattr" for the remote object.

Warning

If a user manually deletes remote objects or folders, Ngenea HSM tools will not delete corresponding shadow metadata files under PATH. Over time, the directory PATH may contain orphaned shadow metadata files.

The parameter ShadowMetadataStore is applicable to the value "auto" or "shadow" of ObjectXAttrManipulationMode parameter.

The default value is "samedir".

Syntax

ShadowMetadataStore=<samedir|subdir|altdir:<PATH>>

Example

ShadowMetadataStore=subdir

StoreObjectBasePath¶

Defines a base path for copying files to the remote storage target. This base path is prepended to a path composed using the StoreObjectName parameter to obtain the full path of a file to copy. Additionally, this base path can act as a base path for listing objects on the remote storage target by ngscan.

The empty base path is interpreted as /. If a base path does not end with /, then / is implicitly appended to the base path.

The base path ordinarily specifies a mount point or a directory on the mount point. Such a base path is the root of a filesystem sub-tree representing the remote storage target. When listing objects on the remote storage target, ngscan lists files in this sub-tree.

The default value for this parameter is /. This value is inconvenient for using with ngscan: it will list files starting at the root of the entire filesystem.

The value of this parameter typically matches the value of RetrieveObjectBasePath parameter.

Example

StoreObjectBasePath=/mnt/targetfs