General Configuration

DataCore Software Ngenea HSM requires the following components to correctly function:

• a modified PixStor file system policy

• a master DataCore Software Ngenea HSM configuration file with at least one storage target

• a configuration file for each storage target

• active accounts with the appropriate cloud storage providers

The following sections describe how to apply the above prerequisites.

PixStor Scale Configuration

Transparent Recall file system policy

In order to enable transparent recall on a file system, insert the transparent recall rules provided by ArcaStream / Pixit Media at the beginning of the current file system policy.

Note that if no current policy is in place, you will also need to include a default placement policy rule at the end of the policy, ensuring that new files are written to the correct PixStor storage pool. This is not required if you only have a system PixStor storage pool. For example:

/* By default place all data on the sata1 pool */
RULE 'default' SET POOL 'sas1'

Ensure that any future alterations to the GPFS file system policy do not replace, modify or precede any of the DataCore Software Ngenea HSM policy.

Cluster Configuration

In order to provide the maximum number of concurrent transparent recall and migration threads, set the following GPFS cluster configuration parameters on any node which is expected to issue transparent recall requests.

policySystemEvalLimit 0
dmapiWorkerThreads 64

In prior versions of ngenea, policySystemEvalLimit 64 was suggested. More recent versions of GPFS have been observed to work better with values of 0.

Also, ensure that workerThreads is set to greater than 64:

workerThreads 128

Master Configuration File

The master configuration file defines the available storage targets, and directs migration and recall requests to the appropriate target.

The default location for DataCore Software Ngenea HSM configuration files is within /opt/arcapix/etc/, and the default location for the master configuration file is /opt/arcapix/etc/ngenea.conf.

Any storage configuration files are relative to the location of the master configuration file.

Configuration files should not be world-readable. This applies to the master configuration file and any files referenced by the master configuration file.

Below is an example configuration file with two storage targets defined. It contains the optional [General] section and optional [FileMatch ...] sections.

[General]
MinMigrateEndpointCount=1

[Storage bpearl1]
StorageType=BlackPearl
ConfigFile=blackpearl1.conf
LocalFileRegex=/mmfs1/(archive/.+)
RemoteLocationXAttrRegex=blackpearl:(.+)

[Storage awss3]
StorageType=AmazonS3
ConfigFile=awss3.conf
LocalFileRegex=/mmfs1/(active/.+)
RemoteLocationXAttrRegex=awss3:(.+)

[FileMatch text]
BasenameMask=*.txt
StubSize=-1

[FileMatch video]
PathnameRegex=.*/avi/
StubSize=16777216

[FileMatch default]
StubSize=0

Section Keywords

Keyword
[General]
[Storage <storage target name>]
[FileMatch <file match set>]

Section [General]

This optional section specifies general parameters for all storage endpoints.

All keywords in this section are optional.

Keyword	Default Value
DMAPI	true
Flock	true
LockLevel	partial
MaxJobPartFileCount	512
MaxReverseStubCount	-1
MaxTransparentRecallLockCount	19
MinMigrateEndpointCount	0
UpdateAccessTime	false
WatchdogTimeoutSeconds	None

DMAPI

Specifies whether ngrecall calls DMAPI-related functions while performing the --fetch operation.

If the parameter is equal to True, ngrecall calls DMAPI and GPFS-specific functions and can set DMAPI extended attributes for a fetched file and make it premigrated.

If the parameter is equal to False, ngrecall uses ordinary file I/O functions for accessing a fetched file---it is only downloaded and can never become premigrated. In this case, ngrecall can be run by a non-privileged user.

Setting the parameter to False is equivalent to passing the option --no-dmapi to ngrecall on performing the --fetch operation.

The default value is True.

Syntax:

DMAPI=true|false

Example:

DMAPI=false

Flock

Controls the use of the complementary locking mechanism. If this parameter is equal to True, the complementary locking mechanism is enabled. If this parameter is equal to False, the complementary locking mechanism is disabled.

The complementary locking mechanism prevents simultaneous processing the same file by more than one ngmigrate, ngrecall, ngcopy, ngunmanage, or ngmakestub process running on any node in a GPFS cluster. Preventing simultaneous processing the same file is achieved by creating a lock file in a subdirectory of .ngenea/flock/ directory in a GPFS filesystem. The name of the lock file contains an inode number and inode generation number for the locked file. After creating the lock file, an Ngenea HSM process locks it to prevent concurrent access to the locked (E.G. migrated or recalled) file. If the lock file cannot be created and locked, an Ngenea HSM tool does not begin processing the locked file and waits until the lock becomes available.

Disabling the complementary locking mechanism provides an increase in the speed of processing files by Ngenea HSM tools.

Warning

Concurrent access to files by multiple Ngenea HSM processes may cause corrupted files. It is recommended to set Flock=false where non-concurrent access can be guaranteed.

It is prohibited to disable the complementary locking mechanism and use the "Partial" DMAPI locking level (see the parameter LockLevel), because a filesystem deadlock may occur. Disabling the complementary locking mechanism automatically sets DMAPI locking level to "Implicit" unless it was set to "Partial".

The default value of the parameter Flock is True. Setting the parameter to False is equivalent to passing the option --no-flock to ngmigrate, ngrecall, or ngcopy tool.

Syntax:

Flock=true|false

Example:

Flock=false

LockLevel

Specifies a DMAPI locking level for file I/O operations. The option --lock-level=partial|implicit of ngmigrate and ngrecall overrides the value of this parameter.

Partial (default)

In this mode, ngmigrate obtains a SHARED access right and uses it for file read operations while uploading a file. This access right protects the file from modifying by other programs while it is being migrated. The protection requires the Transparent Recall file system policy to be installed to the filesystem placement policy. The ngmigrate tool can also obtain EXCLUSIVE access rights for short time periods to perform write operations on a file, for example, to update file attributes or punch a hole into the file.

The ngrecall tool obtains EXCLUSIVE access rights for the duration of file write operations while downloading a file. In this mode, ngrecall can also obtain SHARED or EXCLUSIVE access rights to perform other operations on the file.

Using partial mode, file system snapshot operations will block until all ngenea operations are completed.

Implicit

In this mode, ngmigrate and ngrecall do not obtain SHARED and EXCLUSIVE access rights explicitly. DMAPI file I/O functions implicitly lock a file at the beginning of an I/O operation and unlock the file at the end of the I/O operation. The minimum possible locking is achieved when using this locking mode. It is recommended to utilise implicit locking mode in situations whereby ngenea operations interfere with file system snapshot operations.

Syntax:

LockLevel=partial|implicit

Example:

LockLevel=implicit

MaxJobPartFileCount

Specifies the maximum number of local files to migrate, recall, or delete remote objects for in one batch. If there are more local files to process, ngmigrate or ngrecall automatically creates subsequent batches.

For file recall and remote object deletion operations, ngrecall automatically adjusts the value of this parameter to not exceed the value of "MaxTransparentRecallLockCount" parameter.

The default value is 512.

Syntax:

MaxJobPartFileCount=<Maximum number of files per batch>

Example:

MaxJobPartFileCount=512

MaxReverseStubCount

Specifies the maximum number of concurrent operations creating stub files with ngrecall while performing reverse premigration (before downloading file content) or reverse stubbing.

Stub files are created in two phases: first, ngrecall obtains parameters of the stub files (E.G. their sizes) from the metadata of corresponding remote objects; second, ngrecall creates the stub files according to the obtained parameters.

For storage target types except Filesystem or Spectra Logic BlackPearl, the operations for fetching metadata of multiple remote objects can be performed by multiple threads in parallel.

Parameters in storage-specific configuration files, E.G. MaxMetadataReadThreadCount, set the number of threads for fetching remote object metadata and control parallelizing the first phase of creating stub files operation.

The parameter MaxReverseStubCount controls parallelizing the second phase of the operation for all ngrecall processes.

If this parameter is positive, operations for creating stub files are multi-threaded with the total number of threads across all ngrecall processes being not greater than (the number of ngrecall processes x the number of CPU cores) or the value of the parameter, whichever is the smaller.

If the limit is reached, an ngrecall process waits until the number of concurrent operations decreases to continue creating stub files.

Example scenario:

Available CPU Cores: 10
Number of ngrecall processes: 3
MaxReverseStubCount=15
Aggregate active Reverse Stub threads: 15

Available CPU Cores: 10
Number of ngrecall processes: 3
MaxReverseStubCount=50
Aggregate active Reverse Stub threads: 30

If this parameter is equal to 0, operations for creating stub files are multi-threaded with the number of threads equal to (the number of ngrecall processes x the number of CPU cores).

Example scenario:

Available CPU Cores: 10
Number of ngrecall processes: 3
MaxReverseStubCount=0
Aggregate active Reverse Stub threads: 30

If this parameter is equal to -1, operations for creating stub files are single-threaded.

Example scenario:

Available CPU Cores: 10
Number of ngrecall processes: 3
MaxReverseStubCount=-1
Aggregate active Reverse Stub threads: 3

The default value of -1 aligns with the behaviour of ngrecall versions prior to 1.34 whereby operations for creating stub files according to parameters obtained from remote object metadata are single-threaded.

Syntax:

MaxReverseStubCount=<Maximum number of concurrent operations of creating stub files>

Example:

MaxReverseStubCount=8

MaxTransparentRecallLockCount

Specifies the maximum number of concurrent locks put on distinct local files accessed via DMAPI in file recall and remote object deletion operations.

If ngrecall exceeds that maximum number when trying to lock a local file, ngrecall terminates with an error.

To minimize the number of such errors, ngrecall automatically adjusts the maximum number of local files to recall or delete remote objects for in one batch specified by the "MaxJobPartFileCount" parameter to not exceed the value of "MaxTransparentRecallLockCount" parameter.

The default value is 63 as of version 1.31.

Though the maximum allowed value is 64, the underlying file system DMAPI threads require resources for stat operations, etc., therefore the maximum recommended value is 63. A value of 64 could incur situations where DMAPI threads are locked due to resource conflicts.

Prior versions used a very conservative value of 19, whereas field observations show 63 provides increased performance with no loss of reliability.

Syntax:

MaxTransparentRecallLockCount=<Maximum number of concurrent locks>

Example:

MaxTransparentRecallLockCount=10

MinMigrateEndpointCount

Specifies the minimum number of successful migrations required in order to report success for a migration to multiple storage endpoints. See Multi-Target Support for an overview of multi-target configuration and operation.

The default value is 0.

Syntax:

MinMigrateEndpointCount=<Minimum number of successful migrations>

Example:

MinMigrateEndpointCount=3

UpdateAccessTime

This parameter controls updating the last access time and last status change time to current time for a file:

• after its successful migration or premigration by ngmigrate

• after its successful recall, reverse stubbing, or reverse premigration by ngrecall

If the parameter is set to True, the last access time and last status change time are updated to current time, which is equivalent to passing the option --update-atime to ngmigrate or ngrecall. If the parameter is set to False, and the option --update-atime is absent on the command line, ngmigrate does not modify the last access time and last status change time, and ngrecall restores the times from remote object metadata.

The default value is False.

Syntax:

UpdateAccessTime=true|false

Example:

UpdateAccessTime=true

WatchdogTimeoutSeconds

Defines the number of seconds after which an ngmigrate or ngrecall process will terminate with exit status 1 if none of monitored operations are progressed within the timeout period. Operations are monitored per processed data block of a file. Successfully progressing an operation or completing a data block resets the watchdog timer to 0.

The monitored operations are:

• high-level operations of communicating with a storage endpoint

• file I/O when uploading or downloading data to/from a storage endpoint

• file I/O when calculating file hashes

• syncing file I/O buffers to the physical medium

It is not required to set this configuration parameter for successful operation. The parameter might be applied if ngmigrate or ngrecall operations consistently fail to complete and/or are observed to be in indefinite exponential backoff due to utilising links observed to be oversubscribed or demonstrating significant jitter or packet loss.

Values less than 300 are not recommended.

Syntax:

WatchdogTimeoutSeconds=<Timeout period in seconds>

Example:

WatchdogTimeoutSeconds=600

Section [Storage <storage target name>]

The Storage section defines how a storage target will be used for migrating and recalling data. Each storage target definition includes a name, type, scope of operation and a reference to the Storage Target configuration file.

Syntax:

[Storage <storage target name>]

Example:

[Storage target1]

Required keywords

ConfigFile

StorageType

Optional keywords

Keyword	Default Value
CheckHash	true
CheckUUID	true
LocalFileRegex	".+"
LocalFileRegexExclude	"$^"
LocalSymlinkTargets	"endpoint"
MetadataUpdate	true
ObjectInstances	latest
OverwriteRemote	false
RemoteLocationXAttrRegex	""
RestoreACL	true
StorageKey	"lc"
Writable	true

ConfigFile

Defines the location of the associated configuration file for the Storage entry, either specified as the full path and filename, or path and filename relative to the location of this master configuration file.

Syntax:

ConfigFile=<filename>

Example:

ConfigFile=/opt/arcapix/etc/target1.conf

StorageType

Defines the supported mode of storage operation. Available options are:

• BlackPearl - SpectraLogic Black Pearl Spectra S3 target

• AmazonS3 - Amazon S3, or compatible, protocol target

• Azure - Microsoft Azure blob storage target

• Google - Google Cloud storage target

• FS - POSIX file system target

Syntax:

StorageType=<AmazonS3|BlackPearl|Azure|Google|FS>

Example:

StorageType=AmazonS3

CheckHash

This parameter controls checking the SHA-512 hashes of remote objects and the updating of their SHA-512 hash metadata by ngrecall. The operations undertaken are:

• verifying that a SHA-512 hash calculated for the content of a remote object downloaded on recalling a file is equal to a SHA-512 hash stored in the dmapi.APXsh512 extended attribute of the file

• setting a missing SHA-512 hash metadata for a remote object on recalling or reversely premigrating a file (when ngrecall calculates the SHA-512 hash of the remote object). Setting a missing SHA-512 hash metadata is also disabled by specifying the parameter MetadataUpdate=false (see MetadataUpdate).

• checking the equality of SHA-512 hashes in the metadata of the copies of a remote object to the SHA-512 hash of a primary object when stamping remote location extended attributes for the copies at other storage endpoints. If the hashes are different, ngrecall does not stamp a remote location extended attribute referencing a copy of the remote object. To disable the check, passing the command line option --skip-check-uuid or specifying the paramater CheckUUID=false is also necessary (see CheckUUID).

• verifying that the size and SHA-512 hash of a remote object are equal to the size and SHA-512 hash of a local file on deleting the remote object

If the parameter CheckHash is True, the aforementioned operations are enabled. If the parameter CheckHash is False, the aforementioned operations are disabled.

The default value is True.

Passing the option --skip-check-hash to ngrecall implicitly sets CheckHash=false for all storage endpoints described in the master configuration file.

Syntax:

CheckHash=true|false

Example:

CheckHash=false

CheckUUID

This parameter controls checking UUIDs of remote objects by the ngrecall and ngscan tools and updating UUID metadata of remote objects by the ngrecall tool.

If the parameter is equal to True, checking UUIDs of remote objects and updating their UUID metadata is enabled. If the parameter is equal to False, checking UUIDs of remote objects and updating their UUID metadata is disabled. Setting the parameter to False has the same effect as passing the option --skip-check-uuid to the tools, but the effect is restricted to a storage endpoint described in the master configuration file.

For ngrecall, setting the parameter to False results in the following behaviour:

• when recalling a file, do not verify that the value of dmapi.APXguuid extended attribute of the file and the UUID of a remote object fetched from its metadata are equal

• when recalling or reversely premigrating a file, do not set a missing UUID metadata for a remote object

• when reversely stubbing/premigrating a file or syncing its metadata, do not verify that the UUID-like name suffix of a remote object matches a UUID fetched from the metadata of the remote object

• when scanning storage endpoints for reverse stubbing/premigration or metadata syncing, do not check that the UUID-like name suffix of a remote object matches a UUID fetched from the metadata of the remote object. Disabling this UUID check can result in a larger set of remote objects being analyzed for finding the latest remote object instance (see the parameter ObjectInstances) and can result in more situations where the UUID-like suffix of a remote object name is truncated when inferring a local file name.

• when determining whether a remote object is a copy of a remote object in a primary storage endpoint to stamp a remote location extended attribute for the copy on reverse stubbing/premigration, do not check if UUIDs of the two objects fetched from their metadata are equal. Disabling this check also requires disabling checking the equality of SHA-512 hashes of the two objects (see the parameter CheckHash).

• when analyzing whether it is safe to delete a remote object, do not read the extended attribute dmapi.APXguuid of a local file, do not fetch the UUID of the remote object from its metadata, and do not verify that they are equal. Disabling this UUID check can result in data loss---deleting an incorrect remote object for a local file by ngrecall.

For ngscan, setting the parameter to False results in the following behaviour:

• do not check that the UUID-like name suffix of a remote object matches a UUID fetched from the metadata of the remote object. Disabling this UUID check can result in a larger set of remote objects being analyzed for finding the latest remote object instance (see the parameter ObjectInstances) and can result in more situations where the UUID-like suffix of a remote object name is truncated when printing the field name_no_uuid (nnu) or name_no_uuid_raw (nnur) and when inferring a local file name printed in the field fln or fln_raw (fr).

• when determining remote object names for local files specified on the command line by reading their remote location extended attributes, disable verifying that the value of dmapi.APXguuid extended attribute of a local file and the UUID of a remote object fetched from its metadata are equal

The default value for the parameter is True.

Syntax:

CheckUUID=true|false

Example:

CheckUUID=false

LocalFileRegex

Defines the pathname match, in POSIX regex (see man 7 regex), used to determine which filesystem paths will trigger migrations to this storage target. It also controls the generation of the file path used in the remote storage.

Syntax:

LocalFileRegex=/<mountpoint>/([path filter].+)

The following example will cause this storage target to be used as a migration target for all files on file system /mmfs1/, and store the files in remote storage with their full path minus the /mmfs1/ component:

Example: all filesystem objects can be migrated to this target

LocalFileRegex=/mmfs1/(.+)

The following example will cause this storage target to be used as a migration target for all files underneath /mmfs1/data/fileset1/, and store the files in remote storage with the path data/fileset1/path/to/file:

Example: targeted migration

LocalFileRegex=/mmfs1/(data/fileset1/.+)

LocalFileRegexExclude

Defines a regular expression in the POSIX format (see man 7 regex) for the exclusion of a subset of pathnames matching a regular expression specified by the parameter LocalFileRegex. If a pathname matches both the parameter LocalFileRegex and the parameter LocalFileRegexExclude, ngmigrate will not migrate the file to the storage target, and all operations involving the use of the pathname for the storage target will fail.

Syntax:

LocalFileRegex=<regular_expression>

The following example excludes the content of tmp sub-folder from pathnames under the folder /mmfs1/myproject/.

Example: excluding a sub-folder

LocalFileRegex=/mmfs1/(myproject/.+)
LocalFileRegexExclude=/mmfs1/myproject/tmp/.*

The following example excludes the content of tmp and .snapshots sub-folders from pathnames under the folder /mmfs1/myproject/.

Example: excluding two sub-folders

LocalFileRegex=/mmfs1/(myproject/.+)
LocalFileRegexExclude=/mmfs1/myproject/(tmp|\.snapshots)/.*

The following example excludes the sub-folder .snapshots and files matching the glob pattern *.core at all nesting levels under /mmfs1/myproject/. The character . in \.snapshots is escaped because it must exactly match the character . in a pathname. The character . in /.* is not escaped because it can match any character in a pathname.

Example: excluding a sub-folder and files matching a pattern at all nesting levels

LocalFileRegex=/mmfs1/(myproject/.+)
LocalFileRegexExclude=.*/(\.snapshots/.*|[^/]*\.core)

LocalSymlinkTargets

Defines the behaviour of symbolic link restoration by ngrecall on reverse stubbing/premigration.

The ngrecall tool processes each symbolic link encountered by:

• Converting to an absolute path with transposing paths containing multiple consequent slashes, ., and ...

• Validating the computed path against the defined behaviour.

• If the computed path is restricted by the defined behaviour, ngrecall does not restore the symbolic link.

The parameter has the following supported values:

• endpoint --- restricts ngrecall to restoring symbolic links referencing directories and files contained within the namespace of the endpoint target

• any --- restricts ngrecall to restoring symbolic links referencing directories and files contained within the namespace of all endpoint targets. Symbolic link computed paths are validated against LocalFileRegex parameters for all storage endpoints. A symbolic link is restored if the computed path matches a regular expression for at least one endpoint

• regex:REGULAR_EXPRESSION --- restricts ngrecall to restoring symbolic links where computed paths match the defined regular expression

The default value is "endpoint".

Syntax:

LocalSymlinkTargets=endpoint|any|regex:<regular_expression>

Example:

LocalSymlinkTargets=regex:/mmfs1(/.*)?

MetadataUpdate

This parameter controls updating the metadata of already existing remote objects:

• by ngmigrate on migration or premigration if the UUID metadata of a remote object is equal to the UUID of a local file (stored in the extended attribute dmapi.APXguuid), and the SHA-512 hash metadata of a remote object is equal to a SHA-512 hash calculated for the content of the local file. If the parameter is True, and local file information does not match remote object metadata, ngmigrate updates the remote object metadata to match the local file information. If the parameter is False, ngmigrate leaves the remote object as is in this case. Setting the parameter to False has the same effect as passing the option --skip-metadata-update to ngmigrate, but the effect is restricted to a storage endpoint described in the master configuration file.

• by ngrecall on recall or reverse premigration if a remote object does not have the UUID metadata and SHA-512 hash metadata (usually when the remote object was uploaded by a third-party tool). If the parameter is True, ngrecall sets the UUID metadata of a remote object equal to the extended attribute dmapi.APXguuid of the local file and sets the SHA-512 hash metadata of a remote object equal to the extended attribute dmapi.APXsh512 of the local file. If the parameter is False, ngrecall does not set the UUID metadata and SHA-512 hash metadata of a remote object. Setting the parameter to False has the same effect as passing the option --skip-metadata-update to ngrecall, but the effect is restricted to a storage endpoint described in the master configuration file.

The default value for the parameter is True.

Syntax:

MetadataUpdate=true|false

Example:

MetadataUpdate=false

ObjectInstances

Specifies the selection of object instances by the ngrecall, ngscan, and ngcopy tools. The parameter can have the value either "latest" or "all". Setting the parameter to "all" has the same effect as passing the option --all-obj-instances to the tools, but the effect is restricted to a storage endpoint described in the master configuration file.

For ngrecall, a parameter value affects the following behaviour:

• latest. When reversely stubbing, reversely premigrating, or metadata syncing (without the use of a remote location extended attribute) a file, find and use the latest remote object instance (with or without a UUID suffix) corresponding to the file. The latest remote object instance has the most recent content of a migrated or premigrated file. When scanning the storage endpoint with reverse stubbing, reverse premigrating, or metadata syncing, filter only latest instances of remote objects for processing.

• all. When reversely stubbing, reversely premigrating, or metadata syncing (without the use of a remote location extended attribute) a file, use a remote object with a name inferred from the name of the file; if a remote object name ends with a UUID suffix, the file name must end with the suffix too. The remote object can correspond to older content of a migrated or premigrated file (if a file name does not end with a UUID suffix, the remote object may correspond to the oldest file content). When scanning the storage endpoint with reverse stubbing, reverse premigrating, or metadata syncing, process all remote object instances.

For ngscan, a parameter value affects the following behaviour:

• latest. Find and list only latest instances of remote objects.

• all. Do not filter latest instances of remote objects for listing.

For ngcopy, a parameter value affects the following behaviour:

• latest. Copy only latest instances of remote objects.

• all. Copy all instances of remote objects in the order of object instance migration time.

The default value for the parameter is "latest".

Syntax:

ObjectInstances=latest|all

Example:

ObjectInstances=all

OverwriteRemote

This parameter controls overwriting remote objects, folders, and symbolic links in a storage endpoint by ngmigrate.

If this parameter is True, ngmigrate overwrites remote objects, folders, and symbolic links without fetching and analyzing their metadata. Overwriting a remote object means that ngmigrate never appends a UUID suffix to the name of the remote object if it has a UUID different from the UUID of a local file. Overwriting a remote folder means that ngmigrate always updates folder metadata.

Setting the parameter to True has the same effect as passing the option --overwrite-remote on the command line, but the effect is restricted to a particular storage endpoint described in the master configuration file.

If the parameter is False, the default behavior takes place, where ngmigrate fetches and analyzes the metadata of a remote object, folder, or symbolic link to decide whether it is necessary to upload the remote object, folder, or symbolic link or only update its changed metadata. If the UUID of a local file is different from the UUID of the remote object, ngmigrate uploads a new remote object instance with a name suffix containing the UUID of the local file.

The default value is False.

Syntax:

OverwriteRemote=true|false

Example:

OverwriteRemote=true

RemoteLocationXAttrRegex

Defines a regular expression for matching a remote location extended attribute and converting it to a remote object name. The extended attribute is added to a file upon migration or premigration.

If the extended attribute matches the regular expression, the parameter RemoteLocationXAttrRegex along with the parameters RetrieveObjectBasePath and RetrieveObjectName in a storage target configuration file determine how to convert the extended attribute to a remote object name when performing the following operations:

• by ngmigrate---on migration, premigration, and metadata syncing

• by ngrecall---on manual recalling, transparent recalling, metadata syncing, and remote object deletion

• by ngscan---on listing remote objects for local files specified on the command line

If the parameter RemoteLocationXAttrRegex is absent or equal to an empty string, ngmigrate, ngrecall, and ngscan do not read a remote location extended attribute and always determine a remote object name based on a local file name. This behaviour is equivalent to passing the option --ignore-rmtlc to ngmigrate, ngrecall, or ngscan where the effect of that option is limited to a storage endpoint.

Warning

As a local file name is unavailable for transparent recalls, and they always rely on a remote location extended attribute to determine the name of a remote object to download, transparent recalls will not work for storage endpoints where the parameter RemoteLocationXAttrRegex is absent or equal to an empty string.

Syntax:

RemoteLocationXAttrRegex=<identifier>:(.+)

Example:

RemoteLocationXAttrRegex=target1:(.+)

Note that regular expressions specified by LocalFileRegex and RemoteLocationXAttrRegex behave as if they are prefixed with the caret anchor ^ (match beginning of text) and finalized with the dollar anchor $ (match end of text).

RestoreACL

Specifies whether to restore ACLs for files and directories by ngrecall. ACLs are restored when the files and directories are created or overwritten (using the --overwrite-local option) on fetching, reverse premigrating, reverse stubbing or updated on syncing their metadata.

If the parameter is set to True, the ACLs are restored if corresponding remote objects and folders have the ACL metadata.

If the parameter is set to False, or the remote objects and folders do not have the ACL metadata, the ACLs are left as is. In this case, their content depends on permissions, owners, groups, and parent ACLs for the files and directories.

Setting the parameter to False has the same effect as passing the option --no-restore-acl to ngrecall, but the effect is restricted to a storage endpoint described in the master configuration file from where the ACL metadata is fetched.

The default value for the parameter is True.

Syntax:

RestoreACL=true|false

Example:

RestoreACL=false

StorageKey

Defines a custom identifier attributable to the storage endpoint.

See the description of this keyword in Multi-Target Support.

Syntax

StorageKey=<char><char>

Examples

StorageKey=zz

Writable

Specifies whether the storage target supports read/write operations (Writable=true) or is read-only (Writable=false).

Read-only storage targets have the following restrictions:

• The ngmigrate tool excludes read-only storage targets from a list of matching storage targets for migration, premigration, or metadata syncing

• The ngrecall tool treats read-only storage targets as storage targets with the DeleteOnRecall=false parameter set

• The ngrecall tool excludes read-only storage targets from a list of matching storage targets for remote object deletion explictly requested by passing the --delete-remote option

• The ngrecall tool does not attempt to set the UUID and hash metadata for remote objects in read-only storage targets

The default value is True.

Syntax:

Writable=true|false

Example:

Writable=false

Section [FileMatch <file match set>]

These sections specify stub sizes for various names and paths of local files.

All keywords in these sections are optional.

Keyword	Default Value
BasenameMask	Null
PathnameRegex	".+"
StubSize	-1 (I.E. premigrate)

The program ngmigrate applies those stub sizes to local files when migrating them. The program ngrecall applies those stub sizes to local files when reversely stubbing their remote objects.

The header of every file match section must contain the name of a file match set.

Syntax:

[FileMatch <file match set>]

Example:

A file match section describing text files might have the header

[FileMatch text]

The names of all file match sets described in a master configuration file must be different.

The programs ngmigrate and ngrecall process file match sections in the order they are present in a master configuration file. If a section matches a local file, ngmigrate or ngrecall applies the section to the file and does not process subsequent file match sections for that file.

The last file match section described in a master configuration file might specify the default stub size to be applied to a local file when no previous sections matched the file. For example, such default file match section might look like this:

Example: A default file match section, specifying a StubSize of 0 bytes

[FileMatch default]
; Use zero stub size for all other files.
StubSize=0

Every file match section can contain the parameters "BasenameMask", "PathnameRegex", and "StubSize". The parameters "BasenameMask" and "PathnameRegex" specify conditions joined by logical AND.

BasenameMask

Specifies a glob pattern for the base name part of a matched file name.

Separate with '|' (the pipe character) multiple glob patterns joined by logical OR. Example:

Syntax:

BasenameMask=<glob pattern>

If this parameter is absent, all file names will be matched.

Example:

BasenameMask=*.wav|*.mp3|*.ogg

PathnameRegex

Specifies a regular expression for matching against a resolved absolute file name.

Syntax:

PathnameRegex=<regular expression>

If this parameter is absent, all paths will be matched.

Example:

PathnameRegex=.*/images/.*

StubSize

Specifies the stub size in bytes.

If the stub size exceeds the size of a file, or if the stub size is -1, ngmigrate or ngrecall will premigrate the file.

Syntax:

StubSize=<stub size in bytes (or -1)>

The default value is -1; i.e. the file will be premigrated.

Example:

StubSize=1024

Cloud Storage Requirements

Cloud (Object) storage providers have specific requirements for configuring access accounts and providing authentication. These requirements are described below for each storage provider.

It is advised that security best practice be implemented as detailed and updated time to time by the cloud storage provider.

Amazon S3 and S3 Compatible Storage

A bucket must be created which allows multipart upload - refer to Multipart Upload API and Permissions or the documentation for the S3 compatible storage provider.

The authentication details for the user account with privleges to access the bucket and the bucket name must be added to the Amazon S3 configuration file.

Secure the bucket by applying the principal of least privilege. Grant the account for Ngenea HSM operations the minimal required operations as detailed in Amazon S3 Security.

Microsoft Azure Storage

A container must be created.

The authentication details (account name, account key) for the user account with privleges to access the bucket and the bucket name must be added to the Azure configuration file.

Secure the bucket by applying the principal of least privilege. Grant the account for Ngenea HSM operations the minimal required operations as detailed in Azure Blob Storage Security.

Google Cloud Storage

A bucket must be created.

An administrator should create a service account and enable the the Compute Engine API. Refer to the Google Dashboard.

The service account access keys should be downloaded to the local system as JSON and referenced in the Google configuration file. Ensure the JSON access account keys are only readable by those user(s) which require access to the file, typically root.

Secure the bucket by applying the principal of least privilege. Grant the service account for Ngenea HSM operations the minimal required operations as detailed in Google Cloud Storage Security.