Spectra Logic BlackPearl Target¶
DataCore Software Ngenea HSM software is fully certified by Spectra Logic as an approved BlackPearl client.
Deprecated since version 1.32: Ngenea HSM 1.32 deprecates the Spectra Logic DS3 protocol. Future releases containing the DS3 protocol are limited to bugfixes and security patches.
Ngenea HSM requires configuration and activation of the BlackPearl's Spectra DS3 Service. Please refer to the relevant section of a BlackPearl User Guide.
For Spectra Logic Veil with BlackPearl, the AmazonS3 target type must be configured. Veil is S3 compatible and does not support DS3 protocol.
The following example describes the information required to be present withing the Storage Target Configuration File to enable migration to the specified bucket.
Example Configuration File¶
[General]
Endpoint=https://IPADDRESS_OF_BLACKPEARL:443
AccessKeyId=MyAccessKeyId
SecretAccessKey=MySecretAccessKey
Bucket=MyBucketName
RemoteLocationXAttr=blackpearl:$1
RetrieveObjectName=$1
StoreObjectName=$1
CacheFullMaxWaitSeconds=300
CacheFullRetryAfterSeconds=20
CacheLoadRetryAfterSeconds=20
JobCompletionPollSeconds=20
JobCompletionDoWait=true
DeleteOnRecall=true
Configuration Keywords¶
Required keywords
AccessKeyId |
Bucket |
Endpoint |
SecretAccessKey |
Optional keywords
Keyword |
Default Value |
|---|---|
ACLSave |
False |
CacheFullMaxWaitSeconds |
1800 secs |
CacheFullRetryAfterSeconds |
300 secs |
CacheLoadRetryAfterSeconds |
300 secs |
DeleteOnRecall |
False |
JobCompletionDoWait |
False |
JobCompletionPollSeconds |
20 secs |
MaxKeysListObjects |
1000 |
MaxUploadSize |
10 MB |
MaxUploadThreadCount |
Number of cores |
MaxXAttrNativeMetadataSize |
0 |
RemoteLocationXAttr |
"$0" |
RetrieveObjectBasePath |
"" |
RetrieveObjectName |
"$0" |
SDKLogging |
False |
ShadowFolderMetadataSave |
False |
StoreObjectBasePath |
"" |
StoreObjectName |
"$0" |
AccessKeyId¶
Defines the DS3 Access ID for a specified BlackPearl User Account. Please refer to the relevant section of the BlackPearl User Guide.
Syntax
AccessKeyId=<Spectra DS3 Access Key>
Example
AccessKeyId=ABCDEFGHIJLKMNOPQRST
Bucket¶
Defines the Bucket Name of the target BlackPearl Bucket. This bucket must exist on the BlackPearl storage target prior to migrating data using Ngenea HSM. THe bucket must be configured with the Versioning type set to Keep Latest. (see the Create a Data Policy section of the BlackPearl User Guide).
Syntax
Bucket=<Bucket Name>
Example
Bucket=MyBucketName
Endpoint¶
Defines the Hostname or IP Address, protocol and port whereby Ngenea HSM can access the BlackPearl Appliance.
Syntax
Endpoint=<http|https>://<Hostname or IP Address>[:port]
Example
Endpoint=https://blackpearl.example.com:443
SecretAccessKey¶
The DS3 Secret Key for a specified BlackPearl User Account. Please refer to the relevant section of the BlackPearl User Guide.
Syntax
SecretAccessKey=<Spectra DS3 Secret Key>
Example
SecretAccessKey=abcdefghijklmnopqrstuvwxyz+abcdefghijklm
ACLSave¶
Defines whether ngmigrate saves the ACLs of local files in the metadata of remote objects and whether the ACLs of local directories are saved in shadow metadata remote objects (provided that the parameter ShadowFolderMetadataSave is True).
If this parameter is False, ngrecall does not attempt to restore the ACLs of local files and created local directories on reverse stubbing/premigration.
The default value is False.
Syntax
ACLSave=<True|False>
Example
ACLSave=True
CacheFullMaxWaitSeconds¶
Defines the maximum number of seconds Ngenea HSM waits for space to be made available in the BlackPearl Appliance cache for migration of new data prior to cancelling the current job if no space is made available.
Syntax
CacheFullMaxWaitSeconds=<time in seconds>
Example
CacheFullMaxWaitSeconds=300
CacheFullRetryAfterSeconds¶
Defines the interval in seconds upon which Ngenea HSM will poll the BlackPearl Appliance to verify if the cache has free space.
Syntax
CacheFullRetryAfterSeconds=<time in seconds>
Example
CacheFullRetryAfterSeconds=20
CacheLoadRetryAfterSeconds¶
Defines the Interval in seconds upon which Ngenea HSM will poll the BlackPearl Appliance to verify if retrieved files are present in the cache.
Syntax
CacheLoadRetryAfterSeconds=<time in seconds>
Example
CacheLoadRetryAfterSeconds=20
DeleteOnRecall¶
Defines whether to delete the data from the remote BlackPearl target when successfully recalled back to the file system.
Setting this to False will result in files entering a "premigrated" state on successful recall.
Syntax
DeleteOnRecall=<True|False>
Example
DeleteOnRecall=True
JobCompletionDoWait¶
Defines whether Ngenea HSM waits for a BlackPearl Job to complete prior to exiting the current operation.
Setting this to 'true' will cause ngmigrate to wait until the BlackPearl has secured data to tape from the BlackPearl cache before completing.
Setting this to 'false' will cause ngmigrate to return once the data has been successfully uploaded to the BlackPearl cache.
Syntax
JobCompletionDoWait=<true|false>
Example
JobCompletionDoWait=true
JobCompletionPollSeconds¶
Defines the interval in seconds upon which Ngenea HSM will poll the BlackPearl Appliance to verify if an ongoing job has completed.
Syntax
JobCompletionPollSeconds=<time in seconds>
Example
JobCompletionPollSeconds=20
MaxKeysListObjects¶
The maximum number of remote object names (keys) returned in a response to a "get bucket" request when listing remote objects by ngscan. If there are more remote objects, they are returned in responses to subsequent "get bucket" requests.
The default value for this parameter is 1000.
Syntax
MaxKeysListObjects=<# of keys>
Example
MaxKeysListObjects=100
MaxUploadSize¶
This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate parts equal to this size (except the final part, which may be smaller).
The minimum value of this setting is 10485760 (10MB). The maximum value is determined by the BlackPearl target.
If this parameter is unset, the part sizes will be determined by the BlackPearl target (this is the recommended configuration).
Syntax
MaxUploadSize=<size_in_bytes>
Example
MaxUploadSize=10737418240
MaxUploadThreadCount¶
Defines the maximum number of threads an ngmigrate command will use when uploading files to the BlackPearl target.
Defaults to the number of CPU cores in the system.
Adjusting this parameter where network bandwidth is limited, or multiple ngmigrate commands are executed in parallel, may be beneficial.
Syntax
MaxUploadThreadCount=<# of threads>
Example
MaxUploadThreadCount=8
MaxXAttrNativeMetadataSize¶
Specifies the total size in bytes allocated for storing extended attributes of a local file or directory as metadata elements.
Ngenea HSM supports storing of extended attributes under the trusted. and user. namespaces.
The size of each metadata element for storing an extended attribute is calculated as the number of bytes of an encoded element key plus the number of bytes of an encoded element value.
Metadata element keys and values are encoded to preserve case and special characters. Keys have a constant prefix to distinguish metadata elements for storing extended attributes from other metadata elements.
If the size of all metadata elements exceeds the value of MaxXAttrNativeMetadataSize, ngmigrate repeatedly discards the largest extended attribute until the total size of remaining elements is less than or equal to the value of MaxXAttrNativeMetadataSize.
A warning is issued for every extended attribute that was not stored due to the discarding of the metadata element.
Valid settings are between 0 and 16384.
A current limitation on metadata size for BlackPearl is 4096.
A recommended setting for MaxXAttrNativeMetadataSize is 2048 --- a half of that limitation.
The default parameter value is 0; that is, no limit will be applied.
Syntax
MaxXAttrNativeMetadataSize=<maximum xattr metadata size>
Example
MaxXAttrNativeMetadataSize=1024
RemoteLocationXAttr¶
Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the Master Configuration File.
Syntax
RemoteLocationXAttr=targetname:<LocalFileRegexReference>
Example
RemoteLocationXAttr=blackpearl:$1
RetrieveObjectBasePath¶
Defines a base path for downloading objects from the remote storage target. This base path is prepended to a path composed using the RetrieveObjectName parameter to obtain the full path of a remote object to retrieve. By default, this base path also acts as a base path for listing objects on the remote storage target by ngscan.
If a non-empty base path does not end with /, then / is implicitly appended to the base path.
The default value for this parameter is the empty string: do not prepend any string to a path composed using the RetrieveObjectName parameter and list objects on the remote storage target starting at its root.
The value of this parameter typically matches the value of StoreObjectBasePath parameter.
Example
RetrieveObjectBasePath=dir1/subdir1
RetrieveObjectName¶
Defines a mapping from the value of a remote location (APXrmtXX) DMAPI extended attribute of a local file to the name suffix of an associated object on the remote storage target. The value of RetrieveObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to retrieve (download).
In a value of this parameter, the reference $0 represents the entire value of a remote location (APXrmtXX) DMAPI extended attribute.
The references $1, $2, ..., $9 represent parts of that entire value matched to parts of a regular expression specified by RemoteLocationXAttrRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
The default value for this parameter is $0.
The value of this parameter typically matches the value of StoreObjectName parameter.
Example
RetrieveObjectName=$1
SDKLogging¶
Specifies a log level for the Spectra DS3 C SDK library.
Log messages are prefixed "BlackPearl: " and are sent to the same log target as other log messages.
Syntax
SDKLogging=<error|warn|info|debug|trace|false>
Boolean "True" is equivalent to "trace".
Example
SDKLogging=True
The default value is "False" which disables logging.
ShadowFolderMetadataSave¶
Defines whether ngmigrate saves the ACLs and permissions (file mode, owner, and group) of local directories in shadow metadata objects for remote folders when migrating files (i.e. passing file names on the command line).
When migrating directories (i.e. passing directory names on the command line), ngmigrate always saves their ACLs and permissions.
Shadow metadata objects have the names FOLDERNAME/, where FOLDERNAME is a remote folder name.
Saved information can be restored by ngrecall when recreating local directories on reverse stubbing/premigration. Information about a local directory includes its file mode, owner, and group. If the parameter ACLSave is True, the information also includes the ACL of this local directory.
If the parameter ShadowFolderMetadataSave is False, ngmigrate does not save information about local directories when migrating files (but not directories), and ngrecall will recreate the local directories with ACLs and permissions (file mode, owner, and group) as determined by the local file system environment unless specifically overridden with the --default-<option> arguments of ngrecall.
The default value is False.
Syntax
ShadowFolderMetadataSave=<True|False>
Example
ShadowFolderMetadataSave=True
StoreObjectBasePath¶
Defines a base path for uploading objects to the remote storage target. This base path is prepended to a path composed using the StoreObjectName parameter to obtain the full path of a remote object to store. Additionally, this base path can act as a base path for listing objects on the remote storage target by ngscan.
If a non-empty base path does not end with /, then / is implicitly appended to that path.
The default value for this parameter is the empty string: do not prepend any string to a path composed using the StoreObjectName parameter and list objects on the remote storage target starting at its root.
The value of this parameter typically matches the value of RetrieveObjectBasePath parameter.
Example
StoreObjectBasePath=dir1/subdir1
StoreObjectName¶
Defines a mapping from the name of a local file to the name suffix of an associated object on the remote storage target. The value of StoreObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to store (upload).
In a value of StoreObjectName parameter, the reference $0 represents the entire absolute name of a local file.
The references $1, $2, ..., $9 represent parts of that entire name matched to parts of a regular expression specified by LocalFileRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
The default value for this parameter is $0.
The value of this parameter typically matches the value of RetrieveObjectName parameter.
Example
StoreObjectName=$1