Azure Blob Storage Type¶
The following section describes how to setup an Azure storage target.
Azure Storage Targets require several key pieces of information. The following example describes the information required to be present within the Storage Target Configuration File to enable migration to the specified account and container.
Example Configuration File¶
[General]
StorageAccount=MyStorageAccount
AccessKey=MyAccessKey
Container=MyContainerName
Scheme=HTTPS
RemoteLocationXAttr=target1:$1
RetrieveObjectName=$1
StoreObjectName=$1
DeleteOnRecall=True
Configuration Keywords¶
Required keywords
AccessKey / CredentialsFile |
Container |
StorageAccount |
Optional keywords
Keyword |
Default Value |
|---|---|
ACLSave |
False |
DeleteOnRecall |
False |
EscapeNames |
True |
MaxCopyThreadCount |
MaxUploadThreadCount |
MaxDownloadSize |
20 MB |
MaxDownloadThreadCount |
Number of cores |
MaxKeysListObjects |
5000 |
MaxMetadataReadThreadCount |
MaxDownloadThreadCount |
MaxRetryCount |
3 |
MaxUploadSize |
20 MB |
MaxUploadThreadCount |
Number of cores |
MaxXAttrNativeMetadataSize |
0 |
Port |
Null |
RemoteLocationXAttr |
"$0" |
RetrieveObjectBasePath |
"" |
RetrieveObjectName |
"$0" |
Scheme |
"https" |
SDKLogging |
False |
ShadowFolderMetadataSave |
False |
StorageClass |
Null |
StoreObjectBasePath |
"" |
StoreObjectName |
"$0" |
AccessKey / CredentialsFile¶
Defines the storage account access key used to authenticate to Azure.
For further information regarding Azure Account Access Keys please refer to the Microsoft Azure Documentation.
The AccessKey may be defined in the storage target configuration file with the keyword AccessKey, or in a separate file referenced by the keyword CredentialsFile.
Syntax
AccessKey=<Azure Account Access Key>
or
CredentialsFile=<file-name>
Example
AccessKey=ABCDEFGHIJLKMNOPQRST
or
CredentialsFile=access_key
Container¶
Defines the 'container' into which files are migrated. This container must exist on the Azure storage target prior to migrating data using Ngenea HSM.
For further information regarding Azure Containers and Blobs please refer to the Microsoft Azure Documentation.
Syntax
Container=<Name of the Azure container>
Example
Container=MyContainerName
StorageAccount¶
Defines the storage account name used to authenticate to Azure.
For further information regarding Azure storage accounts please refer to the Microsoft Azure Documentation.
Syntax
StorageAccount=<Account Name>
Example
StorageAccount=MyStorageAccount
ACLSave¶
Defines whether ngmigrate saves the ACLs of local files in the metadata of remote objects and whether the ACLs of local directories are saved in shadow metadata remote objects (provided that the parameter ShadowFolderMetadataSave is True).
If this parameter is False, ngrecall does not attempt to restore the ACLs of local files and created local directories on reverse stubbing/premigration.
The default value is False.
Syntax
ACLSave=<True|False>
Example
ACLSave=True
DeleteOnRecall¶
Defines whether to delete the data from the remote Azure target when successfully recalled back to the file system.
Setting this to False will result in files entering a "premigrated" state on successful recall.
Syntax
DeleteOnRecall=<True|False>
Example
DeleteOnRecall=True
EscapeNames¶
Defines whether to percent-encode the names of files or directories containing special characters (E.G. spaces / extended unicode character sets) in the names of remote objects when migrating or recalling.
If this parameter is False the names of remote objects are equal to the names of corresponding local files and directories upon successful migration.
Not all storage endpoints support storing of objects without percent-encoding enabled. In such cases this paramater should be set to True.
Where storage endpoints do support non-percent-encoded object names this parameter can enable remote processing workflows whereby applications can directly read data held in storage endpoints via the object URI or direct mount without path translation. Consider setting the parameter ShadowFolderMetadataSave along with EscapeNames to False in this case.
The default value is True.
Syntax
EscapeNames=<True|False>
Example
EscapeNames=False
MaxCopyThreadCount¶
Defines the maximum number of threads an ngcopy command will use when copying or moving objects to the Azure target. Each thread copies an individual object. Moving an object from a source location to a target location is implemented as copying the object to the target location with subsequent deleting the object at the source location.
The default value of this parameter is equal to the value of MaxUploadThreadCount parameter.
As copying an object does not require transferring its content to the Azure target, specifying a value of this parameter greater than the value of MaxUploadThreadCount may be beneficial.
Syntax
MaxCopyThreadCount=<# of threads>
Example
MaxCopyThreadCount=32
MaxDownloadSize¶
As for the uploads, downloads also use a multi-part transfer process. Files larger than 1MiB are downloaded in parts. This will split files into multiple parts and transfer each part separately.
This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate transactions equal to this size (except the final part, which may be smaller).
Set this value to be less than or equal to that supported by the remote storage target.
Valid settings are between 1048576 (1MiB) and 104857600 (100MiB).
This optional parameter defaults to 20MiB if not explicitly defined.
Syntax
MaxDownloadSize=<size_in_bytes>
Example
MaxDownloadSize=134217728
MaxDownloadThreadCount¶
Defines the maximum number of threads a ngrecall command will use when downloading files from the Azure Storage target. The threads download multiple files in parallel and download multiple parts of an individual file in parallel.
Defaults to the number of CPU cores in the system.
Adjusting this parameter where network bandwidth is limited, or multiple ngrecall commands are executed in parallel, may be beneficial.
Syntax
MaxDownloadThreadCount=<# of threads>
Example
MaxDownloadThreadCount=8
MaxKeysListObjects¶
The maximum number of remote object names (keys) returned in a response to a "list blobs" request when listing remote objects by ngscan. If there are more remote objects, they are returned in responses to subsequent "list blobs" requests.
The default value for this parameter is 5000.
Syntax
MaxKeysListObjects=<# of keys>
Example
MaxKeysListObjects=2000
MaxMetadataReadThreadCount¶
Defines the maximum number of threads an ngmigrate, ngrecall, ngscan, or ngcopy command will use to fetch remote object metadata. Each thread issues a single request for fetching object metadata or downloads a single shadow metadata object for a folder.
The default value of this parameter is equal to the value of MaxDownloadThreadCount parameter.
As object metadata fetching requests do not transfer object content, specifying a value of this parameter greater than the value of MaxDownloadThreadCount may be beneficial.
Syntax
MaxMetadataReadThreadCount=<# of threads>
Example
MaxMetadataReadThreadCount=32
MaxRetryCount¶
Specifies the number of retries of operations to access an Azure storage endpoint.
Every subsequent attempt to perform an operation is done after an increasing delay.
By default, a failed operation is retried up to 3 times.
Syntax
MaxRetryCount=<number of retries>
Example
MaxRetryCount=5
MaxUploadSize¶
Files larger than 5MiB are uploaded using the multi-part transfer process. This will split files into multiple parts and transfer each part separately, with the parts reconstructed on the server.
This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate transactions equal to this size (except the final part, which may be smaller), and reconstructed on the Azure server into one object.
Set this value to be less than or equal to that supported by the remote storage target.
Valid settings are between 1048576 (1MiB) and 104857600 (100MiB).
This optional parameter defaults to 20MiB if not explicitly defined.
Syntax
MaxUploadSize=<size_in_bytes>
Example
MaxUploadSize=104857600
MaxUploadThreadCount¶
Defines the maximum number of threads an ngmigrate command will use when uploading files to the Azure target. The threads upload multiple files in parallel and upload multiple parts of an individual file in parallel.
Defaults to the number of CPU cores in the system.
Adjusting this parameter where network bandwidth is limited, or multiple ngmigrate commands are executed in parallel, may be beneficial.
Syntax
MaxUploadThreadCount=<# of threads>
Example
MaxUploadThreadCount=8
MaxXAttrNativeMetadataSize¶
Specifies the total size in bytes allocated for storing extended attributes of a local file as metadata elements.
Ngenea HSM supports storing of extended attributes under the trusted. and user. namespaces.
The size of each metadata element for storing an extended attribute is calculated as the number of bytes of an encoded element key plus the number of bytes of an encoded element value.
Metadata element keys and values are encoded to preserve case and special characters. Keys have a constant prefix to distinguish metadata elements for storing extended attributes from other metadata elements.
If the size of all metadata elements exceeds the value of MaxXAttrNativeMetadataSize, ngmigrate repeatedly discards the largest extended attribute until the total size of remaining elements is less than or equal to the value of MaxXAttrNativeMetadataSize.
A warning is issued for every extended attribute that was not stored due to the discarding of the metadata element.
Valid settings are between 0 and 16384.
A current limitation on metadata size for Azure is 8192.
A recommended setting for MaxXAttrNativeMetadataSize is 4096 --- a half of that limitation.
The default parameter value is 0; that is, no limit will be applied.
Syntax
MaxXAttrNativeMetadataSize=<maximum xattr metadata size>
Example
MaxXAttrNativeMetadataSize=1024
Port¶
For Azure Storage targets, do not include this parameter.
For non-Azure Storage targets, this parameter defines the TCP port to connect to on the remote storage.
The default value is 443 if this parameter is unspecified.
Syntax
Port:<TCP port number>
Example
Port:443
RemoteLocationXAttr¶
Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the Master Configuration File.
Syntax
RemoteLocationXAttr=targetref:<LocalFileRegexReference>
Example
RemoteLocationXAttr=target1:$1
RetrieveObjectBasePath¶
Defines a base path for downloading objects from the remote storage target. This base path is prepended to a path composed using the RetrieveObjectName parameter to obtain the full path of a remote object to retrieve. By default, this base path also acts as a base path for listing objects on the remote storage target by ngscan.
If a non-empty base path does not end with /, then / is implicitly appended to the base path.
The default value for this parameter is the empty string: do not prepend any string to a path composed using the RetrieveObjectName parameter and list objects on the remote storage target starting at its root.
The value of this parameter typically matches the value of StoreObjectBasePath parameter.
Example
RetrieveObjectBasePath=dir1/subdir1
RetrieveObjectName¶
Defines a mapping from the value of a remote location (APXrmtXX) DMAPI extended attribute of a local file to the name suffix of an associated object on the remote storage target. The value of RetrieveObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to retrieve (download).
In a value of this parameter, the reference $0 represents the entire value of a remote location (APXrmtXX) DMAPI extended attribute.
The references $1, $2, ..., $9 represent parts of that entire value matched to parts of a regular expression specified by RemoteLocationXAttrRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
The default value for this parameter is $0.
The value of this parameter typically matches the value of StoreObjectName parameter.
Example
RetrieveObjectName=$1
Scheme¶
Defines the transport protocol used to communicate with the Azure target.
ArcaStream / Pixit Media only supports HTTPS protocol for Ngenea HSM solutions.
This optional parameter defaults to HTTPS if not explicitly defined.
Syntax
Scheme=<PROTOCOL>
Example
Scheme=HTTPS
SDKLogging¶
Specifies a log level for the Azure Storage C++ SDK library. If a master configuration file defines multiple Azure storage endpoints, then a log level specified for the first accessed Azure storage endpoint will be in effect.
Log messages are prefixed "Azure: " and are sent to the same log target as other log messages.
Syntax
SDKLogging=<error|warn|info|verbose|false>
Boolean "True" is equivalent to "verbose".
The default value is "False" which disables logging.
ShadowFolderMetadataSave¶
Defines whether ngmigrate saves the ACLs and permissions (file mode, owner, and group) of local directories in shadow metadata objects for remote folders when migrating files (i.e. passing file names on the command line).
When migrating directories (i.e. passing directory names on the command line), ngmigrate always saves their ACLs and permissions.
Shadow metadata objects have the names FOLDERNAME/, where FOLDERNAME is a remote folder name.
Saved information can be restored by ngrecall when recreating local directories on reverse stubbing/premigration. Information about a local directory includes its file mode, owner, and group. If the parameter ACLSave is True, the information also includes the ACL of this local directory.
If the parameter ShadowFolderMetadataSave is False, ngmigrate does not save information about local directories when migrating files (but not directories), and ngrecall will recreate the local directories with ACLs and permissions (file mode, owner, and group) as determined by the local file system environment unless specifically overridden with the --default-<option> arguments of ngrecall.
The default value is False.
Syntax
ShadowFolderMetadataSave=<True|False>
Example
ShadowFolderMetadataSave=True
StorageClass¶
Defines a storage class for objects uploaded to the target container.
Valid storage classes are HOT, COOL, and ARCHIVE.
Syntax
StorageClass=<HOT|COOL|ARCHIVE>
Example
StorageClass=COOL
StoreObjectBasePath¶
Defines a base path for uploading objects to the remote storage target. This base path is prepended to a path composed using the StoreObjectName parameter to obtain the full path of a remote object to store. Additionally, this base path can act as a base path for listing objects on the remote storage target by ngscan.
If a non-empty base path does not end with /, then / is implicitly appended to that path.
The default value for this parameter is the empty string: do not prepend any string to a path composed using the StoreObjectName parameter and list objects on the remote storage target starting at its root.
The value of this parameter typically matches the value of RetrieveObjectBasePath parameter.
Example
StoreObjectBasePath=dir1/subdir1
StoreObjectName¶
Defines a mapping from the name of a local file to the name suffix of an associated object on the remote storage target. The value of StoreObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to store (upload).
In a value of StoreObjectName parameter, the reference $0 represents the entire absolute name of a local file.
The references $1, $2, ..., $9 represent parts of that entire name matched to parts of a regular expression specified by LocalFileRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
The default value for this parameter is $0.
The value of this parameter typically matches the value of RetrieveObjectName parameter.
Example
StoreObjectName=$1
Security¶
The permissions listed in the table below must be enabled to allow Ngenea HSM to correctly function. These permissions are included in Azure's built-in role Storage Blob Data Contributor.
Blob Service Operation |
Description |
|---|---|
List Blobs |
read (limited to the container) |
Put Blob |
write |
Get Blob |
read blob |
Get Blob Properties |
read blob properties (blob size, MIME type} |
Set Blob Properties |
write or update blob properties (MIME type) |
Get Blob Metadata |
read Ngenea HSM-specific attributes |
Set Blob Metadata |
write / update Ngenea HSM-specific attributes |
Delete Blob |
delete the blob |
Put Block |
write a block (part of a multi-block operation) |
Put Block List |
submit blocklist (completes a multi-block operation) |