Amazon S3 Type¶
The following section describes how to setup a Storage Target to an S3 compatible storage target.
S3 Storage Targets require several key pieces of information. The following example describes the information required to be present within the Storage Target Configuration File to enable migration to the specified end point and bucket.
Example Configuration File¶
[General]
Endpoint=endpoint.host.name
Port=443
AccessKeyId=MyAccessKeyId
SecretAccessKey=MySecretAccessKey
Bucket=MyBucketName
Region=eu-west-2
Scheme=HTTPS
RemoteLocationXAttr=target1:$1
RetrieveObjectName=$1
StoreObjectName=$1
DeleteOnRecall=True
Configuration Keywords¶
Required keywords
AccessKeyId |
Bucket |
Region |
SecretAccessKey / CredentialsFile |
Optional keywords
Keyword |
Default Value |
|---|---|
ACLSave |
False |
ConnectionTimeoutMs |
3000 |
DeleteOnRecall |
False |
Endpoint |
Null |
EscapeNames |
True |
MaxCopyThreadCount |
MaxUploadThreadCount |
MaxDownloadSize |
20 MB |
MaxDownloadThreadCount |
Number of cores |
MaxKeysListObjects |
1000 |
MaxMetadataReadThreadCount |
MaxDownloadThreadCount |
MaxRetryCount |
10 |
MaxUploadSize |
20 MB |
MaxUploadThreadCount |
Number of cores |
MaxXAttrNativeMetadataSize |
0 |
Port |
80 |
RemoteLocationXAttr |
"$0" |
RequestTimeoutMs |
3000 |
RetrieveObjectBasePath |
"" |
RetrieveObjectName |
"$0" |
Scheme |
"https" |
SDKLogging |
False |
SequentialDeletes |
False |
ShadowFolderMetadataMode |
auto |
ShadowFolderMetadataSave |
False |
SSLVerify |
True |
StorageClass |
Null |
StoreObjectBasePath |
"" |
StoreObjectName |
"$0" |
SymlinkMode |
object |
AccessKeyId¶
Defines the Account Access Key used to authenticate against the S3 target.
For further information regarding Amazon AWS Account Access Keys please refer to the Amazon Documentation.
Syntax
AccessKeyId=<Amazon Account Access Key>
Example
AccessKeyId=ABCDEFGHIJLKMNOPQRST
Bucket¶
Defines the bucket, or 'container', into which files are migrated. This bucket must exist on the S3 storage target prior to migrating data using Ngenea HSM.
For further information regarding Amazon AWS Buckets please refer to the Amazon Documentation.
Syntax
Bucket=<Name of the S3 bucket>
Example
Bucket=MyBucketName
Region¶
Defines the regional endpoint in use for the target bucket.
For further information regarding Amazon AWS Regions please refer to the Amazon Documentation.
Other S3 targets may have significantly different requirements for how to specify this parameter. Consult your S3 target documentation for further information.
Syntax
Region=<Name of the S3 Region>
Example
Region=eu-west-2
SecretAccessKey / CredentialsFile¶
Defines the Secret Key used to authenticate against the S3 target.
For further information regarding Amazon AWS Account Access Keys please refer to the Amazon Documentation.
The Secret Key may be defined in the storage target configuration file with the keyword SecretAccessKey, or in a separate file referenced by the keyword CredentialsFile.
Syntax
SecretAccessKey=<Amazon Secret Access Key>
or
CredentialsFile=<file-name>
Example
SecretAccessKey=abcdefghijklmnopqrstuvwxyz+abcdefghijklm
or
CredentialsFile=access_key
ACLSave¶
Defines whether ngmigrate saves the ACLs of local files in the metadata of remote objects and whether the ACLs of local directories are saved in shadow metadata remote objects (provided that the parameter ShadowFolderMetadataSave is True).
If this parameter is False, ngrecall does not attempt to restore the ACLs of local files and created local directories on reverse stubbing/premigration.
The default value is False.
Syntax
ACLSave=<True|False>
Example
ACLSave=True
ConnectionTimeoutMs¶
Specifies the timeout (in milliseconds) for opening a socket connection to an S3 storage endpoint.
Default parameter value is 3000.
Syntax
ConnectionTimeoutMs=<timeout in milliseconds>
Example
ConnectionTimeoutMs=2000
DeleteOnRecall¶
Defines whether to delete the data from the remote S3 target when successfully recalled back to the file system.
Setting this to False will result in files entering a "premigrated" state on successful recall.
Syntax
DeleteOnRecall=<True|False>
Example
DeleteOnRecall=True
Endpoint¶
For Amazon AWS S3 targets, do not include this parameter. It will be automatically determined by Amazon based on the region and bucket name.
For non-Amazon AWS S3 targets, this parameter defines the Hostname or IP Address whereby Ngenea HSM can access the S3 target.
Syntax
Endpoint=<Hostname or IP address>
Example
Endpoint=s3.host.domain
EscapeNames¶
Defines whether to percent-encode the names of files or directories containing special characters (E.G. spaces / extended unicode character sets) in the names of remote objects when migrating or recalling.
If this parameter is False the names of remote objects are equal to the names of corresponding local files and directories upon successful migration.
Not all storage endpoints support storing of objects without percent-encoding enabled. In such cases this paramater should be set to True.
Where storage endpoints do support non-percent-encoded object names this parameter can enable remote processing workflows whereby applications can directly read data held in storage endpoints via the object URI or direct mount without path translation. Consider setting the parameter ShadowFolderMetadataSave along with EscapeNames to False in this case.
The default value is True.
Syntax
EscapeNames=<True|False>
Example
EscapeNames=False
MaxCopyThreadCount¶
Defines the maximum number of threads an ngmigrate, ngrecall, or ngcopy command will use when copying objects to the S3 target. Each thread copies an individual object part.
Where ngmigrate and ngrecall perform operations to update the metadata of objects, this parameter also controls how many metadata updates can be performed in parallel.
The ngcopy tool copies objects to a different location, as per the parameters passed. Moving an object from a source location to a target location by ngcopy is implemented as copying the object to the target location with subsequent deleting the object at the source location.
The default value of this parameter is equal to the value of MaxUploadThreadCount parameter.
As copying an object does not require transferring its content to the S3 target, specifying a value of this parameter greater than the value of MaxUploadThreadCount may be beneficial.
Syntax
MaxCopyThreadCount=<# of threads>
Example
MaxCopyThreadCount=128
MaxDownloadSize¶
The maximum size of a downloaded file part. The ngrecall command can download multiple file parts in parallel. If a network error is experienced while downloading a file part, ngrecall retries downloading the failed file part.
Valid settings are between 5242880 (5MB) and 5368709120 (5GB).
This optional parameter defaults to 20MB if not explicitly defined.
Syntax
MaxDownloadSize=<size_in_bytes>
Example
MaxDownloadSize=10485760
MaxDownloadThreadCount¶
Defines the maximum number of threads an ngrecall command will use when downloading files from the S3 target. Each thread downloads an individual file part.
Defaults to the number of CPU cores in the system.
Adjusting this parameter where network bandwidth is limited, or multiple ngrecall commands are executed in parallel, may be beneficial.
Syntax
MaxDownloadThreadCount=<# of threads>
Example
MaxDownloadThreadCount=8
MaxKeysListObjects¶
The maximum number of remote object names (keys) returned in a response to a "get bucket" request when listing remote objects by ngscan. If there are more remote objects, they are returned in responses to subsequent "get bucket" requests.
The default value for this parameter is 1000. A value specified for this parameter should be less than or equal to that default value.
Syntax
MaxKeysListObjects=<# of keys>
Example
MaxKeysListObjects=100
MaxMetadataReadThreadCount¶
Defines the maximum number of threads an ngmigrate, ngrecall, ngscan, or ngcopy command will use to fetch remote object metadata. Each thread issues a single "head object" request or downloads a single shadow metadata object for a folder.
The default value of this parameter is equal to the value of MaxDownloadThreadCount parameter.
As "head object" requests do not transfer object content, specifying a value of this parameter greater than the value of MaxDownloadThreadCount may be beneficial.
Syntax
MaxMetadataReadThreadCount=<# of threads>
Example
MaxMetadataReadThreadCount=32
MaxRetryCount¶
Specifies the number of retries of operations to access an S3 storage endpoint.
Every subsequent attempt to perform an operation is done after an increasing delay.
By default, a failed operation is retried up to 10 times.
Syntax
MaxRetryCount=<number of retries>
Example
MaxRetryCount=5
MaxUploadSize¶
Files larger than 5MB are uploaded using the multi-part transfer process. This will split files into multiple parts and transfer each part separately, with the parts reconstructed on the server.
This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate transactions equal to this size (except the final part, which may be smaller), and reconstructed on the S3 server into one object.
Set this value to be less than or equal to that supported by the remote storage target.
Valid settings are between 5242880 (5MB) and 5368709120 (5GB).
This optional parameter defaults to 20MB if not explicitly defined.
Syntax
MaxUploadSize=<size_in_bytes>
Example
MaxUploadSize=10485760
MaxUploadThreadCount¶
Defines the maximum number of threads an ngmigrate command will use when uploading files to the S3 target. Each thread uploads an individual file part.
Defaults to the number of CPU cores in the system.
Adjusting this parameter where network bandwidth is limited, or multiple ngmigrate commands are executed in parallel, may be beneficial.
Syntax
MaxUploadThreadCount=<# of threads>
Example
MaxUploadThreadCount=8
MaxXAttrNativeMetadataSize¶
Specifies the total size in bytes allocated for storing extended attributes of a local file as metadata elements.
Ngenea HSM supports storing of extended attributes under the trusted. and user. namespaces.
The size of each metadata element for storing an extended attribute is calculated as the number of bytes of an encoded element key plus the number of bytes of an encoded element value.
Metadata element keys and values are encoded to preserve case and special characters. Keys have a constant prefix to distinguish metadata elements for storing extended attributes from other metadata elements.
If the size of all metadata elements exceeds the value of MaxXAttrNativeMetadataSize, ngmigrate repeatedly discards the largest extended attribute until the total size of remaining elements is less than or equal to the value of MaxXAttrNativeMetadataSize.
A warning is issued for every extended attribute that was not stored due to the discarding of the metadata element.
Valid settings are between 0 and 16384.
A current limitation on metadata size for S3 is 2048.
A recommended setting for MaxXAttrNativeMetadataSize is 1024 --- a half of that limitation.
The default parameter value is 0; that is, no limit will be applied.
Syntax
MaxXAttrNativeMetadataSize=<maximum xattr metadata size>
Example
MaxXAttrNativeMetadataSize=1024
Port¶
For Amazon AWS S3 targets, do not include this parameter.
For non-Amazon AWS S3 targets, this parameter defines the tcp port to connect to on the remote storage.
The default value is 443 if this parameter is unspecified.
Syntax
Port:<TCP port number>
Example
Port:443
RemoteLocationXAttr¶
Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the Master Configuration File.
Syntax
RemoteLocationXAttr=targetref:<LocalFileRegexReference>
Example
RemoteLocationXAttr=target1:$1
RequestTimeoutMs¶
Specifies the timeout (in milliseconds) for transferring data from or to an S3 storage endpoint.
If the speed of a data transfer via a socket connection is too slow during that time period, then the data transfer is aborted, and the socket connection is closed.
Default parameter value is 3000.
Syntax
RequestTimeoutMs=<timeout in milliseconds>
Example
RequestTimeoutMs=10000
RetrieveObjectBasePath¶
Defines a base path for downloading objects from the remote storage target. This base path is prepended to a path composed using the RetrieveObjectName parameter to obtain the full path of a remote object to retrieve. By default, this base path also acts as a base path for listing objects on the remote storage target by ngscan.
If a non-empty base path does not end with /, then / is implicitly appended to the base path.
The default value for this parameter is the empty string: do not prepend any string to a path composed using the RetrieveObjectName parameter and list objects on the remote storage target starting at its root.
The value of this parameter typically matches the value of StoreObjectBasePath parameter.
Example
RetrieveObjectBasePath=dir1/subdir1
RetrieveObjectName¶
Defines a mapping from the value of a remote location (APXrmtXX) DMAPI extended attribute of a local file to the name suffix of an associated object on the remote storage target. The value of RetrieveObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to retrieve (download).
In a value of this parameter, the reference $0 represents the entire value of a remote location (APXrmtXX) DMAPI extended attribute.
The references $1, $2, ..., $9 represent parts of that entire value matched to parts of a regular expression specified by RemoteLocationXAttrRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
The default value for this parameter is $0.
The value of this parameter typically matches the value of StoreObjectName parameter.
Example
RetrieveObjectName=$1
Scheme¶
Defines the transport protocol used to communicate with the S3 target.
ArcaStream / Pixit Media only supports HTTPS protocol for Ngenea HSM solutions.
Syntax
Scheme=<PROTOCOL>
Example
Scheme=HTTPS
SDKLogging¶
Specifies a log level for the AWS C++ SDK library. If a master configuration file defines multiple S3 storage endpoints, then a log level specified for the first accessed S3 storage endpoint will be in effect.
Log messages are prefixed "AWS: " and are sent to the same log target as other log messages.
Syntax
SDKLogging=<fatal|error|warn|info|debug|trace|false>
Boolean "True" is equivalent to "trace".
Example
SDKLogging=True
The default value is "False" which disables logging.
SequentialDeletes¶
Specifies the methodology for issuing S3 requests for remote object deletion.
If this parameter is False, always issue DeleteObjects (delete multiple objects) requests to delete objects.
If this parameter is True, always issue DeleteObject (delete a single object) requests to delete objects.
Deleting multiple objects incurs one DeleteObject request per object.
The default value is False.
Setting the parameter to True is necessary when using the GCS S3 XML API, as DeleteObjects requests to delete multiple objects using one API call are not supported.
Syntax
SequentialDeletes=<True|False>
Example
SequentialDeletes=True
ShadowFolderMetadataMode¶
Defines the naming scheme of shadow metadata objects for remote folders.
In the following description FOLDERNAME is the remote folder name.
The following values are permitted:
"native" --- use shadow metadata objects with names
FOLDERNAME/. Store/retrieve metadata as object content in JSON format."shadow" --- use shadow metadata objects with names
.FOLDERNAME.xattr. Store/retrieve metadata as object content in JSON format. For MinIO targets, "shadow" is recommended."metadata" --- use empty shadow metadata objects with names
FOLDERNAME/. Store/retrieve metadata as S3 metadata (passed in HTTP headers) for the objects. For Versity targets, "metadata" is recommended."auto" --- try using a shadow metadata object with the name
FOLDERNAME/, then with the name.FOLDERNAME.xattr. Store/retrieve metadata as object content in JSON format.
Shadow metadata objects are located at the same hierarchy level as remote folders.
The default value is "auto".
Syntax
ShadowFolderMetadataMode=<auto|native|shadow|metadata>
Example
ShadowFolderMetadataMode=shadow
ShadowFolderMetadataSave¶
Defines whether ngmigrate saves the ACLs and permissions (file mode, owner, and group) of local directories in shadow metadata objects for remote folders when migrating files (i.e. passing file names on the command line).
When migrating directories (i.e. passing directory names on the command line), ngmigrate always saves their ACLs (provided the parameter ACLSave is True) and permissions.
Shadow metadata objects are located at the same hierarchy level as remote folders and have the names FOLDERNAME/ or .FOLDERNAME.xattr, where FOLDERNAME is a remote folder name.
The parameter ShadowFolderMetadataMode controls the naming scheme for shadow metadata objects and the method of storing metadata within the objects.
Saved information can be restored by ngrecall when recreating local directories on reverse stubbing/premigration. Information about a local directory includes its file mode, owner, and group. If the parameter ACLSave is True, the information also includes the ACL of this local directory.
If the parameter ShadowFolderMetadataSave is False, ngmigrate does not save information about local directories, and ngrecall will recreate them with default file mode, owner, group, and ACL.
The default value is False.
Syntax
ShadowFolderMetadataSave=<True|False>
Example
ShadowFolderMetadataSave=True
SSLVerify¶
Controls the behaviour of TLS certificate-related checks.
If the value of this parameter is True, the checks are performed.
Default parameter value is True.
Syntax
SSLVerify=<True|False>
Example
SSLVerify=False
StorageClass¶
Defines a storage class for objects uploaded to the target bucket.
Valid storage classes include STANDARD, STANDARD_IA, REDUCED_REDUNDANCY, and GLACIER.
Example
StorageClass=GLACIER
StoreObjectBasePath¶
Defines a base path for uploading objects to the remote storage target. This base path is prepended to a path composed using the StoreObjectName parameter to obtain the full path of a remote object to store. Additionally, this base path can act as a base path for listing objects on the remote storage target by ngscan.
If a non-empty base path does not end with /, then / is implicitly appended to that path.
The default value for this parameter is the empty string: do not prepend any string to a path composed using the StoreObjectName parameter and list objects on the remote storage target starting at its root.
The value of this parameter typically matches the value of RetrieveObjectBasePath parameter.
Example
StoreObjectBasePath=dir1/subdir1
StoreObjectName¶
Defines a mapping from the name of a local file to the name suffix of an associated object on the remote storage target. The value of StoreObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to store (upload).
In a value of StoreObjectName parameter, the reference $0 represents the entire absolute name of a local file.
The references $1, $2, ..., $9 represent parts of that entire name matched to parts of a regular expression specified by LocalFileRegex for this storage target in the Master Configuration File.
The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.
The default value for this parameter is $0.
The value of this parameter typically matches the value of RetrieveObjectName parameter.
Example
StoreObjectName=$1
SymlinkMode¶
Defines whether Ngenea HSM stores the target of a symbolic link as data content within an object or within an object's metadata at the storage target.
If the parameter SymlinkMode is object, the local target of the symlink is stored as plaintext data within the named symlink object at the storage target.
If the parameter SymlinkMode is metadata, the local target of the symlink is stored within the apxsymln metadata key for the object at the storage target.
The default value is object.
Syntax
SymlinkMode=<object|metadata>
Example
SymlinkMode=metadata
Security¶
The permissions listed in the table below must be enabled to allow Ngenea HSM to correctly function. See Amazon S3 API Reference.
Action |
Description |
|---|---|
AbortMultipartUpload |
Abort a multipart upload |
CompleteMultipartUpload |
Complete a multipart upload |
CopyObject |
Creates a copy of an existing object |
CreateMultipartUpload |
Initiate a multipart upload |
DeleteObject |
Delete an object |
DeleteObjects |
Delete multiple objects from a bucket |
GetObject |
Retrieve an object |
HeadObject |
Retrieves metadata from an object |
ListBucket |
List objects in a bucket |
ListObjectsV2 |
List objects in a bucket |
PutObject |
Adds an object to a bucket |
UploadPart |
Uploads a part in a multipart upload |