Google Cloud Storage Type¶

The following section describes how to setup a Google Cloud Storage target.

Google Cloud Storage targets require several key pieces of information. The following example describes the information required to be present within the Storage Target Configuration File to enable migration to the specified account and bucket.

Example Configuration File¶

[General]
CredentialsFile=MyCredentialsFile
( or CredentialsJSON="multi-line JSON" )
Bucket=MyBucketName
RemoteLocationXAttr=target1:$1
RetrieveObjectName=$1
StoreObjectName=$1
DeleteOnRecall=True

Configuration Keywords¶

Required keywords

CredentialsFile / CredentialsJSON

Bucket

Optional keywords

Keyword	Default Value
ACLSave	False
DeleteOnRecall	False
Endpoint	Null
EscapeNames	True
MaxCopyThreadCount	MaxUploadThreadCount
MaxDownloadSize	20 MB
MaxDownloadThreadCount	Number of cores
MaxMetadataReadThreadCount	MaxDownloadThreadCount
MaxUploadSize	20 MB
MaxUploadThreadCount	Number of cores
MaxXAttrNativeMetadataSize	0
OAuth2Endpoint	Null
PercentDecodeXAttrNames	False
RemoteLocationXAttr	"$0"
RequestTimeoutMs	300000
RetrieveObjectBasePath	""
RetrieveObjectName	"$0"
ShadowFolderMetadataSave	False
StorageClass	Null
StoreObjectBasePath	""
StoreObjectName	"$0"

CredentialsFile / CredentialsJSON¶

Defines the Google Cloud Storage authentication credentials. CredentialsFile specifies the path to a file containing the credentials (a JSON file). CredentialsJSON specifies the credentials as a multi-line JSON string.

If neither of these keywords is specified, the Ngenea HSM applications use the file specified in the environment variable GOOGLE_APPLICATION_CREDENTIALS.

For further information regarding Google Cloud Storage accounts please refer to the Google Cloud Storage Documentation.

Syntax

CredentialsFile=<path to credentials file>

or

CredentialsJSON=<multi-line JSON credentials>

Example:

CredentialsFile=/user/cred.json

or

CredentialsJSON={
    "type": "service_account",
    ...
    }

Bucket¶

Defines the 'bucket' into which files are migrated. This bucket must exist on the Google Cloud Storage target prior to migrating data using Ngenea HSM.

For further information regarding Google Cloud Storage Buckets and Blobs please refer to the Google Cloud Storage Documentation.

Syntax

Bucket=<Name of the Google Cloud Storage bucket>

Example

Bucket=MyBucketName

ACLSave¶

Defines whether ngmigrate saves the ACLs of local files in the metadata of remote objects and whether the ACLs of local directories are saved in shadow metadata remote objects (provided that the parameter ShadowFolderMetadataSave is True).

If this parameter is False, ngrecall does not attempt to restore the ACLs of local files and created local directories on reverse stubbing/premigration.

The default value is False.

Syntax

ACLSave=<True|False>

Example

ACLSave=True

DeleteOnRecall¶

Defines whether to delete the data from the remote Google Cloud Storage target when successfully recalled back to the file system.

Setting this to False will result in files entering a "premigrated" state on successful recall.

Syntax

DeleteOnRecall=<True|False>

Example

DeleteOnRecall=True

Endpoint¶

This parameter defines the Hostname to be used by Ngenea HSM to access on-premises Google Storage.

For access to Google's public cloud storage, do not include this parameter.

Syntax

Endpoint=<Hostname>

Example

Endpoint=storage-vialink1.p.googleapis.com

EscapeNames¶

Defines whether to percent-encode the names of files or directories containing special characters (E.G. spaces / extended unicode character sets) in the names of remote objects when migrating or recalling.

If this parameter is False the names of remote objects are equal to the names of corresponding local files and directories upon successful migration.

Not all storage endpoints support storing of objects without percent-encoding enabled. In such cases this paramater should be set to True.

Where storage endpoints do support non-percent-encoded object names this parameter can enable remote processing workflows whereby applications can directly read data held in storage endpoints via the object URI or direct mount without path translation. Consider setting the parameter ShadowFolderMetadataSave along with EscapeNames to False in this case.

The default value is True.

Syntax

EscapeNames=<True|False>

Example

EscapeNames=False

MaxCopyThreadCount¶

Defines the maximum number of threads an ngcopy command will use when copying or moving objects to the Google Cloud Storage target. Each thread copies an individual object. Moving an object from a source location to a target location is implemented as copying the object to the target location with subsequent deleting the object at the source location.

The default value of this parameter is equal to the value of MaxUploadThreadCount parameter.

As copying an object does not require transferring its content to the Google Cloud Storage target, specifying a value of this parameter greater than the value of MaxUploadThreadCount may be beneficial.

Syntax

MaxCopyThreadCount=<# of threads>

Example

MaxCopyThreadCount=32

MaxDownloadSize¶

As for the uploads, downloads also use a multi-part transfer process. Files larger than 5MiB are downloaded in parts. This will split files into multiple parts and transfer each part separately.

This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate transactions equal to this size (except the final part, which may be smaller).

Set this value to be less than or equal to that supported by the remote storage target.

Valid settings are between 5242880 (5MiB) and 5368709120 (5GiB).

This optional parameter defaults to defaults to 20MiB if not explicitly defined.

Syntax

MaxDownloadSize=<size_in_bytes>

Example

MaxDownloadSize=134217728

MaxDownloadThreadCount¶

Defines the maximum number of threads a ngrecall command will use when downloading files from the Google Cloud Storage target. The threads download multiple files in parallel and download multiple parts of an individual file in parallel.

Defaults to the number of CPU cores in the system.

Adjusting this parameter where network bandwidth is limited, or multiple ngrecall commands are executed in parallel, may be beneficial.

Syntax

MaxDownloadThreadCount=<# of threads>

Example

MaxDownloadThreadCount=8

MaxMetadataReadThreadCount¶

Defines the maximum number of threads an ngmigrate, ngrecall, ngscan, or ngcopy command will use to fetch remote object metadata. Each thread issues a single request for fetching object metadata or downloads a single shadow metadata object for a folder.

The default value of this parameter is equal to the value of MaxDownloadThreadCount parameter.

As object metadata fetching requests do not transfer object content, specifying a value of this parameter greater than the value of MaxDownloadThreadCount may be beneficial.

Syntax

MaxMetadataReadThreadCount=<# of threads>

Example

MaxMetadataReadThreadCount=32

MaxUploadSize¶

Files larger than 5MiB are uploaded using the multi-part transfer process. This will split files into multiple parts and transfer each part separately, with the parts reconstructed on the server.

This parameter sets the maximum size (in bytes) of an individual data part transfer. If a file exceeds this size, it is transferred in multiple separate transactions equal to this size (except the final part, which may be smaller), and reconstructed on the Google Cloud Storage server into one object.

Set this value to be less than or equal to that supported by the remote storage target.

Valid settings are between 5242880 (5MiB) and 5368709120 (5GiB).

This optional parameter defaults to 20MiB if not explicitly defined.

Syntax

MaxUploadSize=<size_in_bytes>

Example

MaxUploadSize=134217728

MaxUploadThreadCount¶

Defines the maximum number of threads an ngmigrate command will use when uploading files to the Google Cloud Storage target. The threads upload multiple files in parallel and upload multiple parts of an individual file in parallel.

Defaults to the number of CPU cores in the system.

Adjusting this parameter where network bandwidth is limited, or multiple ngmigrate commands are executed in parallel, may be beneficial.

Syntax

MaxUploadThreadCount=<# of threads>

Example

MaxUploadThreadCount=8

MaxXAttrNativeMetadataSize¶

Specifies the total size in bytes allocated for storing extended attributes of a local file as metadata elements. Ngenea HSM supports storing of extended attributes under the trusted. and user. namespaces.

The size of each metadata element for storing an extended attribute is calculated as the number of bytes of an encoded element key plus the number of bytes of an encoded element value.

Metadata element keys and values are encoded to preserve case and special characters. Keys have a constant prefix to distinguish metadata elements for storing extended attributes from other metadata elements.

If the size of all metadata elements exceeds the value of MaxXAttrNativeMetadataSize, ngmigrate repeatedly discards the largest extended attribute until the total size of remaining elements is less than or equal to the value of MaxXAttrNativeMetadataSize.

A warning is issued for every extended attribute that was not stored due to the discarding of the metadata element.

Valid settings are between 0 and 16384. A current limitation on metadata size for Google Cloud Storage is 8192. A recommended setting for MaxXAttrNativeMetadataSize is 4096 --- a half of that limitation.

The default parameter value is 0; that is, no limit will be applied.

Syntax

MaxXAttrNativeMetadataSize=<maximum xattr metadata size>

Example

MaxXAttrNativeMetadataSize=1024

OAuth2Endpoint¶

This parameter defines the Hostname to be used by Ngenea HSM to perform OAuth2 authentication.

The Endpoint parameter must also be included.

Syntax

Endpoint=<Hostname>
OAuth2Endpoint=<OAuth-Hostname>

Example

Endpoint=storage-vialink1.p.googleapis.com
OAuth2Endpoint=oauth2-vialink1.p.googleapis.com

PercentDecodeXAttrNames¶

Defines whether ngrecall percent-decodes names of native extended attributes fetched from remote object metadata. If this parameter is True, the names of native extended attributes are percent-decoded. If this parameter is False, the names of native extended attributes are used as is.

Enabling PercentDecodeXAttrNames=True is required for compatibility where data migration has been undertaken using GCS S3 XML API and data recall occurs via the GCS JSON protocol. Saving file extended attributes in remote object metadata via the GCS S3 XML API requires percent-encoding attribute names which are otherwise not decoded automatically with GCS JSON protocol, as Ngenea HSM cannot automatically determine if percent symbols are literal or used for percent-encoding of stored data.

The default value is False.

Syntax

PercentDecodeXAttrNames=<True|False>

Example

PercentDecodeXAttrNames=True

RemoteLocationXAttr¶

Defines the APXrmtlc Extended Attribute value prepended to the filename on migration, which is the key used to recall the file. $1 represents the use of the filename and path matching the ( ) section of the LocalFileRegex setting for this storage target in the Master Configuration File.

Syntax

RemoteLocationXAttr=targetref:<LocalFileRegexReference>

Example

RemoteLocationXAttr=target1:$1

RequestTimeoutMs¶

Specifies the timeout (in milliseconds) for transferring data from or to a Google storage endpoint.

If the speed of a data transfer via a socket connection is too slow during that time period, then the data transfer is aborted, and the socket connection is closed.

Default parameter value is 300000.

Syntax

RequestTimeoutMs=<timeout in milliseconds>

Example

RequestTimeoutMs=60000

RetrieveObjectBasePath¶

Defines a base path for downloading objects from the remote storage target. This base path is prepended to a path composed using the RetrieveObjectName parameter to obtain the full path of a remote object to retrieve. By default, this base path also acts as a base path for listing objects on the remote storage target by ngscan.

If a non-empty base path does not end with /, then / is implicitly appended to the base path.

The default value for this parameter is the empty string: do not prepend any string to a path composed using the RetrieveObjectName parameter and list objects on the remote storage target starting at its root.

The value of this parameter typically matches the value of StoreObjectBasePath parameter.

Example

RetrieveObjectBasePath=dir1/subdir1

RetrieveObjectName¶

Defines a mapping from the value of a remote location (APXrmtXX) DMAPI extended attribute of a local file to the name suffix of an associated object on the remote storage target. The value of RetrieveObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to retrieve (download).

In a value of this parameter, the reference $0 represents the entire value of a remote location (APXrmtXX) DMAPI extended attribute. The references $1, $2, ..., $9 represent parts of that entire value matched to parts of a regular expression specified by RemoteLocationXAttrRegex for this storage target in the Master Configuration File. The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.

The default value for this parameter is $0.

The value of this parameter typically matches the value of StoreObjectName parameter.

Example

RetrieveObjectName=$1

ShadowFolderMetadataSave¶

Defines whether ngmigrate saves the ACLs and permissions (file mode, owner, and group) of local directories in shadow metadata objects for remote folders when migrating files (i.e. passing file names on the command line). When migrating directories (i.e. passing directory names on the command line), ngmigrate always saves their ACLs and permissions. Shadow metadata objects have the names FOLDERNAME/, where FOLDERNAME is a remote folder name.

Saved information can be restored by ngrecall when recreating local directories on reverse stubbing/premigration. Information about a local directory includes its file mode, owner, and group. If the parameter ACLSave is True, the information also includes the ACL of this local directory.

If the parameter ShadowFolderMetadataSave is False, ngmigrate does not save information about local directories when migrating files (but not directories), and ngrecall will recreate the local directories with ACLs and permissions (file mode, owner, and group) as determined by the local file system environment unless specifically overridden with the --default-<option> arguments of ngrecall.

The default value is False.

Syntax

ShadowFolderMetadataSave=<True|False>

Example

ShadowFolderMetadataSave=True

StorageClass¶

Defines a storage class for objects uploaded to the target bucket.

Valid storage classes include STANDARD, NEARLINE, COLDLINE, and ARCHIVE.

Example

StorageClass=ARCHIVE

StoreObjectBasePath¶

Defines a base path for uploading objects to the remote storage target. This base path is prepended to a path composed using the StoreObjectName parameter to obtain the full path of a remote object to store. Additionally, this base path can act as a base path for listing objects on the remote storage target by ngscan.

If a non-empty base path does not end with /, then / is implicitly appended to that path.

The default value for this parameter is the empty string: do not prepend any string to a path composed using the StoreObjectName parameter and list objects on the remote storage target starting at its root.

The value of this parameter typically matches the value of RetrieveObjectBasePath parameter.

Example

StoreObjectBasePath=dir1/subdir1

StoreObjectName¶

Defines a mapping from the name of a local file to the name suffix of an associated object on the remote storage target. The value of StoreObjectBasePath parameter is prepended to that suffix to obtain the full path of a remote object to store (upload).

In a value of StoreObjectName parameter, the reference $0 represents the entire absolute name of a local file. The references $1, $2, ..., $9 represent parts of that entire name matched to parts of a regular expression specified by LocalFileRegex for this storage target in the Master Configuration File. The brackets ( and ) in the regular expression specify those parts: the reference $1 corresponds to a pair of brackets with the leftmost (, the reference $2 corresponds to a pair of brackets with the subsequent (, and so on.

The default value for this parameter is $0.

The value of this parameter typically matches the value of RetrieveObjectName parameter.

Example

StoreObjectName=$1

Security¶

The permissions listed in this table must be enabled to allow Ngenea HSM to correctly function.

See Google Cloud Storage - APIs & reference (IAM permissions for Cloud Storage).

Object Permission Name	Description
storage.buckets.get	Read bucket metadata, excluding IAM policies.
storage.objects.create	Add new objects to a bucket.
storage.objects.delete	Delete objects.
storage.objects.get	Read object data and metadata, excluding ACLs.
storage.objects.list	List objects in a bucket. Also read object metadata,
	excluding ACLs, when listing.
storage.objects.update	Update object metadata, excluding ACLs.