Ngenea¶

Ngenea 2.6+ introduces changes to how Ngenea Targets are managed. Buckets [new feature] and Targets work together to provide easy, granular mapping between Spaces and Buckets.

Ngenea provides the capability to off-site data, import data, or send data between pixstor sites.

Concepts¶

A Bucket provides the storage for off-PixStor data. This may be an AWS S3, GCS, Azure Blob, POSIX or Blackpearl DS3 protocol. For versions prior to 2.6, these were previously the types of Ngenea Target.
A Target is the mapping for how data is transferred to and from a Bucket for a Space. For versions prior to 2.6 this was available as part of the Ngenea Target setup.

Viewing Buckets¶

A bucket is area of storage off-PixStor which comprises the following capabilities:

Type
location
Credentials
Bucket specific settings

Click the Ngenea button to navigate to the Ngenea screen

Clicking the Ngenea button in the main menu bar displays the list of Ngenea Buckets:

Viewing Targets¶

A Target is the mapping for how data is transferred to and from a Bucket for a Space.

Click the drop down button of a Bucket to view the associated Targets

Filtering Buckets¶

../_images/ngenea_targets_filter_bar.png

To filter the list of Ngenea Buckets, type the Bucket name or part of a Bucket name in the filter bar.

Bucket Wizard¶

Navigating the Wizard¶

Click the close button to exit the wizard. Changes are not saved.

Click the Next button to advance to the next page of the wizard. The Next button is disabled until all required page elements are completed.

Click the Go Back button to return to the previous wizard page.

Click the Finish & Create button to apply the changes displayed on the wizard summary page.

Adding a Bucket¶

Alternatively, a bucket for a Space can be created during the Space Creation Wizard. See Managing Spaces

Important

This function can only be performed by a Hub Administrator.

Click the Add Ngenea Bucket button to start the Ngenea Bucket Wizard

Type & reference¶

Ngenea supports the following storage target types:

Bucket Type	Description
S3	AWS S3 and S3 compatible buckets
Microsoft Azure	Azure Blob Storage
SpectraLogic BlackPearl	Spectralogic DS3 buckets
Google Object Storage	Google Cloud Storage
POSIX file system mount	Locally mounted targets, such as NFS

Select the required Bucket type from the drop down menu.

|brand_backend_name| Bucket wizard Type Reference

Enter the name of the Ngenea Bucket name.

E.G.:

mybucketname

Provide an optional friendly name as a Bucket label. If specified, users will see the Bucket label when referring to the Bucket in the Hub UI.

E.G.:

‘mybucketname data’

Depending on the Bucket type selected, a type-specific configuration page will be shown after selecting Next.

|brand_backend_name| Bucket wizard Could storage configuration

Type-specific config AWS S3 or compatible¶

|brand_backend_name| Bucket Wizard S3 Target Type

Enter the settings as required, which must match those set in the S3 object storage provider:

Setting	Description
Bucket	The name of the storage bucket as specified at the object storage service.
Access key ID	The unique access key for the AWS user account performing data transfers. This will be stored encrypted.
Secret access key	The unique security key for the AWS user account performing data transfers. This will be stored encrypted.
Region	The AWS (or S3 compliant provider) region hosting the S3 Cloud Storage
Storage server address	Not used for Amazon S3 Cloud Storage. For services which reside at specific IPs, such as AWS Snowball, MinIO or LocalStack, specific the host or IP address to connect to.
Storage server port	The TCP/IP port used to communicate
Scheme	HTTP or HTTPS transfer. HTTPS is recommended. Data integrity cannot be guaranteed over HTTP transfer schemes.
Verify SSL	Whether to verify the SSL connection of the target. Disabling the SSL verification allows connections to storage targets which do not provide valid SSL certificates. Connecting to invalid SSL certificates is insecure.

Type-specific config Microsoft Azure¶

|brand_backend_name| Bucket Wizard Azure Target Type

Enter the settings as required, which must match those set in the Azure object storage provider:

Setting	Description
Access key ID	The unique access key for the Azure user account performing data transfers. This will be stored encrypted.
Scheme	HTTP or HTTPS transfer. HTTPS is recommended. Data integrity cannot be guaranteed over HTTP transfer schemes.
Container	The storage container for the blob data.
Storage account.	The Azure namespace containing the Container

Type-specific config Spectra Logic BlackPearl¶

|brand_backend_name| Bucker Wizard BlackPearl Target Type

Enter the settings as required, which must match those set in the BlackPearl DS3 object storage provider:

Setting	Description
Bucket	The name of the storage bucket as specified at the object storage service.
Access key ID	The unique access key for the BlackPearl user account performing data transfers. This will be stored encrypted.
Secret access key	The unique security key for the BlackPearl user account performing data transfers. This will be stored encrypted.
Storage server address	The FQDN hostname of the BlackPearl.

Type-specific config Google Object Storage¶

|brand_backend_name| Bucket Wizard Google Cloud Storage Target Type

Enter the settings as required, which must match those set in the Google Cloud Storage object storage provider:

Setting	Description
Bucket	The name of the storage bucket as specified at the object storage service.
Google authentication JSON	Enter the contents of the JSON key for the user or service account granted permission to transfer data to Cloud Storage bucket. This will be stored encrypted. For more information refer to Google documentation

Important

If the Ngenea Bucket is to be used as a backup only bucket, do not define JSON information in the Google Authentication JSON. Instead define keyword CredentialsFile with the path to the file containing JSON credentials on pixstor. For more information refer to Backup-Only Targets.

Type-specific config Filesystem Mount¶

|brand_backend_name| Bucket Wizard File System Target Type

Enter the settings as required, which must match the location of the NAS mount point on the pixstor:

Important

Bucket mounts points are site-specific however Hub will create the target across sites associated with Space.

Setting	Description
Target mount point	The location of the POSIX compliant mounted storage. E.G. If the external POSIX storage is mounted on `/mmfs1/myexternalnas` enter `/mmfs1/myexternalnas`

Advanced configuration¶

To add configuration settings to a target to control specific behaviour during Ngenea data operations.

Click the Add new key to define a new configuration setting

../_images/ngenea_target_advanced_keyword_edit.png

Enter the name of the configuration setting in the Keyword field

../_images/ngenea_target_advanced_key_type_dropdown.png

Select the type of configuration setting from the drop down menu. Choose or enter the value for the configuration setting.

Example of added Advanced configuration settings:

|brand_backend_name| Target Wizard Advanced Configuration Completed Example

Summary¶

Upon completing the wizard steps a summary is presented:

Click the Finish & Create button to apply the changes displayed on the wizard summary page.

Alternatively Go back and change the proposed configuration as required or close the wizard to cancel the creation of the Ngenea target.

Editing a Bucket¶

Important

This function can only be performed by a Hub Administrator.

Clicking the Ngenea button in the main menu bar displays the list of Ngenea Buckets:

Click the edit icon on the required Ngenea Bucket row to edit the Ngenea Bucket

Modify the Ngenea Bucket settings as required. Refer to Adding a Bucket for settings guidance.

|brand_backend_name| Buckets Edit Dialog

Deleting a Bucket¶

Important

This function can only be performed by a Hub Administrator.

Clicking the Ngenea button in the main menu bar displays the list of Ngenea Buckets:

Click the delete icon on the required Ngenea bucket row to delete the Ngenea Bucket

A confirmation dialog is raised:

|brand_backend_name| Bucket Delete Confirmation Modal

Type the name of the Ngenea Bucket
Click Yes to delete the Ngenea Bucket. This action is irreversable.
Alternately click no, or close the confirmation dialog.

Important

Deleting a space automatically deletes associated targets. The parent bucket of the targets is not deleted, the bucket configuration still exists in pixstor and requires manual removal at the pixstor CLI.

Browsing a Target¶

Important

This function can only be performed by a Hub Administrator.

Clicking the Ngenea button in the main menu bar displays the list of Ngenea targets:

Click the drop down button of a Bucket to view the associated Targets

Click the browse icon on the Ngenea target row to browse the Ngenea target

For more information on browsing and importing from an Ngenea target, see Target Import

Adding a Target¶

A Target links a Space to a Bucket. A Target that is linked in this way must:

have a Regex filter that matches the Space mountpoint which can also include subdirectories with the Space mountpoint.
be linked to the same sites as the Space.

Click the Ngenea button to navigate to the Ngenea screen

Clicking the Ngenea button in the main menu bar displays the list of Ngenea Buckets:

Click the Add Target button to start the External Target Wizard

External Target Wizard¶

Navigating the Wizard¶

Click the close button to exit the wizard. Changes are not saved.

Click the Next button to advance to the next page of the wizard. The Next button is disabled until all required page elements are completed.

Click the Go Back button to return to the previous wizard page.

Click the Finish & Create button to apply the changes displayed on the wizard summary page.

Name & Location¶

Upon starting the External target wizard, the user is presented with the Name & Location page.

|brand_backend_name| Bucket wizard name and location settings

Configure the migration settings as required to handle data migration accordingly:

Setting	Description
Bucket	The name of the Bucket the Target will be associated with. The bucket name cannot be changed.
Target Space	The name of the Space the Target will be associated with. Selecting the Space generates a suggested File match entry. If no space is selected, you must manually select which sites it is available on.
File match	Refer to the RegEx Filters example table below
Ngenea Target reference	The metadata key to add to all files processed by Ngenea HSM.
Available on sites	The sites the Space is associated with and therefore the Target will be created are listed. The sites cannot be changed.

Note

When pre-creating a target not associated to a Space where the Target Space is set to ‘No space is selected’, Hub will not suggest automatic settings for the dialog. In this case it required to manually set and review the dialog setting so that the wizard can proceed further.

Example regex filters where a space named myspace is present on the pixstor filesystem at location /mmfs1/data/myspace:

RegEx Filter Examples	Outcome
`/mmfs1/data/myspace/(.*)`	Dehydrated files within the myspace folder are present at the root of the storage target.
`/mmfs1/data/(myspace/.*)`	The myspace folder is present at the root of the storage target.
`/mmfs1/data/myspace/subdirectory/(.*)`	Dehydrated files within the subdirectory of the myspace directory are present at the root of the storage target. The myspace directory is not present. Data immediately within the myspace directory (other than that within the subdirectory) is not eligible for Ngenea operations.
`/mmfs1/data/myspace/(subdirectory/.*)`	The subdirectory of the myspace directory is present at the root of the storage target. The myspace directory is not present. Data immediately within the myspace directory (other than that within the subdirectory) is not eligible for Ngenea operations.
`/mmfs1/data/(myspace/.+)`	This regex matches paths under /mmfs1/data/ that contain myspace/ followed by one or more characters. The myspace folder is present at the root of the storage target.
`/mmfs1/(data/myspace/.+)`	This regex matches paths containing data/myspace/ after /mmfs1/. The data folder is present at the root of the storage target.
`/(mmfs1/data/myspace/.+)`	This regex matches paths under mmfs1/data/myspace/.
`/mmfs1/data/((myspace1\|myspace2\|myspace3)/.+)`	This regex matches paths under /mmfs1/data/ where the space is one of myspace1, myspace2, or myspace3.
`/mmfs1/(data/(myspace1\|myspace2\|myspace3)/.+)`	This regex matches paths under /mmfs1/ where the path contains data/ followed by one of myspace1, myspace2, or myspace3, and then additional path segments.
`/(mmfs1/data/(myspace1\|myspace2\|myspace3)/.+)`	This regex is similar to the previous one but matches paths that start directly with mmfs1/data/ and then one of the three space names, followed by additional segments.

Advanced configuration¶

Configure the migration settings as required to handle data migration accordingly:

Setting	Description
Delete on recall	Determines whether to delete the recalled data from the Bucket after the data has been successfully recalled.
Key setup	Add configuration settings to a target to control specific behaviour during Ngenea data operations.

Important

Advanced Keys for a Target override those defined for the associated Bucket.

Click the Add new key to define a new configuration setting

Enter the name of the configuration setting in the Keyword field

Select the type of configuration setting from the drop down menu. Choose or enter the value for the configuration setting.

Summary¶

Upon completing the wizard steps a summary is presented:

Click the Finish & Create button to apply the changes displayed on the wizard summary page.

Alternatively Go back and change the proposed configuration as required or close the wizard to cancel the creation of the Ngenea target.