7.1. Default Workflows

This section documents all the workflows which come installed in Ngenea Hub by default. See Custom Workflows for guidance on how to create your own workflows.

7.1.1. migrate

Migrate one or more files from a site.

discovery: recursive

steps:

• dynamo.tasks.migrate

fields:

• lock_level (choice): Filesystem locking level for ngenea operations. One of: partial, implicit, none. Default=partial

7.1.2. premigrate

Premigrate one or more files from a site.

discovery: recursive

steps:

• dynamo.tasks.migrate

  • premigrate: True

fields:

• lock_level (choice): Filesystem locking level for ngenea operations. One of: partial, implicit, none. Default=implicit

7.1.3. recall

Recall one or more files on to a site.

discovery: recursive

steps:

• dynamo.tasks.recall

fields:

• lock_level (choice): Filesystem locking level for ngenea operations. One of: partial, implicit, none. Default=implicit

7.1.4. send

Send files from one site to another via cloud storage.

discovery: recursive

steps:

• dynamo.tasks.remove_location_xattrs_for_moved

• dynamo.tasks.migrate

  • premigrate: True

• dynamo.tasks.reverse_stub

fields:

• destinationsite (string): site to send files to

• destinationqueue (string): queue on the destination site to run the task on

• hydrate (bool): hydrate files on the destination site

7.1.5. site_sync

Sync a fileset from one site to another via cloud storage.

The snapdiff discovery looks for changes within the fileset on the source site since the last time the workflow was invoked. These changes are then synced to the destination site.

The workflow should be invoked with a single path which is the link point of the independent fileset to be synced.

discovery: snapdiff

fields:

• destinationsite (string): site to sync changes to

• destinationqueue (string): queue on the destination site to run the task on

• sync_preference (string): determines how conflicts should be resolved on the remote site. One of: newest, local, ignore

7.1.5.1. created

Files with state created are sent to the destination site, subject to sync_preference

steps:

• dynamo.tasks.remove_location_xattrs_for_moved

• dynamo.tasks.migrate

  • premigrate: True

  • overwrite: True

  • abort_missing: True

• dynamo.tasks.reverse_stub

  • overwrite: True

7.1.5.2. updated

Files with state updated are sent to the destination site, subject to sync_preference

steps:

• dynamo.tasks.check_sync_state

• dynamo.tasks.remove_location_xattrs_for_moved

• dynamo.tasks.migrate

  • premigrate: True

  • overwrite: True

  • abort_missing: True

• dynamo.tasks.reverse_stub

  • overwrite: True

7.1.5.3. moved

Files with state moved are moved 'in-place' on the destination site

steps:

• dynamo.tasks.move_paths_on_gpfs

7.1.5.4. deleted

Files with state deleted are removed on the destination site

steps:

• dynamo.tasks.delete_paths_from_gpfs

7.1.5.5. import_to_site_hydrated

This workflow is designed to import fully hydrated remote objects from external ngenea targets to the local filesystem.

discovery: null

steps:

• dynamo.tasks.import_files_from_external_target

  • hydrate: True

fields:

• endpoint (string): Name of the ngenea target configuration

• location (string): An absolute path for where to place the file/folder in the local filesystem, this can be a different path than the remote location. Defaults to None

• lock_level (string): Filesystem locking level for ngenea operations. One of: partial, implicit, none. Default=implicit

7.1.5.6. import_to_site_dehydrated

This workflow is designed to import stubbed remote objects from external ngenea targets to the local filesystem.

discovery: null

steps:

• dynamo.tasks.import_files_from_external_target

  • hydrate: False

fields:

• endpoint (string): Name of the ngenea target configuration

• location (string): An absolute path for where to place the file/folder in the local filesystem, this can be a different path than the remote location. Defaults to None

• lock_level (string): Filesystem locking level for ngenea operations. One of: partial, implicit, none. Default=implicit

7.2. Hidden Workflows

This section documents all the workflows which come installed in Ngenea Hub by default that are not provided in the UI. Some of these are intended for advanced users making use of the API, and some are used for GPFS policy automation.

7.2.1. transparent_recall

This is the workflow that will be performed on dehydrated files on the system when accessed, if the policy is configured to make use of Ngenea Hub within its Transparent Recall policy.

Typically, all of these Transparent Recalls are placed into a single job within Ngenea Hub.

steps:

• dynamo.tasks.recall

  • lock_level: "partial"

7.2.2. reverse_stub

This workflow is designed to create stubs files or fully hydrated files using remote content on a given site without the need for an existing file on the site for a given path. Due to this requiring the data to be within the cloud and not present on the filesystem, it requires the paths to target cloud targets through the API.

steps:

• dynamo.tasks.reverse_stub

fields:

• hydrate (bool): If the file should be fully hydrated during the recall operation

• overwrite (bool): If the operation should overwrite a local file that already exists

7.2.3. delete_file

This workflow deletes one or more file/folder paths from the filesystem.

steps:

• dynamo.tasks.delete_paths_from_gpfs

fields:

• recursive (bool): If non-empty directories should be deleted. if recursive is false additional jobs will be needed to delete empty directories. Defaults to false