Ngenea Hub

Ngenea Hub harnesses the power of Ngenea to provide global workflows, enabling your data to be where you need it, when you need it.

• 1. Installation
• 2. Upgrade
  • 2.1. Before you start
  • 2.2. Backup Workflows
  • 2.3. Stop Services
  • 2.4. Upgrade Packages
  • 2.5. Upgrade Configurations
  • 2.6. Restart Services
  • 2.7. Validation
  • 2.8. Restoring Workflows
• 3. Configuration
  • 3.1. Hub Initial Configuration
  • 3.2. Hub Configuration
    • 3.2.1. Settings
      • 3.2.1.1. Mandatory Settings
      • 3.2.1.2. Optional settings
    • 3.2.2. Server Configurations
    • 3.2.3. Docker Compose configuration
  • 3.3. Worker Configuration
    • 3.3.1. Passing Custom Config File to Ngenea Worker
      • 3.3.1.1. To Run ngenea-worker with multiple sites
    • 3.3.2. Setting Ngenea Worker debug level in systemd service script
• 4. Feature Set-up
  • 4.1. Search
    • 4.1.1. Prerequisites
    • 4.1.2. Configuration
      • 4.1.2.1. Search Backend
      • 4.1.2.2. Elasticsearch URL
    • 4.1.3. Search UI
  • 4.2. Cloud Functions
    • 4.2.1. GCP Cloud Function
    • 4.2.2. AWS Lambda Function
      • 4.2.2.1. Create the IAM policy and role
      • 4.2.2.2. Create the Lambda function
    • 4.2.3. Configuration
      • 4.2.3.1. version
      • 4.2.3.2. hub_access
      • 4.2.3.3. sites
      • 4.2.3.4. actions
      • 4.2.3.5. optional
      • 4.2.3.6. vendor
      • 4.2.3.7. vendors
      • 4.2.3.8. Complete Example
  • 4.3. Ngenea Worker Plugins
    • 4.3.1. Creating your plugin
      • 4.3.1.1. Example plugin creation
      • 4.3.1.2. External Dependencies
      • 4.3.1.3. Installing custom plugins
    • 4.3.2. Managing plugins
      • 4.3.2.1. Installing your plugin
      • 4.3.2.2. Uninstalling your plugin
    • 4.3.3. Custom Tasks
      • 4.3.3.1. The Structure of a Task
        • 4.3.3.1.1. Arguments
        • 4.3.3.1.2. Return Payload
      • 4.3.3.2. Support for Streaming Paths from API
    • 4.3.4. Example Plugins
      • 4.3.4.1. Email notification task
      • 4.3.4.2. Running additional script
  • 4.4. Disaster Recovery / Cold Failover
    • 4.4.1. Setup
      • 4.4.1.1. Configure datastore
      • 4.4.1.2. Configure Networking
      • 4.4.1.3. Install Ngenea Hub
    • 4.4.2. Performing failover
    • 4.4.3. Migration from local datastore
  • 4.5. External SSL
    • 4.5.1. Installing NGINX
    • 4.5.2. Configuration for SSL Termination and Reverse Proxy using OpenSSL
      • 4.5.2.1. Create Self-Signed Certificates for Nginx
      • 4.5.2.2. Creating configuration file for Nginx
    • 4.5.3. Configuration for SSL Termination and Reverse Proxy using Certbot and LetsEncrypt (Another method)
      • 4.5.3.1. Add trusted SSL Certificates from Letsencrypt
    • 4.5.4. HTTPS Configuration for NgeneaHub
      • 4.5.4.1. Reload the Nginx service after you have made some configuration changes
      • 4.5.4.2. To auto-renew the certificates, Run
• 5. Usage
  • 5.1. Web Interface
    • 5.1.1. Authentication
      • 5.1.1.1. Login
      • 5.1.1.2. Logout
    • 5.1.2. Browser
      • 5.1.2.1. Recent Jobs
      • 5.1.2.2. Search
        • 5.1.2.2.1. Managing the search filters from the dialog
        • 5.1.2.2.2. Submitting the search and viewing search results
        • 5.1.2.2.3. Advanced usage: Managing the search filters from the search bar
      • 5.1.2.3. Browse Directories
        • 5.1.2.3.1. Migrate
        • 5.1.2.3.2. Premigrate
        • 5.1.2.3.3. Recall
        • 5.1.2.3.4. Send
    • 5.1.3. Jobs
      • 5.1.3.1. Pagination
      • 5.1.3.2. Apply Filter
      • 5.1.3.3. Job ID
      • 5.1.3.4. Owner
    • 5.1.4. Administration
      • 5.1.4.1. Sites
        • 5.1.4.1.1. Add Site
        • 5.1.4.1.2. Delete Site
      • 5.1.4.2. Users
        • 5.1.4.2.1. Add User
        • 5.1.4.2.2. Activate User
        • 5.1.4.2.3. Deactivate User
      • 5.1.4.3. File Status Types
  • 5.2. API
    • 5.2.1. Reference
    • 5.2.2. Authentication
      • 5.2.2.1. JWT Authentication
      • 5.2.2.2. Client Keys
        • 5.2.2.2.1. Creating Client Keys
        • 5.2.2.2.2. Using Client Keys
    • 5.2.3. Submitting Workflow
      • 5.2.3.1. Migrate
    • 5.2.4. Trailing Slashes
  • 5.3. Monitoring and Management
    • 5.3.1. systemd service
    • 5.3.2. ngeneahubctl cli tool
    • 5.3.3. Docker containers
    • 5.3.4. Health endpoints
      • 5.3.4.1. }
      • 5.3.4.2. }
  • 5.4. Custom Workflows
    • 5.4.1. Defining workflows
      • 5.4.1.1. Filter Rules
      • 5.4.1.2. Runtime fields
    • 5.4.2. Running Workflows
      • 5.4.2.1. Discovery Steps
        • 5.4.2.1.1. Discovery Options
    • 5.4.3. Includes and Excludes
  • 5.5. Jobs
    • 5.5.1. Housekeeping
  • 5.6. Site Sync
    • 5.6.1. Schedule
    • 5.6.2. Workflow
    • 5.6.3. Additional Options
    • 5.6.4. Troubleshooting
  • 5.7. Bidirectional Site Sync
    • 5.7.1. Schedule
    • 5.7.2. Workflow
    • 5.7.3. Troubleshooting
  • 5.5. Jobs
    • 5.5.1. Housekeeping
  • 5.8. Search
    • 5.8.1. Submitting a query
      • 5.8.1.1. Filters
    • 5.8.2. Retrieving results
      • 5.8.2.1. Parameters
      • 5.8.2.2. Merged results
    • 5.8.3. Max Results
    • 5.8.4. Housekeeping
  • 5.9. Controlling Bandwidth Usage
    • 5.9.1. Enabling the bandwidth feature in the UI
    • 5.9.2. Checking Node status
    • 5.9.3. Registering Datastores
      • 5.9.3.1. Cloud IPAddresses
      • 5.9.3.2. Using manual IP addresses
    • 5.9.4. Linking a Datastore to a Site
    • 5.9.5. Applying the bandwidth
  • 5.10. Configuration
    • 5.10.1. Global Configurations
      • 5.10.1.1. Available Settings
      • 5.10.1.2. REST API
      • 5.10.1.3. Settings Migration
    • 5.10.2. Site-specific Configurations
      • 5.10.2.1. Available Settings
        • 5.10.2.1.1. File Batching
      • 5.10.2.2. REST API
  • 5.11. Feature Flags
    • 5.11.1. Available Features
    • 5.11.2. REST API
    • 5.11.3. ngclient
  • 5.12. Actions
    • 5.12.1. Available Actions
      • 5.12.1.1. discover_nodes
    • 5.12.2. REST API
  • 5.13. Limitations
    • 5.13.1. Site Sync
      • 5.13.1.1. Site Sync must only be used in one direction
      • 5.13.1.2. Site Sync only considers changes within an applicable time window for synchronisation
      • 5.13.1.3. Site Sync supports Independent Filesets
      • 5.13.1.4. Destination site Independent Filesets must exist prior to synchronisation
      • 5.13.1.5. Directory deletion is not supported
    • 5.13.2. Bidirectional Site Sync
      • 5.13.2.1. Bidirectional Site Sync implements eventual consistency
      • 5.13.2.2. Bidirectional Site Sync is only supported via schedules
      • 5.13.2.3. Bidirectional synchronisation is sequential
      • 5.13.2.4. Swapping files / Last write wins
      • 5.13.2.5. Renaming or deleting files and directories causes re-sending of data
      • 5.13.2.6. Renaming or files and directories on both bidirectional Site Sync sites causes duplication of data
      • 5.13.2.7. Creation or deletion of empty directories on site 1 does not synchronise to site 2
• 6. Troubleshooting
  • 6.1. Service Status
  • 6.2. Service Logs
  • 6.3. Specific Features
    • 6.3.1. Site Sync
      • 6.3.1.1. Snapshot Rotation
      • 6.3.1.2. Manual Resolution
      • 6.3.1.3. Re-sync all
      • 6.3.1.4. Force rotate
      • 6.3.1.5. Locking Errors
• 7. Reference
  • 7.1. Default Workflows
    • 7.1.1. migrate
    • 7.1.2. premigrate
    • 7.1.3. recall
    • 7.1.4. send
    • 7.1.5. site_sync
      • 7.1.5.1. created
      • 7.1.5.2. updated
      • 7.1.5.3. moved
      • 7.1.5.4. deleted
  • 7.2. Hidden Workflows
    • 7.2.1. transparent_recall
    • 7.2.2. reverse_stub
    • 7.2.3. delete_file
  • 7.3. Discovery Steps
    • 7.3.1. dynamo.tasks.recursive_action - recursive
    • 7.3.2. dynamo.tasks.snapdiff - snapdiff
  • 7.4. Workflow Steps
    • 7.4.1. dynamo.tasks.migrate
    • 7.4.2. dynamo.tasks.recall
    • 7.4.3. dynamo.tasks.reverse_stub
    • 7.4.4. dynamo.tasks.ngenea_sync_metadata
    • 7.4.5. dynamo.tasks.delete_paths_from_gpfs
    • 7.4.6. dynamo.tasks.check_sync_state
    • 7.4.7. dynamo.tasks.move_paths_on_gpfs
    • 7.4.8. dynamo.tasks.one_step_move_paths_on_gpfs
    • 7.4.9. dynamo.tasks.remove_location_xattrs_for_moved
    • 7.4.10. dynamo.tasks.move_in_cloud
    • 7.4.11. dynamo.tasks.remove_from_cloud
    • 7.4.12. dynamo.tasks.ensure_cloud_file_exists
  • 7.5. Job States
  • 7.6. Task States
  • 7.7. File States
    • 7.7.1. Processed
    • 7.7.2. Skipped
    • 7.7.3. Failed
    • 7.7.4. Aborted
  • 7.8. API
• 8. Tools
  • 8.1. ngclient
    • 8.1.1. NGCLIENT
      • 8.1.1.1. SYNOPSIS
      • 8.1.1.2. DESCRIPTION
      • 8.1.1.3. OPTION SUMMARY
        • 8.1.1.3.1. OPTIONS
      • 8.1.1.4. EXAMPLES
        • 8.1.1.4.1. GENERATE A CLIENT KEY
        • 8.1.1.4.2. RECALL A FILE
        • 8.1.1.4.3. PREMIGRATE A DIRECTORY RECURSIVELY
        • 8.1.1.4.4. SEND A FILE TO A REMOTE SITE
      • 8.1.1.5. AVAILABILITY
      • 8.1.1.6. SEE ALSO
      • 8.1.1.7. LICENSE
  • 8.2. ngclient-workflows
    • 8.2.1. NGCLIENT-WORKFLOWS
      • 8.2.1.1. SYNOPSIS
      • 8.2.1.2. DESCRIPTION
      • 8.2.1.3. OPTION SUMMARY
        • 8.2.1.3.1. OPTIONS
      • 8.2.1.4. EXAMPLES
        • 8.2.1.4.1. GENERATE A CLIENT KEY
        • 8.2.1.4.2. GET AN EXISTING WORKFLOW
        • 8.2.1.4.3. LIST ALL EXISTING WORKFLOWS IN YAML FORMAT
        • 8.2.1.4.4. IMPORT A CUSTOM WORKFLOW
        • 8.2.1.4.5. RENAME A WORKFLOW
        • 8.2.1.4.6. DELETE A WORKFLOW
      • 8.2.1.5. AVAILABILITY
      • 8.2.1.6. SEE ALSO
      • 8.2.1.7. LICENSE
  • 8.3. ngclient-features
    • 8.3.1. NGCLIENT-FEATURES
      • 8.3.1.1. SYNOPSIS
      • 8.3.1.2. DESCRIPTION
      • 8.3.1.3. OPTION SUMMARY
        • 8.3.1.3.1. OPTIONS
      • 8.3.1.4. EXAMPLES
        • 8.3.1.4.1. LIST AVAILABLE FEATURES
        • 8.3.1.4.2. ENABLE A FEATURE
        • 8.3.1.4.3. DISABLE A FEATURE
      • 8.3.1.5. AVAILABILITY
      • 8.3.1.6. SEE ALSO
      • 8.3.1.7. LICENSE
  • 8.4. ngenea-client.conf
    • 8.4.1. NGENEACLIENTCONF
      • 8.4.1.1. SYNOPSIS
      • 8.4.1.2. FILE FORMAT
      • 8.4.1.3. PARAMETERS
      • 8.4.1.4. EXAMPLE
      • 8.4.1.5. SEE ALSO
      • 8.4.1.6. LICENSE
  • 8.5. transparent_recall
    • 8.5.1. TRANSPARENT_RECALL
      • 8.5.1.1. SYNOPSIS
      • 8.5.1.2. DESCRIPTION
      • 8.5.1.3. OPTION SUMMARY
        • 8.5.1.3.1. OPTIONS
      • 8.5.1.4. CONFIGURATION
      • 8.5.1.5. TRANSPARENT RECALL POLICY
        • 8.5.1.5.1. WARNING
      • 8.5.1.6. TROUBLESHOOTING
    • 8.5.2. AVAILABILITY
      • 8.5.2.1. SEE ALSO
      • 8.5.2.2. LICENSE
• 9. Contact
• 10. Ngenea Hub Changelog
• 11. License