Administration Guide

Ngenea Hub harnesses the power of Ngenea to provide global workflows, enabling your data to be where you need it, when you need it.

• 1. Download
  • 1.1. Ngenea Hub
  • 1.2. Ngenea Worker
  • 1.3. Ngenea Hub Client
  • 1.4. Cloud Functions
• 2. Installation Guide
• 3. Upgrade
  • 3.1. Before you start
  • 3.2. Backup Workflows
  • 3.3. Stop Services
  • 3.4. Upgrade Packages
  • 3.5. Upgrade Configurations
  • 3.6. Restart Services
  • 3.7. Validation
  • 3.8. Restoring Workflows
• 4. Configuration
  • 4.1. Hub Initial Configuration
  • 4.2. Hub Configuration
    • 4.2.1. Settings
      • 4.2.1.1. Mandatory Settings
      • 4.2.1.2. Optional settings
    • 4.2.2. Enabling LDAP/Active Directory Authentication
    • 4.2.3. Broker Settings
    • 4.2.4. Server Configurations
    • 4.2.5. Docker Compose Configuration
  • 4.3. Worker Configuration
    • 4.3.1. Settings
    • 4.3.2. Controlling Functions
    • 4.3.3. Creating Custom Queues
    • 4.3.4. Using a Custom Configuration File
    • 4.3.5. Running Ngenea Worker with Multiple Sites
    • 4.3.6. Setting Ngenea Worker Debug Level in systemd Service Script
  • 4.4. Hub Messaging Queue Configuration
    • 4.4.1. Choosing the Messaging Queue
    • 4.4.2. Redis Setting Files
      • 4.4.2.1. Customizing Redis
      • 4.4.2.2. Restarting Redis to Apply Changes
    • 4.4.3. RabbitMQ Setting Files
    • 4.4.4. Using RabbitMQ’s Admin Interface
    • 4.4.5. Limitations
• 5. Feature Set-up
  • 5.1. Search
    • 5.1.1. Requirements
    • 5.1.2. Configuration Steps
      • 5.1.2.1. Choose Your Search Backend
      • 5.1.2.2. Set the Elasticsearch URL
    • 5.1.3. Using the Search Feature
  • 5.2. Cloud Functions
    • 5.2.1. GCP Cloud Function
    • 5.2.2. AWS Lambda Function
      • 5.2.2.1. Create an IAM Policy and Role
      • 5.2.2.2. Create the Lambda function
      • 5.2.2.3. Set Up the Trigger (S3 Bucket Event)
      • 5.2.2.4. Monitoring Lambda Functions
      • 5.2.2.5. AWS Lambda with Python 3.9: Adding Requests Module
    • 5.2.3. Configuration
      • 5.2.3.1. version
      • 5.2.3.2. hub_access
      • 5.2.3.3. Sites
      • 5.2.3.4. Actions
      • 5.2.3.5. optional
      • 5.2.3.6. vendor
      • 5.2.3.7. vendors
      • 5.2.3.8. Complete Example
  • 5.3. Ngenea Worker Plugins
    • 5.3.1. Creating your plugin
      • 5.3.1.1. Example plugin creation
      • 5.3.1.2. External Dependencies
      • 5.3.1.3. Installing custom plugins
    • 5.3.2. Managing plugins
      • 5.3.2.1. Enabling your plugin
      • 5.3.2.2. Installing your plugin
      • 5.3.2.3. Uninstalling your plugin
      • 5.3.2.4. Listing your plugin
    • 5.3.3. Custom Tasks
      • 5.3.3.1. The Structure of a Task
        • 5.3.3.1.1. Arguments
        • 5.3.3.1.2. Return Payload
      • 5.3.3.2. Support for Streaming Paths from API
      • 5.3.3.3. Queues
    • 5.3.4. Example Plugins
      • 5.3.4.1. Email notification task
      • 5.3.4.2. Running additional script
    • 5.3.5. SubDAG Helpers
  • 5.4. Disaster Recovery / Cold Failover
    • 5.4.1. Setup
      • 5.4.1.1. Configure datastore
      • 5.4.1.2. Configure Networking
      • 5.4.1.3. Install Ngenea Hub
    • 5.4.2. Performing failover
    • 5.4.3. Migration from local datastore
  • 5.5. External SSL
    • 5.5.1. Installing NGINX
    • 5.5.2. Configuration for SSL Termination and Reverse Proxy using OpenSSL
      • 5.5.2.1. Create Self-Signed Certificates for Nginx
      • 5.5.2.2. Creating configuration file for Nginx
    • 5.5.3. Configuration for SSL Termination and Reverse Proxy using Certbot and LetsEncrypt (Another method)
      • 5.5.3.1. Add trusted SSL Certificates from Letsencrypt
    • 5.5.4. HTTPS Configuration for NgeneaHub
      • 5.5.4.1. Reload the Nginx service after you have made some configuration changes
      • 5.5.4.2. To auto-renew the certificates, Run
  • 5.6. LDAP / Active Directory Login
    • 5.6.1. Configuration
      • 5.6.1.1. Example Configurations
    • 5.6.2. Managing users and groups from AD in HUB
  • 5.7. Queues
    • 5.7.1. Configuration
    • 5.7.2. Functions
      • 5.7.2.1. Disabling Functions
    • 5.7.3. Queue discovery
    • 5.7.4. Queue removal
    • 5.7.5. Memory considerations
    • 5.7.6. Workflows
• 6. Usage
  • 6.1. API
    • 6.1.1. Reference
    • 6.1.2. Authentication
      • 6.1.2.1. JWT Authentication
      • 6.1.2.2. Client Keys
        • 6.1.2.2.1. Creating Client Keys
        • 6.1.2.2.2. Using Client Keys
    • 6.1.3. Submitting Workflow
      • 6.1.3.1. Migrate
    • 6.1.4. Trailing Slashes
  • 6.2. Monitoring and Management
    • 6.2.1. systemd service
    • 6.2.2. ngeneahubctl cli tool
    • 6.2.3. Docker containers
    • 6.2.4. Hub Task Metrics
    • 6.2.5. Health endpoints
  • 6.3. Custom Workflows
    • 6.3.1. Defining workflows
      • 6.3.1.1. Filter Rules
      • 6.3.1.2. Runtime fields
    • 6.3.2. Running Workflows
      • 6.3.2.1. Discovery Steps
        • 6.3.2.1.1. Discovery Options
    • 6.3.3. Includes and Excludes
  • 6.4. Jobs
    • 6.4.1. Housekeeping
  • 6.5. Site Sync
    • 6.5.1. Troubleshooting
  • 6.6. Bidirectional Site Sync
    • 6.6.1. Schedule
    • 6.6.2. Workflow
    • 6.6.3. Troubleshooting
  • 6.7. Search
    • 6.7.1. Submitting a query
      • 6.7.1.1. Filters
    • 6.7.2. Retrieving results
      • 6.7.2.1. Proxies
      • 6.7.2.2. Parameters
      • 6.7.2.3. Merged results
      • 6.7.2.4. Results grouped by name
    • 6.7.3. Max Results
    • 6.7.4. Housekeeping
  • 6.8. Controlling Bandwidth Usage
    • 6.8.1. Bandwidth configuration in the UI
    • 6.8.2. Checking Node status
    • 6.8.3. Registering Datastores
      • 6.8.3.1. Cloud IPAddresses
      • 6.8.3.2. Using manual IP addresses
    • 6.8.4. Linking a Datastore to a Site
    • 6.8.5. Applying the bandwidth
  • 6.9. Configuration
    • 6.9.1. Global Configurations
      • 6.9.1.1. Available Settings
      • 6.9.1.2. REST API
      • 6.9.1.3. Settings Migration
    • 6.9.2. Site-specific Configurations
      • 6.9.2.1. Available Settings
        • 6.9.2.1.1. File Batching
        • 6.9.2.1.2. Dynamic File Batching
      • 6.9.2.2. REST API
  • 6.10. Certificate Lifetime
    • 6.10.1. Viewing the Certificate Lifetime
    • 6.10.2. Modifying the Certificate Lifetime
    • 6.10.3. Summary
  • 6.11. Feature Flags
    • 6.11.1. REST API
    • 6.11.2. ngclient
  • 6.12. Actions
    • 6.12.1. Available Actions
      • 6.12.1.1. discover_nodes
    • 6.12.2. REST API
  • 6.13. Limitations
    • 6.13.1. Site Sync
      • 6.13.1.1. Site Sync must only be used in one direction
      • 6.13.1.2. Site Sync only considers changes within an applicable time window for synchronisation
      • 6.13.1.3. Site Sync supports Independent Filesets
      • 6.13.1.4. Destination site Independent Filesets must exist prior to synchronisation
      • 6.13.1.5. Directory deletion is not supported
    • 6.13.2. Bidirectional Site Sync
      • 6.13.2.1. Bidirectional Site Sync implements eventual consistency
      • 6.13.2.2. Bidirectional Site Sync is only supported via schedules
      • 6.13.2.3. Bidirectional synchronisation is sequential
      • 6.13.2.4. Swapping files / Last write wins
      • 6.13.2.5. Renaming or deleting files and directories causes re-sending of data
      • 6.13.2.6. Renaming or files and directories on both bidirectional Site Sync sites causes duplication of data
      • 6.13.2.7. Creation or deletion of empty directories on site 1 does not synchronise to site 2
• 7. Troubleshooting
  • 7.1. Service Status
  • 7.2. Service Logs
  • 7.3. Specific Features
    • 7.3.1. Site Sync
      • 7.3.1.1. Snapshot Rotation
      • 7.3.1.2. Manual Resolution
      • 7.3.1.3. Re-sync all
      • 7.3.1.4. Force rotate
      • 7.3.1.5. Locking Errors
• 8. Reference
  • 8.1. Default Workflows
    • 8.1.1. migrate
    • 8.1.2. premigrate
    • 8.1.3. recall
    • 8.1.4. send
    • 8.1.5. unmanage
    • 8.1.6. site_sync
      • 8.1.6.1. created
      • 8.1.6.2. updated
      • 8.1.6.3. moved
      • 8.1.6.4. deleted
      • 8.1.6.5. import_to_site_hydrated
      • 8.1.6.6. import_to_site_dehydrated
  • 8.2. Hidden Workflows
    • 8.2.1. transparent_recall
    • 8.2.2. reverse_stub
    • 8.2.3. delete_file
  • 8.3. Discovery Steps
    • 8.3.1. dynamo.tasks.recursive_navigate - recursive - Function: discovery
    • 8.3.2. dynamo.tasks.recursive_action - recursive(deprecated) - Function: discovery
    • 8.3.3. dynamo.tasks.snapdiff - snapdiff - Function: discovery
  • 8.4. Workflow Steps
    • 8.4.1. dynamo.tasks.migrate - Function: Worker
    • 8.4.2. dynamo.tasks.recall - Function: Worker
    • 8.4.3. dynamo.tasks.reverse_stub - Function: Worker
    • 8.4.4. dynamo.tasks.ngenea_sync_metadata - Function: Worker
    • 8.4.5. dynamo.tasks.delete_paths_from_gpfs - Function: Worker
    • 8.4.6. dynamo.tasks.check_sync_state - Function: Worker
    • 8.4.7. dynamo.tasks.move_paths_on_gpfs - Function: Worker
    • 8.4.8. dynamo.tasks.one_step_move_paths_on_gpfs - Function: Worker
    • 8.4.9. dynamo.tasks.remove_location_xattrs_for_moved - Function: Worker
    • 8.4.10. dynamo.tasks.move_in_cloud - Function: Worker
    • 8.4.11. dynamo.tasks.remove_from_cloud - Function: Worker
    • 8.4.12. dynamo.tasks.ensure_cloud_file_exists - Function: Worker
    • 8.4.13. dynamo.tasks.import_files_from_external_target - Function: Worker
    • 8.4.14. dynamo.tasks.copy_files_from_external_target
    • 8.4.15. dynamo.tasks.import_bytes - Function: Worker
    • 8.4.16. dynamo.tasks.recall_bytes - Function: Worker
  • 8.5. Automated Tasks
    • 8.5.1. Every minute
      • 8.5.1.1. alerts.tasks.fetch_alerts
    • 8.5.2. Every 30 minutes
      • 8.5.2.1. dynamocore.tasks.admin.load_storagepools
      • 8.5.2.2. dynamocore.tasks.space_site.send_check_space_site_quota
    • 8.5.3. Hourly
      • 8.5.3.1. dynamocore.tasks.jobs.update_state_as_failed_for_inactive_tasks
      • 8.5.3.2. dynamocore.tasks.jobs.invalidate_cancelled_job_tasks
      • 8.5.3.3. dynamocore.tasks.network.ensure_consistent_tc_rules
      • 8.5.3.4. dynamocore.tasks.events.cleanup_old_events
      • 8.5.3.5. dynamocore.tasks.browsing.browsing_task_clear
      • 8.5.3.6. dynamocore.tasks.system_settings.create_or_update_site_settings_auto
      • 8.5.3.7. dynamocore.tasks.system_settings.sync_global_settings_auto
      • 8.5.3.8. dynamocore.tasks.spaces.sync_spaces
    • 8.5.4. Every 12 Hours
      • 8.5.4.1. dynamocore.tasks.site.fetch_site_analytics
    • 8.5.5. Daily
      • 8.5.5.1. dynamocore.tasks.search.remove_old_search_results
      • 8.5.5.2. dynamocore.tasks.jobs.expire_old_jobs
      • 8.5.5.3. dynamocore.tasks.network.update_cloud_ips_for_s3
      • 8.5.5.4. dynamocore.tasks.network.update_cloud_ips_for_azure
      • 8.5.5.5. dynamocore.tasks.network.update_cloud_ips_for_google
      • 8.5.5.6. dynamocore.tasks.server.get_remote_servers
      • 8.5.5.7. dynamocore.tasks.jobs.expire_old_fsobjects
      • 8.5.5.8. dynamocore.tasks.refresh.delete_removed_queues
  • 8.6. Job States
  • 8.7. Task States
  • 8.8. File States
    • 8.8.1. Processed
    • 8.8.2. Skipped
    • 8.8.3. Failed
    • 8.8.4. Aborted
  • 8.9. API
• 9. Tools
  • 9.1. ngclient
    • 9.1.1. NGCLIENT
      • 9.1.1.1. SYNOPSIS
      • 9.1.1.2. DESCRIPTION
      • 9.1.1.3. OPTION SUMMARY
        • 9.1.1.3.1. OPTIONS
      • 9.1.1.4. EXAMPLES
        • 9.1.1.4.1. GENERATE A CLIENT KEY
        • 9.1.1.4.2. RECALL A FILE
        • 9.1.1.4.3. PREMIGRATE A DIRECTORY RECURSIVELY
        • 9.1.1.4.4. SEND A FILE TO A REMOTE SITE
      • 9.1.1.5. AVAILABILITY
      • 9.1.1.6. SEE ALSO
      • 9.1.1.7. LICENSE
  • 9.2. ngclient-workflows
    • 9.2.1. NGCLIENT-WORKFLOWS
      • 9.2.1.1. SYNOPSIS
      • 9.2.1.2. DESCRIPTION
      • 9.2.1.3. OPTION SUMMARY
        • 9.2.1.3.1. OPTIONS
      • 9.2.1.4. EXAMPLES
        • 9.2.1.4.1. GENERATE A CLIENT KEY
        • 9.2.1.4.2. GET AN EXISTING WORKFLOW
        • 9.2.1.4.3. LIST ALL EXISTING WORKFLOWS IN YAML FORMAT
        • 9.2.1.4.4. IMPORT A CUSTOM WORKFLOW
        • 9.2.1.4.5. RENAME A WORKFLOW
        • 9.2.1.4.6. DELETE A WORKFLOW
      • 9.2.1.5. AVAILABILITY
      • 9.2.1.6. SEE ALSO
      • 9.2.1.7. LICENSE
  • 9.3. ngclient-features
    • 9.3.1. NGCLIENT-FEATURES
      • 9.3.1.1. SYNOPSIS
      • 9.3.1.2. DESCRIPTION
      • 9.3.1.3. OPTION SUMMARY
        • 9.3.1.3.1. OPTIONS
      • 9.3.1.4. EXAMPLES
        • 9.3.1.4.1. LIST AVAILABLE FEATURES
        • 9.3.1.4.2. ENABLE A FEATURE
        • 9.3.1.4.3. DISABLE A FEATURE
      • 9.3.1.5. AVAILABILITY
      • 9.3.1.6. SEE ALSO
      • 9.3.1.7. LICENSE
  • 9.4. ngclient-settings
    • 9.4.1. NGCLIENT-SETTINGS
      • 9.4.1.1. SYNOPSIS
      • 9.4.1.2. DESCRIPTION
      • 9.4.1.3. OPTION SUMMARY
        • 9.4.1.3.1. OPTIONS
      • 9.4.1.4. EXAMPLES
        • 9.4.1.4.1. REFRESH SETTINGS ON ONE SITE
        • 9.4.1.4.2. REFRESH SETTINGS ON ALL SITES
      • 9.4.1.5. AVAILABILITY
      • 9.4.1.6. SEE ALSO
      • 9.4.1.7. LICENSE
  • 9.5. ngclient-sites
    • 9.5.1. NGCLIENT-SITES
      • 9.5.1.1. SYNOPSIS
      • 9.5.1.2. DESCRIPTION
      • 9.5.1.3. OPTION SUMMARY
        • 9.5.1.3.1. OPTIONS
      • 9.5.1.4. EXAMPLES
        • 9.5.1.4.1. REFRESH METADATA ON ONE SITE
        • 9.5.1.4.2. REFRESH METADATA ON ALL SITES
      • 9.5.1.5. AVAILABILITY
      • 9.5.1.6. SEE ALSO
      • 9.5.1.7. LICENSE
  • 9.6. ngenea-client.conf
    • 9.6.1. NGENEACLIENTCONF
      • 9.6.1.1. SYNOPSIS
      • 9.6.1.2. FILE FORMAT
      • 9.6.1.3. PARAMETERS
      • 9.6.1.4. EXAMPLE
      • 9.6.1.5. SEE ALSO
      • 9.6.1.6. LICENSE
  • 9.7. transparent_recall
    • 9.7.1. TRANSPARENT_RECALL
      • 9.7.1.1. SYNOPSIS
      • 9.7.1.2. DESCRIPTION
      • 9.7.1.3. OPTION SUMMARY
        • 9.7.1.3.1. OPTIONS
      • 9.7.1.4. CONFIGURATION
      • 9.7.1.5. TRANSPARENT RECALL POLICY
        • 9.7.1.5.1. Adjusting the transparent recall queue
      • 9.7.1.6. TROUBLESHOOTING
    • 9.7.2. AVAILABILITY
      • 9.7.2.1. SEE ALSO
      • 9.7.2.2. LICENSE
• 10. Backup
  • 10.1. Backing Up HUB Static Configurations
    • 10.1.1. Backup
      • 10.1.1.1. Command:
    • 10.1.2. Restore
      • 10.1.2.1. Command: