2. Plugin Development

Ngenea Hub supports extending workflow functionality through custom plugins — Python packages containing Celery tasks that execute on Ngenea Worker nodes alongside built-in tasks.

This guide covers everything needed to develop, deploy, and integrate custom plugins with Ngenea Hub.

• 2.1. Quick Start
  • 2.1.1. Prerequisites
  • 2.1.2. 1. Scaffold
  • 2.1.3. 2. Write Your Task
    • 2.1.3.1. The Rules
  • 2.1.4. 3. Configure the Entry Point
  • 2.1.5. 4. Install and Restart
  • 2.1.6. 5. Create a Workflow
  • 2.1.7. Passing Parameters
  • 2.1.8. Chaining Tasks
  • 2.1.9. File States Reference
  • 2.1.10. Updating Your Plugin
  • 2.1.11. Troubleshooting Cheat Sheet
  • 2.1.12. What’s Next?
• 2.2. Architecture Overview
  • 2.2.1. What Is a Plugin?
    • 2.2.1.1. Use Cases
  • 2.2.2. Where Plugins Fit
  • 2.2.3. Key Concepts
• 2.3. Getting Started
  • 2.3.1. Prerequisites
  • 2.3.2. Scaffold the Plugin
  • 2.3.3. Implement the Task
  • 2.3.4. Configure the Entry Point
  • 2.3.5. Install and Activate
  • 2.3.6. Create a Workflow
  • 2.3.7. Run It
• 2.4. The Task Contract
  • 2.4.1. Task Registration
  • 2.4.2. Function Signature
  • 2.4.3. Path Object Structure
  • 2.4.4. Return Payload
    • 2.4.4.1. Using FileActionStatus (Recommended)
  • 2.4.5. Supported File States
  • 2.4.6. Complete Pattern
  • 2.4.7. What Happens If the Contract Is Violated
• 2.5. Workflows
  • 2.5.1. Workflow Structure
  • 2.5.2. Filter Rules
    • 2.5.2.1. Required Keys
    • 2.5.2.2. Optional Keys
    • 2.5.2.3. Action Objects
  • 2.5.3. Task Chaining
  • 2.5.4. Path Filtering with Include/Exclude
  • 2.5.5. Multiple Rules (Branching)
  • 2.5.6. Referencing Your Plugin
  • 2.5.7. Workflow Fields
    • 2.5.7.1. Defining Fields
    • 2.5.7.2. Field Type Reference
    • 2.5.7.3. Referencing Fields in Actions
  • 2.5.8. Submitting a Workflow via API
• 2.6. Advanced Topics
  • 2.6.1. Streaming Task Results
    • 2.6.1.1. Using StreamTaskResults
    • 2.6.1.2. Streaming Path Input
  • 2.6.2. Dynamic Children
    • 2.6.2.1. Using DynamicSubmitter
    • 2.6.2.2. Why will_spawn_children Matters
  • 2.6.3. Sub-DAGs
    • 2.6.3.1. Single-Task Sub-DAG
    • 2.6.3.2. Multi-Task Sub-DAG (Chains)
  • 2.6.4. Queue Utilities
• 2.7. Plugin Lifecycle
  • 2.7.1. Creating a Plugin
  • 2.7.2. Enabling Plugins
  • 2.7.3. Installing a Plugin
  • 2.7.4. Listing Plugins
  • 2.7.5. Updating a Plugin
  • 2.7.6. Managing Dependencies
  • 2.7.7. Uninstalling a Plugin
• 2.8. Troubleshooting
  • 2.8.1. Common Issues
  • 2.8.2. Logging
  • 2.8.3. Inspecting Return Payloads
• 2.9. Examples
  • 2.9.1. Email Notification
    • 2.9.1.1. Task Implementation
    • 2.9.1.2. Entry Point (setup.py)
    • 2.9.1.3. Workflow Definition
  • 2.9.2. Script Execution
    • 2.9.2.1. Task Implementation
    • 2.9.2.2. Entry Point (setup.py)
    • 2.9.2.3. Workflow Definition
  • 2.9.3. File Mover
    • 2.9.3.1. Task Implementation
    • 2.9.3.2. Entry Point (setup.py)
    • 2.9.3.3. Workflow Definition