2.7. Plugin Lifecycle

This section covers the full lifecycle of a plugin: creating, installing, updating, listing, and uninstalling.

2.7.1. Creating a Plugin

Use the CLI to scaffold a new plugin:

ngenea-worker plugins create <plugin_name>

This creates the following structure under /var/lib/ngenea-worker/plugins/<plugin_name>/:

<plugin_name>/
├── setup.py                    # Package metadata and entry points
├── constraints.txt             # Critical package version constraints (read-only reference)
└── <plugin_name>/
    ├── __init__.py
    └── <plugin_name>.py        # Your task implementation goes here

The generated module contains a default example_task function that you should rename and modify for your use case.

2.7.2. Enabling Plugins

Plugins must be enabled in the Ngenea Worker configuration:

  1. Edit /etc/ngenea/ngenea-worker.conf.

  2. Set enable_plugins=true.

  3. Save the file.

2.7.3. Installing a Plugin

Install all plugins in the plugins directory:

ngenea-worker plugins install

Install a specific plugin:

ngenea-worker plugins install <plugin_name>

After installing, restart the ngenea-worker service to load the new tasks:

systemctl restart ngenea-worker

Warning

Restarting the Worker service immediately terminates all in-flight tasks on that host. Tasks that are mid-execution (e.g., file moves, script runs) may leave the system in an inconsistent state — for example, a file that has been deleted from the source but not yet written to the destination. Before restarting, check the Ngenea Hub UI for active jobs on this Worker and either wait for them to complete or ensure the affected workflows are idempotent and can safely be re-run.

2.7.4. Listing Plugins

To list installed plugins and their versions:

ngenea-worker plugins list

This shows two sections:

  • Installed plugins — plugins currently installed with their versions.

  • Available plugins — all plugins in the /var/lib/ngenea-worker/plugins directory, whether installed or not.

2.7.5. Updating a Plugin

To update an installed plugin:

  1. Modify the task code in the plugin module.

  2. Best practice: Increment the version number in setup.py to reflect an updated plugin.

  3. If necessary copy the new code to each worker node that needs the plugin.

  4. Reinstall the plugin on each node:

    ngenea-worker plugins install <plugin_name>

  5. Restart the service on each node:

    systemctl restart ngenea-worker

2.7.6. Managing Dependencies

If your plugin requires external Python packages, add them to the install_requires list in setup.py:

install_requires = [
    "celery<=5.4",
    "requests>=2.28",
    "my-custom-lib==1.0.0",
]

Danger

Never modify or force-install versions of celery or other constrained packages (listed in the templates constraints.txt) by bypassing the ngenea-worker plugins install command. Doing so can cause the Worker to fail to start, crash during task execution, or introduce silent data corruption in task payloads due to serialization incompatibilities. The constraints exist to guarantee compatibility with Ngenea Hub’s DAG engine.

If your plugin requests a dependency that conflicts with a constrained package, you will see an error like:

ERROR: Cannot install example_plugin==0.1.0 because these package versions
have conflicting dependencies.

The conflict is caused by:
    example_plugin 0.1.0 depends on celery==5.4
    The user requested (constraint) celery==5.3.6

The user requested (constraint) refers to Ngenea Worker’s internal constraints.

Your plugin directory includes a constraints.txt file listing the required critical package versions. This file is for reference and testing only — editing it will not change the constraints applied during installation.

2.7.7. Uninstalling a Plugin

Uninstall a specific plugin:

On each worker node the plugin is installed on:

ngenea-worker plugins uninstall <plugin_name>

Uninstall all plugins:

On each worker node the plugins are installed on:

pushd /var/lib/ngenea-worker/plugins/
ngenea-worker plugins uninstall * -y
popd

Danger

Running ngenea-worker plugins uninstall * outside of /var/lib/ngenea-worker/plugins/ or including non-plugin package names can uninstall core Worker components, causing the Worker service to fail entirely. This will halt all workflow processing on the affected host and require manual reinstallation to recover. Always cd to the plugins directory first and verify the glob expansion before confirming.

After uninstalling, restart the service:

systemctl restart ngenea-worker