3.4. Hub Messaging Queue Configuration

When setting up communication between various components of the Ngenea Hub, such as workers and the hub itself, a messaging queue plays a crucial role. These messaging queues act as intermediaries, efficiently managing and delivering messages between components.

The Ngenea Hub supports two messaging queue systems: Redis and RabbitMQ. Below is a step-by-step guide to understanding and configuring these systems.

3.4.1. Choosing the Messaging Queue

The Ngenea Hub provides two primary options for a messaging queue:

  • Redis: Known for its speed and simplicity, Redis is an excellent choice for systems requiring high performance with straightforward configuration.

  • RabbitMQ : RabbitMQ offers advanced features compared to Redis, including more sophisticated tools for managing queues, making it a better choice for complex messaging requirements.

The messaging between workers and Ngenea Hub is configurable between two different supported brokers. This is controlled with variables in /etc/sysconfig/ngeneahub along with additional configuration files.

The following are all broker related configuration values:

Setting

Default

Description

CELERY_BROKER

redis

This controls the messaging queue redis|rabbitmq

RABBITMQ_USER

ngeneahub

User for the rabbitmq user for use with Ngenea Hub

RABBITMQ_PASSWORD

Randomly generated password

Password used for the rabbitmq user for use with Ngenea Hub

RABBITMQ_VHOST

nghub

The virtual host used within the rabbitmq instance within Ngenea Hub

REDIS_HOST

redis

Address of the Redis queue results store. Defaults to the container service address.

REDIS_HEALTH_CHECK_INTERVAL

60

The Redis backend supports health checks. This value must be set as an integer whose value is the number of seconds between health checks.

REDIS_TCP_BACKLOG

511

In high requests-per-second environments you need a high backlog in order to avoid slow client connections issues to redis.

REDIS_SOCKET_TIMEOUT

60

When there are network issues redis backend connection sockets can become stale, this timeout setting will reset the socket connection after this value in seconds after becoming idle and resume operation.

For more details on general configuration see Hub Configuration for more details.

3.4.2. Redis Setting Files

When setting up Redis within Ngenea Hub, you’re provided with a default configuration file located at: /etc/ngenea/redis/redis.conf, any edits made to this file will persist with software upgrades.

This file is pre-configured to help get you started with Redis, but it can be customized to better meet your specific needs. Common adjustments include tuning memory usage, adjusting data persistence settings, or optimizing performance based on the expected workload.

3.4.2.1. Customizing Redis

To make changes to the Redis configuration, follow these steps:

  • Open the Configuration File: Use a text editor to open the redis.conf file: sudo nano /etc/ngenea/redis/redis.conf

  • Edit the File: Make the necessary changes to the configuration. The file allows you to adjust various Redis settings, including memory limits, persistence options, and more. For a full list of configurable options, you can refer to Redis configuration.

  • Save the Changes: After making the required changes, save the file and close the text editor.

3.4.2.2. Restarting Redis to Apply Changes

For your changes to take effect, Redis must be restarted. There are two ways to restart the Redis service:

Preferred Method: Restarting Ngenea Hub

The easiest and preferred method is to restart the entire Ngenea Hub system. This can be done with the following command:

sudo systemctl restart ngeneahub

This will restart all services, including Redis, and apply your changes without any risk of inadvertently disrupting the Redis container.

Alternative: Restarting Only the Redis Container

Note

Content not approved

If you prefer to restart just the Redis container without affecting other services, you can restart it using Docker. However, this method may have unintended consequences, as it only restarts Redis and not the other dependent services in Ngenea Hub.

To restart the Redis container, use the following command:

docker restart ngeneahub_redis_1

While this can be useful for testing small changes, it’s generally safer to restart the entire system to avoid potential issues with container dependencies.

3.4.3. RabbitMQ Setting Files

If Ngenea Hub is configured to use RabbitMQ, additional configuration files can be provided within the directory: /etc/ngenea/rabbitmq/conf.d

This directory can be populated with custom configuration files for broader RabbitMQ configuration.

Default Configuration Files: The provided default configuration files within the /etc/ngenea/rabbitmq/conf.d directory are defined and updated in: /usr/share/ngeneahub/rabbitmq.

These default configuration files will be replaced during software upgrades. Therefore, it is important to apply any changes within custom configuration files to preserve your customizations.

Customizing RabbitMQ: RabbitMQ allows extensive customization, such as setting limits on the number of messages processed concurrently.

Note: Be mindful that some default configuration files may be overwritten during software updates. To avoid losing customizations, save your configuration changes in separate files within the /etc/ngenea/rabbitmq/conf.d directory.

Applying Changes: After modifying the RabbitMQ configuration files, you must restart the RabbitMQ service to apply the changes. This can be done with the following command: sudo systemctl restart rabbitmq-server.

3.4.4. Using RabbitMQ’s Admin Interface

Unlike Redis, RabbitMQ does not come with built-in command line tools to monitor or manage its operations directly.

However, a tool known as the “admin panel” is available for RabbitMQ. By default, it is not turned on, but it can be enabled by running the following command in the Ngenea Hub host’s command line:

ngeneahubctl exec -c rabbitmq rabbitmq-plugins enable rabbitmq_management

Once it is turned on, access can be gained by visiting the address (port 15672) on the Ngenea Hub.

http://<hub-host>:15672

To access this, a login is required using a username and password. These login details are stored in a specific file on the server, called /etc/sysconfig/ngeneahub, where two important pieces of information can be found:

  • RABBITMQ_USER : The username to log in.

  • RABBITMQ_PASSWORD : The password for the username.

3.4.5. Limitations

When using RabbitMQ, it’s important to note that its data—such as queue metrics, worker statuses, and throughput—will not appear in the Ngenea Hub’s built-in monitoring tool, Grafana, which is accessible via:

/hubmetrics

Instead, you must rely on the RabbitMQ admin panel to view and manage such metrics.