3.2. Hub Configuration

3.2.1. Settings

Ngenea Hub requires some important settings for correct operation. These settings tell the system how to connect to its database, process tasks, and secure data.

The main configuration file is located at:

/etc/sysconfig/ngeneahub

This file contains all the necessary information for Ngenea Hub to function. If you do not set these values, the system will use default settings.

3.2.1.1. Mandatory Settings

Here’s a simple explanation of each important setting:

Setting

Description

DJANGO_SECRET

A secret key that protects important system data, like user sessions. Think of it as a password used internally by the system.

POSTGRES_DB

The name of the database where Ngenea Hub stores its information.

POSTGRES_USER

The username that allows the system to connect to the database.

POSTGRES_PASSWORD

The password used to access the database securely.

WORKER_THREADS

Controls how many tasks can be handled at the same time. Default: 2.

DAG_THREADS

Helps process multiple tasks at the same time. Default: 7.

REFRESH_THREADS

Controls how many jobs can refresh at the same time. DefaultL 2

BROKER_PROVIDER

Defines the task queue broker service (rabbitmq, redis, external_rabbitmq or external_redis). Default: redis.

RESULT_PROVIDER

Defines the task result storage (redis, external_redis, or external_redis_cluster). Default: redis.

RABBITMQ_USER

The username for RabbitMQ (if used). Default: ngeneahub.

RABBITMQ_PASSWORD

The password for RabbitMQ (if used). Default: ngeneahub.

RABBITMQ_VHOST

A virtual host inside RabbitMQ for organizing tasks. Default: nghub.

TASK_DAEMON_BATCH_SIZE

Number of tasks that update their status when a job starts. Default: 100.

DAG_REFRESH_INTERVAL

Controls how often in seconds the job refresh daemon polls for new jobs to refresh. Default: 1

DAG_REFRESH_MAX_AGE

Controls how long in seconds the job refresh daemon will wait for a refresh to complete before retrying it. Default: 600

DAG_REFRESH_BATCH_SIZE

Controls the maximum number of job refreshes that will be queued per refresh interval. Default: 5

JWT_PRIVATE_KEY

A key used to sign user authentication tokens. Generated on first startup.

JWT_PUBLIC_KEY

A key used to verify authentication tokens. Generated on first startup.

JWT_EXPOSED_JSON

Stores extra details related to authentication security.

GF_SECURITY_ADMIN_PASSWORD

Password for accessing Grafana, the system monitoring tool.

HUB_PORT

The port where Ngenea Hub runs. Default: 8000.

WEB_BIND_IP

The IP address used for connecting to the system. Default: 0.0.0.0.

SHARED_SECRET

A secret key for secure communication within the system.

DJANGO_ENCRYPTED_FIELDS_KEY

Encryption key used to secure sensitive database information.

DJANGO_EJF_CRYPTER_KEY

Additional security key for encrypting Django responses.

3.2.1.2. Optional settings

Settings and Description

  • REDIS_HOST: The address of the Redis broker or results store, can be external to the hub services. Defaults to the container service address redis.

  • REDIS_PORT: The port of the Redis broker or results store, can be external to the hub services. Defaults to 6379.

  • REDIS_PASSWORD: The password for an external Redis broker or results store, this setting is exclusively for external services.

  • REDIS_TLS: Enables TLS support for external Redis services, defaults to False.

  • RABBITMQ_HOST: The address of a RabbitMQ service. Can be external to the hub services. Can be a comma-separated list. Defaults to the container service address rabbitmq.

  • RABBITMQ_PORT: The port of the RabbitMQ broker service, defaults to 5671.

  • RABBITMQ_TLS: Enables TLS support for external RabbitMQ services, defaults to False.

  • RABBITMQ_QUEUE_TYPE: Queue type to use when creating queues in RabbitMQ, when there is only one RABBITMQ_HOST. Either classic or quorum. Defaults to classic. The type of a queue can’t be changed after it has been created. Configure as quorum if you plan to scale out RabbitMQ.

  • POSTGRES_HOST: The address of the PostgresSQL database instance. Defaults to container service address.

  • POSTGRES_PORT: The port of the PostgresSQL database instance. Defaults to container service port.

  • POSTGRES_TARGET_SESSION_ATTRS: The sesssion properties to apply to the connection to a PostgresSQL database instance. Defaults to any.

  • WORKERS: The number of workers for API requests. More workers allow the system to handle more requests simultaneously. Default: 8. This is deprecated. Please use GUNICORN_WORKERS instead.

  • GUNICORN_THREADS: The number of worker threads for handling requests. Typically, use 2–4 × CPU cores; adjust as needed for your workload. Default: 2

  • GUNICORN_WORKERS: The number of gunicorn workers for API requests. More gunicorn workers allow the system to handle more requests simultaneously. This setting will override any WORKERS settings provided. Default: 8

  • API_TIMEOUT: The timeout in seconds for API requests. Default: 600 seconds.

  • GATEWAY_TIMEOUT: The timeout for requests passing through nginx. Should be greater than or equal to API_TIMEOUT. Default: 600 seconds.

  • CONSUMER_TIMEOUT: The timeout for RabbitMQ consumer delivery acknowledgment in seconds. Default: 10800000 seconds (3 hours).

  • PUBLIC_URL: Configurable base URL for the hub stack. Must not end in a trailing slash.

  • CELERY_THREADS: The number of concurrency threads for handling multiple background tasks. More threads mean faster processing of tasks.

  • EVENT_THREADS: The number of concurrency threads for handling event reporting tasks. More threads help report events faster.

  • RESULT_THREADS: The number of concurrency threads for handling result reporting tasks. More threads mean quicker reporting of task results.

  • HEARTBEAT: Enables or disables celery heartbeats to keep connections alive. Default: True (enabled).

  • GOSSIP: Enables or disables celery gossip communication. Default: False (disabled).

  • MINGLE: Enables or disables celery mingle for coordination. Default: False (disabled).

  • REDIS_CLUSTER_STARTUP_NODES: Comma-separated list of Redis Cluster nodes in the form host:port. For use when Hub is configured to use Redis Cluster as its result provider.

  • REDIS_HEALTH_CHECK_INTERVAL: The interval in seconds between health checks for Redis backend. Default: 60 seconds.

  • REDIS_TCP_BACKLOG: The number of pending requests to Redis. Higher values help avoid slow connections in high request environments. Default: 511.

  • REDIS_SOCKET_TIMEOUT: The timeout in seconds for resetting idle Redis backend connection sockets. Default: 60 seconds.

  • CELERY_SOCKET_TIMEOUT: The timeout in seconds for resetting idle Celery broker connection sockets. Default: 60 seconds.

  • CELERY_CONNECTION_TIMEOUT: The timeout in seconds for resetting idle connections via Redis broker. Default: 60 seconds.

  • BROKER_CONNECTION_RETRY_ON_STARTUP: This enables all internal workers to retry to connect to the broker on startup only. Defaults: True.

  • BROKER_CONNECTION_RETRY: This enables all internal workers to retry to connect to the broker. Defaults: True.

  • BROKER_CONNECTION_MAX_RETRIES: This controls how many attempts the internal works will attempt once connection is lost. Defaults: 10.

  • BROKER_CANCEL_ON_CONNECTION_LOSS: This kills long running tasks in the internal workers if connections has been lost. Defaults: True.

  • CELERY_BROKER_HEARTBEAT: Used to detect if a connection was lost between the broker and clients. Default: 10 seconds. (RabbitMQ only)

  • CELERY_BROKER_HEARTBEAT_CHECKRATE: How often to check for missed broker heartbeats. Default: 2 per interval.(RabbitMQ only)

  • CELERY_SOCKET_KEEPALIVE: Enable TCP socket keepalive. Default: true. (Redis only)

  • CELERY_SOCKET_KEEPALIVE_IDLE: Start probing for liveness after the socket has been idle for this long. Default: 60s. (Redis only)

  • CELERY_SOCKET_KEEPALIVE_COUNT: Number of probe attempts before an idle socket is considered dead. Default: 5. (Redis only)

  • CELERY_SOCKET_KEEPALIVE_INTERVAL: Interval between idle socket probe attempts. Default: 10 seconds(Redis only)

  • EXPIRE_OLD_JOBS_INTERVAL: Cron schedule for expiring old jobs. Default: 0 0 * * * (daily).

  • REMOVE_OLD_SEARCH_RESULTS_INTERVAL: Cron schedule for removing old search results. Default: 0 0 * * * (daily).

  • INVALIDATE_CANCELLED_JOB_TASKS_INTERVAL: Cron schedule for revoking cancelled job tasks. Default: 0 * * * * (every hour).

  • CLEANUP_OLD_EVENTS_INTERVAL: Cron schedule for cleaning up old snapdiff events. Default: 0 * * * * (every hour).

  • INACTIVE_TASKS_INTERVAL: Cron schedule for invalidating inactive tasks. Default: 0 * * * * (every hour).

  • SYNC_SITE_SETTINGS_INTERVAL: Cron schedule for syncing site settings in the database. Default: 0 * * * * (every hour).

  • SYNC_GLOBAL_SETTINGS_INTERVAL: Cron schedule for syncing global settings across all sites. Default: 0 * * * * (every hour).

  • REFRESH_SITE_ANALYTICS_INTERVAL: Cron schedule for refreshing site analytics. Default: 37 */12 * * * (every 12 hours).

  • SYNC_STORAGE_POOLS_INTERVAL: Cron schedule for syncing storage pools. Default: */30 * * * * (every 30 minutes).

  • SYNC_REMOTE_SERVERS_INTERVAL: Cron schedule for syncing remote servers. Default: 0 0 * * * (once a day).

  • SYNC_SPACES_QUOTA_INTERVAL: Cron schedule for syncing spaces’ quotas. Default: */30 * * * * (every 30 minutes).

  • SYNC_SPACES_INTERVAL: Cron schedule for syncing spaces. Default: 0 * * * * (every hour).

  • SYNC_ALERTS_INTERVAL: Cron schedule for syncing alerts from all sites. Default: * * * * * (every minute).

  • EXPIRE_OLD_FSOBJECTS_INTERVAL: Cron schedule for expiring old file system objects. Default: 0 0 * * * (daily).

  • REMOVED_QUEUE_CLEANUP_INTERVAL: Cron schedule for cleaning up removed queues. Default: 24 * 60 (every day).

  • CERTIFICATE_LIFE_TIME_DAYS: Lifetime of SSL certificates in days. Default: 36500 (100 years).

  • QUEUE_ONLINE_CHECK_INTERVAL: Interval in seconds to check if a queue is online. Default: 10 seconds.

  • SCHEDULE_LOCK_EXPIRY: Time after which the schedule lock should expire (in seconds) if the holding process dies without releasing the lock.

  • OVERWRITE_COMPOSE: Enables or disables ngeneahubctl overwriting changes to the compose file when running any service operations

3.2.2. Broker Settings

The messaging queue configuration for Ngenea Hub is a crucial component in managing communication between workers and the Hub.

This system supports two main brokers: Redis and RabbitMQ, offering flexibility depending on your specific requirements and infrastructure. Configuring the messaging queue is handled through various settings in the /etc/sysconfig/ngeneahub file, along with specific configuration files for Redis and RabbitMQ.

In the sections that follow, we will delve into the details of selecting and configuring the messaging queue, including how to manage Redis and RabbitMQ settings, enable the RabbitMQ admin interface, and understand the limitations of these configurations within the Hub environment.

For more details, refer to Hub Messaging Queue Configuration.

3.2.3. RabbitMQ HA

Ngenea Hub supports the use of High Availability (HA) RabbitMQ, using Quorum Queues, as the broker provider.

RabbitMQ HA replicates queues to additional hosts to support failover in the event that one or more hosts go down.

To use RabbitMQ HA as the broker provider, configure

BROKER_PROVIDER=external_rabbitmq
RABBITMQ_HOST=my-pixstor-mn01,my-pixstor-ng01,my-pixstor-ng02
RABBITMQ_PORT=5671
# other RABBITMQ_* settings

Configure all the hosts in the RabbitMQ cluster as a comma-separated list in RABBITMQ_HOST. The same RABBITMQ_PORT will used for all hosts by default. If each host uses a different port, the ports can be included in RABBITMQ_HOST, for example host1:1234,host2:5678,...

If multiple hosts are configured, Hub will assume that queues should be quorum queues. If only one host is configured, Hub will assume the queues are non-quorum classic queues by default.

Note

Queues cannot change type once created. If you wish to move from a single-host to HA, existing ‘classic’ queues will need to be deleted and re-created as quorum queues.

If you have a new system with only one RabbitMQ host, but plan to switch to HA in future, configure RABBITMQ_QUEUE_TYPE=quorum. This ensures that you will not need to delete and re-create your queues when switching to HA.

3.2.4. Redis Cluster

Ngenea Hub supports the use of Redis Cluster as a result provider.

Redis Cluster provides high availability (HA) and improved performance and throughput compared to single instance Redis.

To use Redis Cluster as the result provider, configure

RESULT_PROVIDER=external_redis_cluster
REDIS_HOST=my-pixstor-mn01
REDIS_PORT=6001
# other REDIS_* settings

Where REDIS_HOST and REDIS_PORT point to any node within the Redis Cluster. The Redis client will automatically discover the other nodes in the cluster.

The redis_cluster_startup_nodes configuration can be used to declare all the nodes within the Redis Cluster. This is used during Hub start up, in the event that the node configured as REDIS_HOST is offline or uncontactable. The Hub will identify at least one of the ‘startup nodes’ which is online and contactable, and use it to discover the rest of the available nodes.

Redis Cluster cannot be used as the broker provider.

3.2.5. Server Configurations

In Ngenea Hub, certain settings are stored in the Ngenea Hub database and can be easily viewed and modified via the Ngenea Hub REST API at the /api/configurations/ endpoint.

The Ngenea Hub REST API allows you to manage configuration settings both globally and on a per-site basis. These configurations control various aspects of the system, such as job time-to-live (TTL), search backend setup, snapshot retries, task invalidation times, and more. By using simple API requests, you can optimize your system’s performance and behavior.

For a comprehensive overview of available configurations, including how to view and change them through the API, as well as additional features like dynamic file batching, refer to the full documentation - Configuration.