3.2. Hub Configuration

3.2.1. Settings

Ngenea Hub requires some important settings to work properly. These settings tell the system how to connect to its database, process tasks, and secure data.

The main configuration file is located at:

/etc/sysconfig/ngeneahub

This file contains all the necessary information for Ngenea Hub to function. If you do not set these values, the system will use default settings.

3.2.1.1. Mandatory Settings

Here’s a simple explanation of each important setting:

Setting

Description

DJANGO_SECRET

A secret key that protects important system data, like user sessions. Think of it as a password used internally by the system.

POSTGRES_DB

The name of the database where Ngenea Hub stores its information.

POSTGRES_USER

The username that allows the system to connect to the database.

POSTGRES_PASSWORD

The password used to access the database securely.

WORKER_THREADS

Controls how many tasks can be handled at the same time. Default: 2.

DAG_THREADS

Helps process multiple tasks at the same time. Default: 7.

REFRESH_THREADS

Controls how many jobs can refresh at the same time. DefaultL 2

CELERY_BROKER

Defines the task queue broker (RabbitMQ or Redis). Default: Redis.

RABBITMQ_USER

The username for RabbitMQ (if used). Default: ngeneahub.

RABBITMQ_PASSWORD

The password for RabbitMQ (if used). Default: ngeneahub.

RABBITMQ_VHOST

A virtual space inside RabbitMQ for organizing tasks. Default: nghub.

TASK_DAEMON_BATCH_SIZE

Number of tasks that update their status when a job starts. Default: 100.

DAG_REFRESH_INTERVAL

Controls how often in seconds the job refresh daemon polls for new jobs to refresh. Default: 1

DAG_REFRESH_MAX_AGE

Controls how long in seconds the job refresh daemon will wait for a refresh to complete before retrying it. Default: 600

DAG_REFRESH_BATCH_SIZE

Controls the maximum number of job refreshes that will be queued per refresh interval. Default: 5

JWT_PRIVATE_KEY

A key used to sign user authentication tokens. Generated on first startup.

JWT_PUBLIC_KEY

A key used to verify authentication tokens. Generated on first startup.

JWT_EXPOSED_JSON

Stores extra details related to authentication security.

GF_SECURITY_ADMIN_PASSWORD

Password for accessing Grafana, the system monitoring tool.

HUB_PORT

The port where Ngenea Hub runs. Default: 8000.

WEB_BIND_IP

The IP address used for connecting to the system. Default: 0.0.0.0.

SHARED_SECRET

A secret key for secure communication within the system.

DJANGO_ENCRYPTED_FIELDS_KEY

Encryption key used to secure sensitive database information.

DJANGO_EJF_CRYPTER_KEY

Additional security key for encrypting Django responses.

3.2.1.2. Optional settings

Settings and Description

  • REDIS_HOST: The address of the Redis queue results store. Defaults to the container service address.

  • WORKERS: The number of workers for API requests. More workers allow the system to handle more requests simultaneously. Default: 8.

  • API_TIMEOUT: The timeout in seconds for API requests. Default: 600 seconds.

  • GATEWAY_TIMEOUT: The timeout for requests passing through nginx. Should be greater than or equal to API_TIMEOUT. Default: 600 seconds.

  • CONSUMER_TIMEOUT: The timeout for RabbitMQ consumer delivery acknowledgment in seconds. Default: 10800000 seconds (3 hours).

  • PUBLIC_URL: Configurable base URL for the hub stack. Must not end in a trailing slash.

  • CELERY_THREADS: The number of concurrency threads for handling multiple background tasks. More threads mean faster processing of tasks.

  • EVENT_THREADS: The number of concurrency threads for handling event reporting tasks. More threads help report events faster.

  • RESULT_THREADS: The number of concurrency threads for handling result reporting tasks. More threads mean quicker reporting of task results.

  • HEARTBEAT: Enables or disables celery heartbeats to keep connections alive. Default: True (enabled).

  • GOSSIP: Enables or disables celery gossip communication. Default: False (disabled).

  • MINGLE: Enables or disables celery mingle for coordination. Default: False (disabled).

  • REDIS_HEALTH_CHECK_INTERVAL: The interval in seconds between health checks for Redis backend. Default: 60 seconds.

  • REDIS_TCP_BACKLOG: The number of pending requests to Redis. Higher values help avoid slow connections in high request environments. Default: 511.

  • REDIS_SOCKET_TIMEOUT: The timeout in seconds for resetting idle Redis backend connection sockets. Default: 60 seconds.

  • CELERY_SOCKET_TIMEOUT: The timeout in seconds for resetting idle Celery broker connection sockets. Default: 60 seconds.

  • CELERY_CONNECTION_TIMEOUT: The timeout in seconds for resetting idle connections via Redis broker. Default: 60 seconds.

  • EXPIRE_OLD_JOBS_INTERVAL: Cron schedule for expiring old jobs. Default: 0 0 * * * (daily).

  • REMOVE_OLD_SEARCH_RESULTS_INTERVAL: Cron schedule for removing old search results. Default: 0 0 * * * (daily).

  • INVALIDATE_CANCELLED_JOB_TASKS_INTERVAL: Cron schedule for revoking cancelled job tasks. Default: 0 * * * * (every hour).

  • CLEANUP_OLD_EVENTS_INTERVAL: Cron schedule for cleaning up old snapdiff events. Default: 0 * * * * (every hour).

  • INACTIVE_TASKS_INTERVAL: Cron schedule for invalidating inactive tasks. Default: 0 * * * * (every hour).

  • SYNC_SITE_SETTINGS_INTERVAL: Cron schedule for syncing site settings in the database. Default: 0 * * * * (every hour).

  • SYNC_GLOBAL_SETTINGS_INTERVAL: Cron schedule for syncing global settings across all sites. Default: 0 * * * * (every hour).

  • REFRESH_SITE_ANALYTICS_INTERVAL: Cron schedule for refreshing site analytics. Default: 37 */12 * * * (every 12 hours).

  • SYNC_STORAGE_POOLS_INTERVAL: Cron schedule for syncing storage pools. Default: */30 * * * * (every 30 minutes).

  • SYNC_REMOTE_SERVERS_INTERVAL: Cron schedule for syncing remote servers. Default: 0 0 * * * (once a day).

  • SYNC_SPACES_QUOTA_INTERVAL: Cron schedule for syncing spaces’ quotas. Default: */30 * * * * (every 30 minutes).

  • SYNC_SPACES_INTERVAL: Cron schedule for syncing spaces. Default: 0 * * * * (every hour).

  • SYNC_ALERTS_INTERVAL: Cron schedule for syncing alerts from all sites. Default: ** * * * * *** (every minute).

  • EXPIRE_OLD_FSOBJECTS_INTERVAL: Cron schedule for expiring old file system objects. Default: 0 0 * * * (daily).

  • REMOVED_QUEUE_CLEANUP_INTERVAL: Cron schedule for cleaning up removed queues. Default: 24 * 60 (every day).

3.2.2. Enabling LDAP/Active Directory Authentication

This section of the guide explains how to integrate LDAP/Active Directory with Ngenea Hub for user authentication and group management. It covers required configuration settings, automatic user account creation, and group mirroring. It also explains how changes in AD (like adding or removing users from groups) are reflected in Hub after the next login.

For more details, refer to LDAP / Active Directory Login.

3.2.3. Broker Settings

The messaging queue configuration for Ngenea Hub is a crucial component in managing communication between workers and the Hub.

This system supports two main brokers: Redis and RabbitMQ, offering flexibility depending on your specific requirements and infrastructure. Configuring the messaging queue is handled through various settings in the /etc/sysconfig/ngeneahub file, along with specific configuration files for Redis and RabbitMQ.

In the sections that follow, we will delve into the details of selecting and configuring the messaging queue, including how to manage Redis and RabbitMQ settings, enable the RabbitMQ admin interface, and understand the limitations of these configurations within the Hub environment.

For more details, refer to Hub Messaging Queue Configuration.

3.2.4. Server Configurations

In Ngenea Hub, certain settings are stored in the Ngenea Hub database and can be easily viewed and modified via the Ngenea Hub REST API at the /api/configurations/ endpoint.

The Ngenea Hub REST API allows you to manage configuration settings both globally and on a per-site basis. These configurations control various aspects of the system, such as job time-to-live (TTL), search backend setup, snapshot retries, task invalidation times, and more. By using simple API requests, you can optimize your system’s performance and behavior.

For a comprehensive overview of available configurations, including how to view and change them through the API, as well as additional features like dynamic file batching, refer to the full documentation - Configuration.

3.2.5. Docker Compose Configuration

The Docker Compose configuration file for Ngenea Hub is located at /usr/share/ngeneahub/docker/docker-compose.yml.

If you need to make changes or add customizations, you can create an override file at /usr/share/ngeneahub/docker/docker-compose.override.yml. This override file allows you to extend or modify the default settings without changing the original configuration file. This setup makes it easier to customize your Docker Compose environment according to your specific needs.