Programming Guide

This guide is intended for users who work with, build, or integrate with the Hub APIs. It provides a clear overview of how the API is structured, how to interact with it effectively, and the standards followed in its design and implementation. You’ll find information on authentication, endpoint conventions, request/response formats, error handling, versioning, and best practices.

• 1. REST API
  • 1.1. Authentication Methods
    • 1.1.1. Overview
    • 1.1.2. API Reference Location
    • 1.1.3. JWT Authentication - Using Curl
    • 1.1.4. Client Key Authentication
    • 1.1.5. Key Differences Between JWT and API Keys
    • 1.1.6. JWT Authentication - Using Swagger UI
    • 1.1.7. Generate a Client Key - Using Swagger UI
  • 1.2. API Endpoints - User and Access Management
    • 1.2.1. Introduction
    • 1.2.2. API Reference Location
    • 1.2.3. Auth - Authentication
      • 1.2.3.1. GET /auth/clientkeys/
      • 1.2.3.2. GET /auth/clientkeys/{id}/
      • 1.2.3.3. DELETE /auth/clientkeys/{id}/
      • 1.2.3.4. GET /api/auth/token/publickey/
      • 1.2.3.5. POST /api/auth/token/verify/
    • 1.2.4. Users
      • 1.2.4.1. GET /users/
      • 1.2.4.2. POST /users/
      • 1.2.4.3. POST /users/change_password/
      • 1.2.4.4. GET /users/{username}/
      • 1.2.4.5. PATCH /users/{username}/
      • 1.2.4.6. DELETE /users/{username}/
      • 1.2.4.7. POST /users/{username}/activate/
      • 1.2.4.8. POST /users/{username}/deactivate/
    • 1.2.5. My Permissions
      • 1.2.5.1. GET /my-permissions/
    • 1.2.6. Permissions
      • 1.2.6.1. GET /permissions/
      • 1.2.6.2. GET /permissions/{id}/
    • 1.2.7. Groups
      • 1.2.7.1. GET /groups/
      • 1.2.7.2. POST /groups/
      • 1.2.7.3. GET /groups/{id}/
      • 1.2.7.4. PATCH /groups/{id}/
      • 1.2.7.5. DELETE /groups/{id}/
  • 1.3. API Endpoint - Policies
    • 1.3.1. API Reference Location
    • 1.3.2. GET /policies/
    • 1.3.3. POST /policies/
    • 1.3.4. GET /policies/{id}/
    • 1.3.5. PATCH /policies/{id}/
    • 1.3.6. DELETE /policies/{id}/
      • 1.3.6.1. POST /policies/{id}/run/
  • 1.4. API Endpoints - File Management and Storage
    • 1.4.1. File
      • 1.4.1.1. GET /file/
      • 1.4.1.2. POST /file/workflow/
    • 1.4.2. Filesystems
      • 1.4.2.1. GET /filesystems/
      • 1.4.2.2. GET /filesystems/{id}/
    • 1.4.3. Filesets
      • 1.4.3.1. GET /filesets/
    • 1.4.4. Sites
      • 1.4.4.1. GET /sites/
      • 1.4.4.2. POST /sites/
      • 1.4.4.3. GET /sites/{id}/
      • 1.4.4.4. PATCH /sites/{id}/
      • 1.4.4.5. DELETE /sites/{id}/
      • 1.4.4.6. GET /sites/{id}/health/
      • 1.4.4.7. POST /sites/{id}/refresh/
      • 1.4.4.8. GET /sites/{id}/settings/
      • 1.4.4.9. POST /sites/{id}/settings/refresh/
    • 1.4.5. Shares
      • 1.4.5.1. GET /shares/nfs/
      • 1.4.5.2. POST /shares/nfs/
      • 1.4.5.3. GET /shares/nfs/{id}/
      • 1.4.5.4. GET /shares/samba/
      • 1.4.5.5. POST /shares/samba/
      • 1.4.5.6. GET /shares/samba/{id}/
      • 1.4.5.7. PATCH /shares/samba/{id}/
      • 1.4.5.8. DELETE /shares/samba/{id}/
    • 1.4.6. Spaces
      • 1.4.6.1. GET /spaces/
      • 1.4.6.2. POST / spaces/
      • 1.4.6.3. GET /spaces/{id}/
      • 1.4.6.4. DELETE /spaces/{id}/
      • 1.4.6.5. GET /spaces/{id}/files/
    • 1.4.7. Buckets
      • 1.4.7.1. GET /api/buckets/
      • 1.4.7.2. POST /api/buckets/
      • 1.4.7.3. GET /buckets/{id}/
      • 1.4.7.4. PATCH /buckets/{id}/
      • 1.4.7.5. DELETE /buckets/{id}/
    • 1.4.8. Storage Pools
      • 1.4.8.1. GET /storagepools/
      • 1.4.8.2. GET /storagepools/{id}/
  • 1.5. API Endpoints - Configuration & System Settings
    • 1.5.1. Configurations
      • 1.5.1.1. GET /configurations/
      • 1.5.1.2. PATCH /configurations/
    • 1.5.2. Features - Feature Flag
      • 1.5.2.1. GET /features/
      • 1.5.2.2. GET /features/{name}/
      • 1.5.2.3. PATCH /features/{name}/
    • 1.5.3. Nodes
      • 1.5.3.1. GET /nodes/
      • 1.5.3.2. DELETE /nodes/{id}/
    • 1.5.4. Settings Tasks
      • 1.5.4.1. GET /settingstasks/
      • 1.5.4.2. GET /settingstasks/{id}/
    • 1.5.5. Tags
      • 1.5.5.1. GET /tags/
    • 1.5.6. Servers
      • 1.5.6.1. GET /servers/
      • 1.5.6.2. GET /servers/{id}/
  • 1.6. API Endpoints - Workflows, Automation & Jobs
    • 1.6.1. Workflows
      • 1.6.1.1. GET /workflows
      • 1.6.1.2. POST /workflows
      • 1.6.1.3. GET /workflows/{id}/
      • 1.6.1.4. PATCH /workflows/{id}/
      • 1.6.1.5. DELETE /workflows/{id}/
    • 1.6.2. Automated Schedules
      • 1.6.2.1. GET /api/automatedschedules/
    • 1.6.3. Tasks
      • 1.6.3.1. GET/ tasks/
      • 1.6.3.2. GET /tasks/{id}/
      • 1.6.3.3. GET /tasks/{id}/files/
    • 1.6.4. Queues
      • 1.6.4.1. GET /queues/
      • 1.6.4.2. GET /queues/{id}/
  • 1.7. API Endpoints - Monitoring and Analytics
    • 1.7.1. Alerts
      • 1.7.1.1. GET /alerts/
      • 1.7.1.2. GET /alerts/{id}/
    • 1.7.2. Health
      • 1.7.2.1. GET /health/
      • 1.7.2.2. GET /health/sites/
    • 1.7.3. Search
      • 1.7.3.1. POST /search/
      • 1.7.3.2. PATCH /search/metadata/
      • 1.7.3.3. GET /search/metadata_fields/
      • 1.7.3.4. GET /search/{id}/
      • 1.7.3.5. DELETE /search/{id}/
    • 1.7.4. Analytics
      • 1.7.4.1. GET /analytics
• 2. Plugin Development
  • 2.1. Quick Start
    • 2.1.1. Prerequisites
    • 2.1.2. 1. Scaffold
    • 2.1.3. 2. Write Your Task
      • 2.1.3.1. The Rules
    • 2.1.4. 3. Configure the Entry Point
    • 2.1.5. 4. Install and Restart
    • 2.1.6. 5. Create a Workflow
    • 2.1.7. Passing Parameters
    • 2.1.8. Chaining Tasks
    • 2.1.9. File States Reference
    • 2.1.10. Updating Your Plugin
    • 2.1.11. Troubleshooting Cheat Sheet
    • 2.1.12. What’s Next?
  • 2.2. Architecture Overview
    • 2.2.1. What Is a Plugin?
      • 2.2.1.1. Use Cases
    • 2.2.2. Where Plugins Fit
    • 2.2.3. Key Concepts
  • 2.3. Getting Started
    • 2.3.1. Prerequisites
    • 2.3.2. Scaffold the Plugin
    • 2.3.3. Implement the Task
    • 2.3.4. Configure the Entry Point
    • 2.3.5. Install and Activate
    • 2.3.6. Create a Workflow
    • 2.3.7. Run It
  • 2.4. The Task Contract
    • 2.4.1. Task Registration
    • 2.4.2. Function Signature
    • 2.4.3. Path Object Structure
    • 2.4.4. Return Payload
      • 2.4.4.1. Using FileActionStatus (Recommended)
    • 2.4.5. Supported File States
    • 2.4.6. Complete Pattern
    • 2.4.7. What Happens If the Contract Is Violated
  • 2.5. Workflows
    • 2.5.1. Workflow Structure
    • 2.5.2. Filter Rules
      • 2.5.2.1. Required Keys
      • 2.5.2.2. Optional Keys
      • 2.5.2.3. Action Objects
    • 2.5.3. Task Chaining
    • 2.5.4. Path Filtering with Include/Exclude
    • 2.5.5. Multiple Rules (Branching)
    • 2.5.6. Referencing Your Plugin
    • 2.5.7. Workflow Fields
      • 2.5.7.1. Defining Fields
      • 2.5.7.2. Field Type Reference
      • 2.5.7.3. Referencing Fields in Actions
    • 2.5.8. Submitting a Workflow via API
  • 2.6. Advanced Topics
    • 2.6.1. Streaming Task Results
      • 2.6.1.1. Using StreamTaskResults
      • 2.6.1.2. Streaming Path Input
    • 2.6.2. Dynamic Children
      • 2.6.2.1. Using DynamicSubmitter
      • 2.6.2.2. Why will_spawn_children Matters
    • 2.6.3. Sub-DAGs
      • 2.6.3.1. Single-Task Sub-DAG
      • 2.6.3.2. Multi-Task Sub-DAG (Chains)
    • 2.6.4. Queue Utilities
  • 2.7. Plugin Lifecycle
    • 2.7.1. Creating a Plugin
    • 2.7.2. Enabling Plugins
    • 2.7.3. Installing a Plugin
    • 2.7.4. Listing Plugins
    • 2.7.5. Updating a Plugin
    • 2.7.6. Managing Dependencies
    • 2.7.7. Uninstalling a Plugin
  • 2.8. Troubleshooting
    • 2.8.1. Common Issues
    • 2.8.2. Logging
    • 2.8.3. Inspecting Return Payloads
  • 2.9. Examples
    • 2.9.1. Email Notification
      • 2.9.1.1. Task Implementation
      • 2.9.1.2. Entry Point (setup.py)
      • 2.9.1.3. Workflow Definition
    • 2.9.2. Script Execution
      • 2.9.2.1. Task Implementation
      • 2.9.2.2. Entry Point (setup.py)
      • 2.9.2.3. Workflow Definition
    • 2.9.3. File Mover
      • 2.9.3.1. Task Implementation
      • 2.9.3.2. Entry Point (setup.py)
      • 2.9.3.3. Workflow Definition