Skip to Content
DocumentationFeaturesIntegrationsDjango-RQArchitecture

Django-RQ Architecture

Complete technical overview of Django-RQ integration in django-cfg, including system design, component interaction, and implementation details.

System Overview

Django-RQ in django-cfg consists of several interconnected components:

Core Components

1. Configuration Layer

Type-safe configuration using Pydantic v2:

Key Features:

  • Validation: Pydantic ensures type safety and correct values
  • Smart defaults: Uses redis_url from parent DjangoConfig
  • Flexibility: Support for Redis URL, Sentinel, SSL connections
  • Declarative: Schedule tasks directly in config

2. Django-RQ Integration

Django-CFG seamlessly integrates with django-rq:

Generated Settings:

# Django settings.py (auto-generated)
RQ_QUEUES = {
    'default': {
        'URL': 'redis://localhost:6379/0',
        'DEFAULT_TIMEOUT': 360,
        'DEFAULT_RESULT_TTL': 500,
    },
    'high': {
        'URL': 'redis://localhost:6379/0',
        'DEFAULT_TIMEOUT': 180,
        'DEFAULT_RESULT_TTL': 300,
    },
}
RQ_SHOW_ADMIN_LINK = True

3. Task Execution Flow

Complete lifecycle of a job from enqueue to completion:

4. Queue Management

Multiple queues with different priorities:

Queue Strategy:

  • high: Critical tasks (payment processing, auth)
  • default: Normal tasks (emails, notifications)
  • low: Batch operations (reports, cleanup)
  • knowledge: AI/ML tasks (document processing, embeddings)

5. Scheduler Architecture

RQ Scheduler provides cron-like scheduling:

Schedule Registration:

Monitoring Architecture

REST API Layer

Django-CFG provides comprehensive REST API for monitoring:

Data Flow

Pydantic Models

Internal models for type-safe data handling:

Integration Patterns

1. Enqueue from Anywhere

Tasks can be enqueued from any Django component:

2. Result Handling

Multiple ways to handle job results:

3. Error Handling

Comprehensive error handling and retry logic:

Performance Optimization

1. Worker Pool Architecture

Multiple workers for parallelism:

Benefits:

  • Utilize multiple CPU cores
  • Parallel job execution
  • Better throughput
  • Fault isolation

2. Redis Data Structures

Efficient use of Redis:

3. Connection Pooling

Redis connection management:

Deployment Architecture

Single-Server Setup

Multi-Server Setup

See Also

Configuration

Reference

OverviewConfiguration