Django-RQ Architecture

Complete technical overview of Django-RQ integration in django-cfg, including system design, component interaction, and implementation details.

System Overview

Django-RQ in django-cfg consists of several interconnected components:

Core Components

1. Configuration Layer

Type-safe configuration using Pydantic v2:

Key Features:

• Validation: Pydantic ensures type safety and correct values
• Smart defaults: Uses redis_url from parent DjangoConfig
• Flexibility: Support for Redis URL, Sentinel, SSL connections
• Declarative: Schedule tasks directly in config

2. Django-RQ Integration

Django-CFG seamlessly integrates with django-rq:

Generated Settings:

# Django settings.py (auto-generated)
RQ_QUEUES = {
    'default': {
        'URL': 'redis://localhost:6379/0',
        'DEFAULT_TIMEOUT': 360,
        'DEFAULT_RESULT_TTL': 500,
    },
    'high': {
        'URL': 'redis://localhost:6379/0',
        'DEFAULT_TIMEOUT': 180,
        'DEFAULT_RESULT_TTL': 300,
    },
}
RQ_SHOW_ADMIN_LINK = True

3. Task Execution Flow

Complete lifecycle of a job from enqueue to completion:

4. Queue Management

Multiple queues with different priorities:

Queue Strategy:

• high: Critical tasks (payment processing, auth)
• default: Normal tasks (emails, notifications)
• low: Batch operations (reports, cleanup)
• knowledge: AI/ML tasks (document processing, embeddings)

5. Scheduler Architecture

RQ Scheduler provides cron-like scheduling:

Schedule Registration:

Monitoring Architecture

REST API Layer

Django-CFG provides comprehensive REST API for monitoring:

Data Flow

Pydantic Models

Internal models for type-safe data handling:

Integration Patterns

1. Enqueue from Anywhere

Tasks can be enqueued from any Django component:

2. Result Handling

Multiple ways to handle job results:

3. Error Handling

Comprehensive error handling and retry logic:

Performance Optimization

1. Worker Pool Architecture

Multiple workers for parallelism:

Benefits:

• Utilize multiple CPU cores
• Parallel job execution
• Better throughput
• Fault isolation

2. Redis Data Structures

Efficient use of Redis:

3. Connection Pooling

Redis connection management:

Deployment Architecture

Single-Server Setup

Multi-Server Setup

Troubleshooting: Stale Schedule Data

Problem: Scheduled Tasks Don’t Repeat

If your interval-based scheduled tasks run once but don’t repeat, the issue is likely stale job data in Redis.

Symptoms:

• Scheduler logs show ”✓ Registered interval schedule”
• Task executes once at startup
• Task disappears from rq:scheduler:scheduled_jobs after first run
• job.meta is empty ({}) when fetched

Root Cause:

When a job with the same ID already exists in Redis with corrupted/empty meta, rq-scheduler may not properly update it. The interval value in meta is required for repeat execution.

Solution:

Django-cfg 1.7.3+ includes automatic cleanup of stale job data. For older versions, manually clear scheduler data:

# Clear all scheduled job data
redis-cli KEYS "rq:job:schedule_*" | xargs redis-cli DEL
redis-cli DEL "rq:scheduler:scheduled_jobs"
 
# Restart scheduler to re-register
docker restart your-scheduler-container

Verification:

# Check that meta is saved correctly after registration
from rq.job import Job
from django_rq import get_connection
 
conn = get_connection()
job = Job.fetch('schedule_your_task_id', connection=conn)
print(f'meta: {job.meta}')  # Should show {'interval': 60} or similar

Prevention:

Django-cfg 1.7.3+ automatically:

1. Removes stale job data before registering new schedules
2. Verifies meta is saved correctly in Redis
3. Logs warnings if meta appears corrupted

See Also

Configuration

• Configuration Guide - Detailed setup guide
• Examples - Real-world examples
• Deployment - Production deployment

Reference

• Django-RQ Docs  - Official documentation
• RQ Docs  - Core RQ documentation
• Redis Docs  - Redis documentation

Related

• Integration Patterns - Common patterns