Skip to Content
FeaturesModulesLLM IntegrationBalance Monitoring

LLM Balance Monitoring

Production-Ready Balance Monitoring

Django-CFG provides enterprise-grade LLM balance monitoring with modular provider architecture, type-safe Pydantic models, and multi-channel notifications (Email + Telegram). Prevent service interruptions by monitoring OpenAI API keys and OpenRouter prepaid balances automatically.

Monitor your LLM provider account balances automatically to avoid service interruptions. The system uses a modular provider architecture, checks balances hourly, caches results efficiently, and sends multi-channel notifications when balance drops below configurable thresholds.

Features

  • ✅ Modular Architecture - Provider-based design for easy extensibility
  • ✅ Type-Safe - Pydantic v2 models for all responses
  • ✅ Multi-Provider Support - OpenAI (key validation), OpenRouter (balance tracking)
  • ✅ Smart Balance Checking:
    • OpenAI: API key validation (no public balance API)
    • OpenRouter: Real prepaid balance via /api/v1/credits
  • ✅ Intelligent Caching - 1-hour TTL for balances, 24-hour for notifications
  • ✅ Multi-Channel Notifications - Email + Telegram with async queue
  • ✅ Two-Tier Thresholds - Warning ($10) and Critical ($5) levels
  • ✅ Anti-Spam Protection - One notification per level per 24 hours
  • ✅ API Error Alerts - Automatic notifications for invalid keys, quota exceeded
  • ✅ Dashboard API - REST endpoint for frontend integration
  • ✅ Management Command - Easy testing and cron integration

Quick Start

1. Configure API Keys

Set your LLM provider API keys in config.py:

from django_cfg import DjangoConfig, ApiKeys
 
class MyConfig(DjangoConfig):
    project_name: str = "My Project"
 
    # API Keys
    api_keys: ApiKeys = ApiKeys(
        openai=env.api_keys.openai,
        openrouter=env.api_keys.openrouter,
    )
 
    # Admin Notifications
    admin_emails: list[str] = [
        "[email protected]",
        "[email protected]",
    ]
 
    # Telegram (optional but recommended)
    telegram: TelegramConfig = TelegramConfig(
        bot_token=env.telegram.bot_token,
        chat_id=env.telegram.chat_id,
    )

In your .env or .env.secrets file:

# API Keys
API_KEYS__OPENAI=sk-proj-...
API_KEYS__OPENROUTER=sk-or-v1-...
 
# Admin Emails
ADMIN_EMAILS=["[email protected]","[email protected]"]
 
# Telegram
TELEGRAM__BOT_TOKEN=123456:ABC-DEF...
TELEGRAM__CHAT_ID=-1001234567890

2. Run Manual Check

Test the monitoring system:

python manage.py check_llm_balance

Example output:

======================================================================
LLM BALANCE CHECK
======================================================================

Checking balances...

  Openai: API Key Valid ✓ (Balance check not available via API...)
  Openrouter: $0.69 USD 🚨 CRITICAL [limit: $217.50, usage: $216.81]

Checking notification thresholds...
  🚨 Sent CRITICAL notification: Openrouter

======================================================================
CHECK COMPLETE
======================================================================
Total balance across all providers: $0.69 USD

⚠️ Low balance alerts were sent to administrators

Waiting for notification delivery...
✓ Notifications delivered

3. Setup Cron

Add to crontab for automatic hourly checks:

# Check LLM balances every hour
0 * * * * cd /path/to/project && /path/to/venv/bin/python manage.py check_llm_balance

Configuration

Admin Emails

Admin emails are configured at the root DjangoConfig level:

from django_cfg import DjangoConfig
 
class MyConfig(DjangoConfig):
    project_name: str = "My Project"
 
    # Admin emails - used for LLM balance alerts, error notifications, etc.
    admin_emails: list[str] = [
        "[email protected]",
        "[email protected]",
    ]

Why Root Level?

admin_emails is at the root DjangoConfig level because:

  • Used for multiple notification types (email + Telegram)
  • Application-level concept (not email-specific)
  • Simpler access: config.admin_emails vs config.email.admin_emails

Telegram Configuration

For Telegram notifications:

from django_cfg import TelegramConfig
 
class MyConfig(DjangoConfig):
    telegram: TelegramConfig = TelegramConfig(
        bot_token=env.telegram.bot_token,
        chat_id=env.telegram.chat_id,
    )

How to get Telegram credentials:

  1. Bot Token: Create bot via @BotFather 

    • Send /newbot command
    • Follow instructions
    • Copy token (format: 123456:ABC-DEF...)

  2. Chat ID:

Environment Variables

# .env or .env.secrets
 
# API Keys
API_KEYS__OPENAI=sk-proj-your-openai-key
API_KEYS__OPENROUTER=sk-or-v1-your-openrouter-key
 
# Admin Emails
ADMIN_EMAILS=["[email protected]","[email protected]"]
 
# Email Configuration (for sending alerts)
EMAIL__BACKEND=smtp
EMAIL__HOST=smtp.gmail.com
EMAIL__PORT=587
EMAIL__USERNAME=[email protected]
EMAIL__PASSWORD=your-app-password
EMAIL__USE_TLS=True
EMAIL__DEFAULT_FROM=[email protected]
 
# Telegram Configuration (for sending alerts)
TELEGRAM__BOT_TOKEN=
TELEGRAM__CHAT_ID=

Notification Thresholds

Default thresholds can be customized:

from django_cfg.modules.django_llm.monitoring import LLMBalanceNotifier
 
# Customize thresholds
LLMBalanceNotifier.THRESHOLD_WARNING = 20.0   # $20 USD
LLMBalanceNotifier.THRESHOLD_CRITICAL = 10.0  # $10 USD

Or modify in source:

# django_cfg/modules/django_llm/monitoring/notifier.py
 
class LLMBalanceNotifier:
    THRESHOLD_WARNING = 20.0   # Default: 10.0
    THRESHOLD_CRITICAL = 10.0  # Default: 5.0

Management Command

Basic Usage

# Normal check (with caching)
python manage.py check_llm_balance
 
# Force fresh API calls (bypass balance cache)
python manage.py check_llm_balance --force
 
# Force send notifications (bypass notification cache)
python manage.py check_llm_balance --force-notify

Command Options

OptionDescription
--forceBypass 1-hour balance cache and fetch fresh data from provider APIs
--force-notifySend notifications even if already sent in last 24 hours

Output Examples

Healthy balances:

  Openai: API Key Valid ✓ (Balance check not available via API...)
  Openrouter: $85.50 USD ✓ OK [limit: $217.50, usage: $132.00]

Checking notification thresholds...
  ✓ No notifications needed

Low balance (warning):

  Openai: API Key Valid ✓
  Openrouter: $8.50 USD ⚠️ WARNING [limit: $217.50, usage: $209.00]

Checking notification thresholds...
  ⚠️ Sent WARNING notification: Openrouter

Critical balance:

  Openai: API Key Valid ✓
  Openrouter: $3.20 USD 🚨 CRITICAL [limit: $217.50, usage: $214.30]

Checking notification thresholds...
  🚨 Sent CRITICAL notification: Openrouter

⚠️ Low balance alerts were sent to administrators

Waiting for notification delivery...
✓ Notifications delivered

API errors:

  ✗ Openai: Incorrect API key provided
  ✗ Openrouter: OpenRouter API HTTP error: 401

Checking notification thresholds...
  🔴 Sent API error notification: Openai
  🔴 Sent API error notification: Openrouter

Notification Format

Email Notification

Subject: 🚨 CRITICAL: Openrouter Balance Low

Body:

🚨 Openrouter API Balance Alert

Current Balance: $0.69 USD
Threshold: $5.00 USD

Please add funds to your Openrouter account to avoid service interruption.

---
This is an automated alert from LLM Balance Monitoring.
You will receive this notification once per 24 hours until the balance is restored.

Telegram Notification

Same message sent to configured Telegram chat with Markdown formatting.

API Error Notifications

When API key errors occur:

Subject: 🔴 CRITICAL: OpenAI API Key Error

Body:

🔴 OpenAI API Key Error

Error: Incorrect API key provided

This is a critical issue that requires immediate attention:
- Check if the API key is valid and correctly configured
- Verify the API key has not expired
- Ensure sufficient funds are available in the account

Configuration location:
- Check .env or .env.secrets file
- Verify API_KEYS__OPENAI environment variable

---
This is an automated alert from LLM Balance Monitoring.
You will receive this notification once per 24 hours until the issue is resolved.

Programmatic Usage

Type-Safe Balance Checking

Use Pydantic models for type-safe access:

from django_cfg.modules.django_llm.monitoring import BalanceChecker, LLMBalanceNotifier
 
# Check balances
checker = BalanceChecker()
balances = checker.check_all_balances(force=False)
 
# Access balance data (Pydantic models)
for provider, balance_data in balances.items():
    print(f"{provider}:")
    print(f"  Balance: ${balance_data.balance}")
    print(f"  Status: {balance_data.status}")
    print(f"  Error: {balance_data.error}")
    print(f"  Note: {balance_data.note}")
 
# Send notifications if thresholds exceeded
notifier = LLMBalanceNotifier()
results = notifier.check_all_and_notify(balances, force=False)
 
# Check what was sent
for provider, level in results.items():
    if level:
        print(f"Sent {level} notification for {provider}")

Using Individual Providers

from django_cfg.modules.django_llm.monitoring.providers import (
    OpenAIProvider,
    OpenRouterProvider,
)
 
# OpenAI - validates API key
openai_provider = OpenAIProvider()
result = openai_provider.check_balance(force=True)
 
if result.status == "valid":
    print("OpenAI API key is valid")
else:
    print(f"OpenAI error: {result.error}")
 
# OpenRouter - checks real prepaid balance
openrouter_provider = OpenRouterProvider()
result = openrouter_provider.check_balance(force=True)
 
print(f"Balance: ${result.balance}")
print(f"Total credits: ${result.limit}")
print(f"Total usage: ${result.usage}")

Force Fresh Check

# Bypass cache
balances = checker.check_all_balances(force=True)
 
# Force notifications
results = notifier.check_all_and_notify(balances, force=True)

How It Works

Provider Architecture

The module uses a modular provider architecture:

django_llm/monitoring/
├── providers/
│   ├── base.py          # BaseLLMProvider (caching, error handling)
│   ├── openai.py        # OpenAIProvider (key validation)
│   └── openrouter.py    # OpenRouterProvider (balance check)
├── models.py            # Pydantic v2 models
├── balance_checker.py   # Main checker
└── notifier.py          # Notifications

Balance Checking Flow

  1. BalanceChecker iterates through all providers
  2. Each Provider (OpenAI, OpenRouter) implements:
    • get_provider_name() - Returns provider name
    • _fetch_balance() - Fetches balance from API
    • check_balance() - Adds caching and error handling
  3. Returns BalanceResponse (Pydantic model)

Pydantic Models

All responses use type-safe Pydantic models:

from typing import Optional, Literal
from pydantic import BaseModel, Field
 
class BalanceResponse(BaseModel):
    balance: Optional[float] = None      # USD balance (None when unavailable, e.g. OpenAI)
    currency: str = "usd"                # Always "usd"
    usage: Optional[float] = None        # Total spend to date (OpenRouter only)
    limit: Optional[float] = None        # Total credits purchased (OpenRouter only)
    status: Optional[Literal["valid", "invalid", "error", "unavailable"]] = None
    note: Optional[str] = None           # Human-readable extra info
    error: Optional[str] = None          # Error message if the balance check failed

API Endpoints

OpenAI

Status: No public balance API available

Implementation: Validate API key via test request

from openai import OpenAI
 
client = OpenAI(api_key=api_key)
models = client.models.list()  # Validates key
 
# Returns:
BalanceResponse(
    balance=None,  # Not available
    status="valid",
    note="Balance check not available via API. Check manually at: ..."
)

Manual Check: https://platform.openai.com/settings/organization/billing/overview 

OpenRouter

Endpoint: GET https://openrouter.ai/api/v1/credits

Headers:

Authorization: Bearer YOUR_OPENROUTER_KEY

Response:

{
  "data": {
    "total_credits": 217.50,
    "total_usage": 216.81
  }
}

Calculated Balance:

balance = total_credits - total_usage  # $0.69 (real prepaid balance)

Notification Logic

  1. Threshold Check - Compare balance against $10 (warning) and $5 (critical)
  2. Cache Check - Skip if notification sent in last 24 hours (unless force=True)
  3. Multi-Channel Send - Via send_admin_notification():
    • Email: send_admin_email()
    • Telegram: send_telegram_message() → Async queue
  4. Mark Sent - Cache for 24 hours to prevent spam

Telegram Async Queue

Telegram notifications use an asynchronous queue with rate limiting:

  • Singleton Queue: Global instance shared across Django processes
  • Worker Thread: Background daemon processes messages continuously
  • Rate Limiting: Max 20 messages/second (0.05s delay)
  • Priority Support: CRITICAL (1) > HIGH (2) > NORMAL (3) > LOW (4)
  • Management Command Wait: 2-second delay ensures delivery before exit

Why the Delay?

Management commands exit immediately after execution. The 2-second wait in check_llm_balance ensures the Telegram worker thread has time to process the queue before the Python process terminates.

Dashboard API

The module provides a REST API endpoint for dashboard integration:

GET /cfg/dashboard/api/metrics/llm-balances/

Response:

{
  "name": "LLM Provider Balances",
  "description": "API key status and account balances for LLM providers",
  "status": "critical",
  "items": [
    {
      "provider": "openai",
      "provider_display": "Openai",
      "balance": null,
      "currency": "usd",
      "status": "valid",
      "status_level": "info",
      "note": "Balance check not available via API..."
    },
    {
      "provider": "openrouter",
      "provider_display": "Openrouter",
      "balance": 0.69,
      "currency": "usd",
      "usage": 216.81,
      "limit": 217.50,
      "status": null,
      "status_level": "critical"
    }
  ],
  "summary": {
    "total_providers": 2,
    "total_balance": 0.69,
    "providers_with_errors": 0,
    "providers_critical": 1
  }
}

Caching

Balance Cache

  • TTL: 1 hour (3600 seconds)
  • Key Pattern: llm_monitoring:openai_balance, llm_monitoring:openrouter_balance
  • Bypass: Use --force flag or force=True parameter

Notification Cache

  • TTL: 24 hours (86400 seconds)
  • Key Pattern: llm_monitoring:notification_sent:{provider}:{level}
  • Purpose: Prevent spam (max 1 notification per day per provider per level)
  • Bypass: Use --force-notify flag or force=True parameter

Troubleshooting

Notifications Not Arriving in Telegram

Symptoms: Command shows “Sent notification” but no Telegram message

Solutions:

  1. Check Bot Token & Chat ID:

    python manage.py shell -c "
from django_cfg.core.config import get_current_config
config = get_current_config()
print('Telegram config:', config.telegram)
"

  2. Test Direct Send:

    python manage.py shell -c "
from django_cfg.modules.django_telegram import send_telegram_message
result = send_telegram_message('Test message')
print('Sent:', result)
import time; time.sleep(2)  # Wait for queue
"

  3. Verify Bot Has Access:

    • For groups: Bot must be added as member
    • For channels: Bot must be admin
    • Check bot is not blocked

Notifications Cached (Not Sending)

Symptoms: Balance is low but no notifications sent

Cause: Notifications were sent in last 24 hours

Solutions:

  1. Force Send:

    python manage.py check_llm_balance --force-notify

  2. Clear Cache Manually:

    from django.core.cache import cache
for provider in ['openai', 'openrouter']:
    for level in ['warning', 'critical', 'api_error']:
        cache.delete(f'llm_monitoring:notification_sent:{provider}:{level}')

OpenAI Shows $0.00 Balance

This is expected! OpenAI does not provide a public balance API.

The module validates the API key and returns:

BalanceResponse(
    balance=None,  # Not $0.00, but None
    status="valid",
    note="Balance check not available via API..."
)

To check real balance: Visit https://platform.openai.com/settings/organization/billing/overview 

API Key Errors

Error: Incorrect API key provided

Solutions:

  1. Verify key in .env file
  2. Check key format (OpenAI: sk-proj-..., OpenRouter: sk-or-v1-...)
  3. Ensure key has not expired
  4. Test key manually via provider’s dashboard

Security

API Key Security

LLM provider API keys are high-value credentials:

  • ❌ Never commit to version control
  • ✅ Store in .env.secrets (gitignored)
  • ✅ Use environment variables in production
  • ✅ Rotate keys regularly
  • ✅ Monitor usage for anomalies
  • ✅ Use Pydantic SecretStr for sensitive values

Best Practices

Production Deployment

  1. Use Secrets File - Store API keys in .env.secrets (git-ignored)
  2. Set Up Cron - Run checks hourly with proper logging
  3. Monitor Notifications - Test Telegram/email delivery monthly
  4. Configure Thresholds - Adjust based on your usage patterns
  5. Enable Logging - Set up log aggregation (Sentry, CloudWatch)

Cron Setup

# /etc/cron.d/llm-monitoring
SHELL=/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
 
# Check LLM balances every hour
0 * * * * app cd /app && /app/.venv/bin/python manage.py check_llm_balance >> /var/log/llm-balance.log 2>&1

Monitoring Recommendations

  1. Set Realistic Thresholds - Adjust based on your usage patterns
  2. Monitor Multiple Channels - Enable both email and Telegram
  3. Test Regularly - Run manual checks to verify setup
  4. Review Logs - Check for API errors or notification failures
  5. Add Buffer - Top up accounts before hitting thresholds
  6. Track Usage Trends - Monitor spending patterns over time

Adding New Providers

The modular architecture makes it easy to add new providers.

Example: Adding Anthropic

1. Create Provider Class

# providers/anthropic.py
import httpx
from .base import BaseLLMProvider
from ..models import BalanceResponse
 
class AnthropicProvider(BaseLLMProvider):
    """Anthropic Claude balance checker."""
 
    API_URL = "https://api.anthropic.com/v1/organization/balance"
 
    def get_provider_name(self) -> str:
        return "anthropic"
 
    def _fetch_balance(self) -> BalanceResponse:
        """Fetch balance from Anthropic API."""
        from django_cfg.core.config import get_current_config
 
        config = get_current_config()
        api_key = config.api_keys.anthropic
 
        if hasattr(api_key, 'get_secret_value'):
            api_key = api_key.get_secret_value()
 
        client = httpx.Client()
        response = client.get(
            self.API_URL,
            headers={
                "x-api-key": api_key,
                "anthropic-version": "2023-06-01"
            },
            timeout=10.0
        )
        response.raise_for_status()
        data = response.json()
 
        return BalanceResponse(
            balance=data.get("balance"),
            currency="usd",
            limit=data.get("limit"),
            usage=data.get("usage")
        )

2. Update Balance Checker

# balance_checker.py
from .providers import OpenAIProvider, OpenRouterProvider, AnthropicProvider
 
class BalanceChecker:
    def check_all_balances(self, force: bool = False) -> Dict[str, BalanceResponse]:
        providers = {
            "openai": OpenAIProvider(),
            "openrouter": OpenRouterProvider(),
            "anthropic": AnthropicProvider(),  # New
        }
 
        balances = {}
        for name, provider in providers.items():
            balances[name] = provider.check_balance(force=force)
 
        return balances

3. Add API Key to Config

# django_cfg/core/base/config_model.py
class ApiKeys(BaseModel):
    openai: SecretStr = Field(default=SecretStr(""))
    openrouter: SecretStr = Field(default=SecretStr(""))
    anthropic: SecretStr = Field(default=SecretStr(""))  # New

Done! The new provider will be automatically checked.

See Also

Last updated on
CachingArchitecture

Django-CFG documentation — a modern, type-safe Django framework with Pydantic v2 configuration.

Documentation

Frontend

Resources

Company

© 2026 Django-CFG. MIT Licensed.GitHub ↗