LanguageModelRateLimitingPlugin

Simulates token-based rate limiting for language model APIs by tracking prompt and completion token consumption within configurable time windows.

Plugin instance definition

{
  "name": "LanguageModelRateLimitingPlugin",
  "enabled": true,
  "pluginPath": "~appFolder/plugins/DevProxy.Plugins.dll",
  "configSection": "languageModelRateLimitingPlugin",
  "urlsToWatch": [
    "https://api.openai.com/*",
    "http://localhost:11434/*"
  ]
}

Configuration example

{
  "languageModelRateLimitingPlugin": {
    "$schema": "https://raw.githubusercontent.com/dotnet/dev-proxy/main/schemas/v1.0.0/languagemodelratelimitingplugin.schema.json",
    "promptTokenLimit": 5000,
    "completionTokenLimit": 5000,
    "resetTimeWindowSeconds": 60,
    "whenLimitExceeded": "Throttle",
    "headerRetryAfter": "retry-after"
  }
}

Configuration properties

Custom response configuration

When whenLimitExceeded is set to Custom, you can define a custom response in a separate JSON file:

{
  "$schema": "https://raw.githubusercontent.com/dotnet/dev-proxy/main/schemas/v1.0.0/languagemodelratelimitingplugin.customresponsefile.schema.json",
  "statusCode": 429,
  "headers": [
    {
      "name": "retry-after",
      "value": "@dynamic"
    },
    {
      "name": "content-type",
      "value": "application/json"
    }
  ],
  "body": {
    "error": {
      "message": "You have exceeded your token quota. Please wait before making additional requests.",
      "type": "insufficient_quota",
      "code": "token_quota_exceeded"
    }
  }
}

Custom response properties

How it works

The LanguageModelRateLimitingPlugin works by:

1. Intercepting OpenAI API requests: Monitors POST requests to configured URLs that contain OpenAI-compatible request bodies
2. Tracking token consumption: Parses responses to extract prompt_tokens and completion_tokens from the usage section
3. Enforcing limits: Maintains running totals of consumed tokens within the configured time window
4. Providing throttling responses: When limits are exceeded, returns either standard throttling responses or custom responses

Supported request types

The plugin supports both OpenAI completion and chat completion requests:

• Completion requests: Requests with a prompt property
• Chat completion requests: Requests with a messages property

Token tracking

Token consumption is tracked separately for:

• Prompt tokens: Input tokens consumed by the request
• Completion tokens: Output tokens generated by the response

When either limit is exceeded, subsequent requests are throttled until the time window resets.

Time window behavior

• Token limits reset after the configured resetTimeWindowSeconds
• The reset timer starts when the first request is processed
• When a time window expires, both prompt and completion token counters reset to their configured limits

Default throttling response

When whenLimitExceeded is set to Throttle, the plugin returns a standard OpenAI-compatible error response:

{
  "error": {
    "message": "You exceeded your current quota, please check your plan and billing details.",
    "type": "insufficient_quota",
    "param": null,
    "code": "insufficient_quota"
  }
}

The response includes:

• HTTP status code: 429 Too Many Requests
• retry-after header with seconds until token limits reset
• CORS headers when the original request includes an Origin header

Use cases

The LanguageModelRateLimitingPlugin is useful for:

• Testing token-based rate limiting: Simulate how your application behaves when language model providers enforce token quotas
• Development cost simulation: Understand token consumption patterns during development before hitting real API limits
• Resilience testing: Verify that your application properly handles token limit errors and implements appropriate retry logic
• Local LLM testing: Test token limiting scenarios with local language models (like Ollama) that don't enforce their own limits

Example scenarios

Scenario 1: Basic token limiting

{
  "languageModelRateLimitingPlugin": {
    "promptTokenLimit": 1000,
    "completionTokenLimit": 500,
    "resetTimeWindowSeconds": 300
  }
}

This configuration allows up to 1,000 prompt tokens and 500 completion tokens within a 5-minute window.

Scenario 2: Custom error responses

{
  "languageModelRateLimitingPlugin": {
    "promptTokenLimit": 2000,
    "completionTokenLimit": 1000,
    "resetTimeWindowSeconds": 60,
    "whenLimitExceeded": "Custom",
    "customResponseFile": "custom-token-error.json"
  }
}

This configuration uses a custom response file to provide specialized error messages when token limits are exceeded.

Next step