Skip to content

llmfy_usage_tracker

llmfy.llmfy_core.usage.usage_tracker

LLMFY_USAGE_TRACKER_VAR = ContextVar('LLMFY_USAGE_TRACKER', default=None) module-attribute

llmfy_usage_tracker(openai_pricing=None, bedrock_pricing=None, googleai_pricing=None)

LLMfy usage tracker.

Use this to track token usage all provider.

Example: 

with llmfy_usage_tracker() as usage:
        result = llm.generate(messages)
        ...
        print(usage)

[OPENAI] Price per 1M tokens for different models (USD): - https://platform.openai.com/docs/pricing

[BEDROCK] Price per 1K tokens for different models (USD): - https://aws.amazon.com/bedrock/pricing/ - https://aws.amazon.com/bedrock/pricing/ - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html

Parameters:

Name Type Description Default
openai_pricing Optional[Dict[str, Any]]

OpenAI Pricing dictionary source. Defaults to None. If None then use default pricing from this dependency.

Example pricing structure: 

{
    "gpt-4o": {
        "input": 2.50,
        "output": 10.00
    },
    "gpt-4o-mini": {
        "input": 0.15,
        "output": 0.60
    },
    "gpt-3.5-turbo": {
        "input": 0.05,
        "output": 1.50
    }
}

None
bedrock_pricing Optional[Dict[str, Any]]

Bedrock Pricing dictionary source. Defaults to None. If None then use default pricing from this dependency.

Example pricing structure: 

{
    "anthropic.claude-3-5-sonnet-20240620-v1:0": {
        "us-east-1": {
            "region": "US East (N. Virginia)",
            "input": 0.003,
            "output": 0.015,
        },
        "us-west-2": {
            "region": "US West (Oregon)",
            "input": 0.003,
            "output": 0.015,
        },
    },
    "anthropic.claude-3-5-sonnet-20241022-v2:0": {
        "us-east-1": {
            "region": "US East (N. Virginia)",
            "input": 0.003,
            "output": 0.015,
        },
        "us-west-2": {
            "region": "US West (Oregon)",
            "input": 0.003,
            "output": 0.015,
        },
    }
}

None

Yields:

Name Type Description
OpenAIUsage

OpenAI usage accumulation.
Source code in llmfy/llmfy_core/usage/usage_tracker.py 
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
@contextmanager
def llmfy_usage_tracker(
    openai_pricing: Optional[Dict[str, Any]] = None,
    bedrock_pricing: Optional[Dict[str, Any]] = None,
    googleai_pricing: Optional[Dict[str, Any]] = None,
):
    """LLMfy usage tracker.

    Use this to track token usage all provider.

    Example:
    ```python
    with llmfy_usage_tracker() as usage:
            result = llm.generate(messages)
            ...
            print(usage)
    ```

    [OPENAI] Price per 1M tokens for different models (USD):
        - https://platform.openai.com/docs/pricing

    [BEDROCK] Price per 1K tokens for different models (USD):
        - https://aws.amazon.com/bedrock/pricing/
        - https://aws.amazon.com/bedrock/pricing/
        - https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/Concepts.RegionsAndAvailabilityZones.html

    Args:
        openai_pricing (Optional[Dict[str, Any]], optional): OpenAI Pricing dictionary source. Defaults to None.
            If None then use default pricing from this dependency.

            Example pricing structure:
            ```
            {
                "gpt-4o": {
                    "input": 2.50,
                    "output": 10.00
                },
                "gpt-4o-mini": {
                    "input": 0.15,
                    "output": 0.60
                },
                "gpt-3.5-turbo": {
                    "input": 0.05,
                    "output": 1.50
                }
            }
            ```

        bedrock_pricing (Optional[Dict[str, Any]], optional): Bedrock Pricing dictionary source. Defaults to None.
            If None then use default pricing from this dependency.

            Example pricing structure:
            ```
            {
                "anthropic.claude-3-5-sonnet-20240620-v1:0": {
                    "us-east-1": {
                        "region": "US East (N. Virginia)",
                        "input": 0.003,
                        "output": 0.015,
                    },
                    "us-west-2": {
                        "region": "US West (Oregon)",
                        "input": 0.003,
                        "output": 0.015,
                    },
                },
                "anthropic.claude-3-5-sonnet-20241022-v2:0": {
                    "us-east-1": {
                        "region": "US East (N. Virginia)",
                        "input": 0.003,
                        "output": 0.015,
                    },
                    "us-west-2": {
                        "region": "US West (Oregon)",
                        "input": 0.003,
                        "output": 0.015,
                    },
                }
            }
            ```

    Yields:
            OpenAIUsage: OpenAI usage accumulation.
    """

    usage_tracker = LLMfyUsage(
        openai_pricing=openai_pricing,
        bedrock_pricing=bedrock_pricing,
        googleai_pricing=googleai_pricing,
    )
    LLMFY_USAGE_TRACKER_VAR.set(usage_tracker)  # Store usage_tracker it in the context
    try:
        yield usage_tracker  # Expose tracker to the context
    finally:
        pass