User Metrics

Track costs, usage, and performance per user to understand user behavior and optimize spend

What are User Metrics?

User metrics let you track LLM usage at the user level. Monitor which users are driving costs, identify power users, analyze usage patterns, and implement usage-based billing.

Why Track Users?

Cost Attribution

Know exactly how much each user costs your application

Usage-Based Billing

Charge users based on their actual LLM usage

Rate Limiting

Set per-user rate limits and quotas

User Analytics

Understand user engagement and behavior patterns

Support & Debugging

Debug issues for specific users

Churn Prevention

Identify users with issues or low engagement

Adding User IDs

Track users with the Helicone-User-Id header:

Helicone-User-Id · string · required

Unique identifier for the user making the request. Can be any string that uniquely identifies your users (UUID, database ID, email hash, etc.).

Basic Example

Python

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_OPENAI_KEY",
    base_url="https://oai.helicone.ai/v1",
    default_headers={
        "Helicone-Auth": "Bearer YOUR_HELICONE_KEY"
    }
)

# Add user ID to request
response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_headers={
        "Helicone-User-Id": "user_123456"  # Your user's ID
    }
)

What Metrics are Tracked?

For each user, Helicone tracks:

Usage Metrics

• Total Requests: Number of LLM requests made
• Total Tokens: Prompt tokens + completion tokens
• Request Frequency: Requests per day/week/month
• Active Days: Days with at least one request

Cost Metrics

• Total Cost: Cumulative spend in USD
• Average Cost per Request: Mean cost across requests
• Cost Trend: Spending over time
• Most Expensive Requests: Highest cost requests

Performance Metrics

• Average Latency: Mean response time
• Success Rate: Percentage of successful requests
• Error Rate: Percentage of failed requests
• Models Used: Which models the user accesses

Engagement Metrics

• Session Count: Number of sessions
• Requests per Session: Average session length
• Last Active: Most recent request timestamp
• First Seen: First request timestamp

Viewing User Metrics

Access user analytics in the dashboard:

1. Navigate to Users

   Go to the Users page in your Helicone dashboard

2. View User List

   See all users with their key metrics: total cost, request count, last active

3. Click a User

   View detailed metrics, request history, and trends for a specific user

4. Filter Requests

   Click “View Requests” to see all requests from that user

Filtering by User

Find requests from specific users:

Dashboard Filtering

1. Go to Requests page
2. Add filter: User ID equals user_123456
3. See all requests from that user

API Filtering

const response = await fetch('https://api.helicone.ai/v1/request/query', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer YOUR_API_KEY',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    filter: {
      request_response_rmt: {
        user_id: {
          equals: "user_123456"
        }
      }
    },
    limit: 100,
    sort: {
      created_at: "desc"
    }
  })
});

User Context with Properties

Combine user IDs with custom properties for richer analytics:

response = client.chat.completions.create(
    model="gpt-4",
    messages=[{"role": "user", "content": "Hello!"}],
    extra_headers={
        # User identification
        "Helicone-User-Id": user_id,
        
        # User context
        "Helicone-Property-User-Tier": "premium",
        "Helicone-Property-Organization": org_id,
        "Helicone-Property-User-Segment": "power-user",
        "Helicone-Property-Plan": "annual"
    }
)

This enables analysis like:

• Cost per subscription tier
• Usage patterns by user segment
• Performance differences across plans

Use Cases

Usage-Based Billing

Charge users based on their actual consumption:

def calculate_user_bill(user_id: str, start_date: datetime, end_date: datetime):
    # Query user's requests for billing period
    requests = query_helicone_api(
        filter={"user_id": {"equals": user_id}},
        timeFilter={
            "start": start_date.timestamp() * 1000,
            "end": end_date.timestamp() * 1000
        }
    )
    
    # Calculate total cost
    total_cost = sum(r["cost"] for r in requests)
    total_tokens = sum(r["total_tokens"] for r in requests)
    
    # Add markup and generate invoice
    markup = 1.5  # 50% markup
    bill_amount = total_cost * markup
    
    return {
        "user_id": user_id,
        "period": f"{start_date} to {end_date}",
        "total_tokens": total_tokens,
        "base_cost": total_cost,
        "bill_amount": bill_amount,
        "request_count": len(requests)
    }

Per-User Rate Limiting

Implement custom rate limits per user:

import { RateLimiter } from 'your-rate-limiter';

const userRateLimits = {
  'free': 100,      // 100 requests/day
  'pro': 1000,      // 1000 requests/day
  'enterprise': -1  // unlimited
};

async function makeUserRequest(userId: string, userTier: string, prompt: string) {
  // Check rate limit
  const limit = userRateLimits[userTier];
  if (limit > 0) {
    const usage = await getUserDailyUsage(userId);
    if (usage >= limit) {
      throw new Error('Rate limit exceeded');
    }
  }
  
  // Make request with user ID
  const response = await client.chat.completions.create(
    {
      model: 'gpt-4',
      messages: [{ role: 'user', content: prompt }]
    },
    {
      headers: {
        'Helicone-User-Id': userId,
        'Helicone-Property-User-Tier': userTier
      }
    }
  );
  
  return response;
}

Power User Identification

Find your most active users:

# Query top users by cost
top_users = query_helicone_api(
    endpoint="/users/query",
    body={
        "sort": {"total_cost": "desc"},
        "limit": 10
    }
)

for user in top_users:
    print(f"User {user['user_id']}:")
    print(f"  Total Cost: ${user['total_cost']:.2f}")
    print(f"  Requests: {user['request_count']}")
    print(f"  Avg Cost: ${user['avg_cost']:.4f}")
    
    # Consider upgrading to higher tier
    if user['total_cost'] > 100 and user['tier'] == 'free':
        suggest_upgrade(user['user_id'])

User Support

Debug issues for specific users:

// Find failed requests for a user
const response = await fetch('https://api.helicone.ai/v1/request/query', {
  method: 'POST',
  headers: {
    'Authorization': 'Bearer YOUR_API_KEY',
    'Content-Type': 'application/json'
  },
  body: JSON.stringify({
    filter: {
      left: {
        request_response_rmt: {
          user_id: { equals: "user_123456" }
        }
      },
      operator: "and",
      right: {
        request_response_rmt: {
          status: { not_equals: 200 }
        }
      }
    },
    limit: 50
  })
});

// Analyze error patterns
const errors = await response.json();
console.log(`User has ${errors.length} failed requests`);

Churn Prevention

Identify users at risk:

from datetime import datetime, timedelta

def find_at_risk_users():
    all_users = get_all_users()
    at_risk = []
    
    for user in all_users:
        last_active = user['last_request_timestamp']
        days_inactive = (datetime.now() - last_active).days
        
        # Flag users inactive for 7+ days
        if days_inactive >= 7:
            at_risk.append({
                'user_id': user['user_id'],
                'days_inactive': days_inactive,
                'total_cost': user['total_cost'],
                'tier': user['tier']
            })
    
    return at_risk

# Send re-engagement campaigns
at_risk_users = find_at_risk_users()
for user in at_risk_users:
    send_reengagement_email(user['user_id'])

User ID Best Practices

Use Consistent IDs

Use the same identifier across your application:

• Database primary key
• UUID from your auth system
• Hashed email (for privacy)

Don’t use different IDs for the same user.

Hash Sensitive Data

If using email or username, hash it first:

import hashlib
user_id = hashlib.sha256(user_email.encode()).hexdigest()

Handle Anonymous Users

For logged-out users, use session IDs or device IDs:

user_id = logged_in_user_id or f"anonymous_{session_id}"

Don't Store PII

Never use raw email addresses, phone numbers, or other PII as user IDs. Use opaque identifiers.

Combining with Sessions

Track both user identity and conversation context:

import uuid

session_id = str(uuid.uuid4())

# Track user across multiple messages in a session
for message in conversation:
    response = client.chat.completions.create(
        model="gpt-4",
        messages=conversation,
        extra_headers={
            "Helicone-User-Id": user_id,
            "Helicone-Session-Id": session_id,
            "Helicone-Session-Name": "Customer Support Chat"
        }
    )

This enables analysis like:

• How many sessions per user?
• What’s the average session length per user?
• Which users have the longest conversations?

User Privacy Considerations

Protect User Privacy

When tracking users:

• Use opaque identifiers (UUIDs, hashed IDs)
• Don’t include PII in properties
• Comply with GDPR/CCPA requirements
• Provide users ability to export/delete their data
• Document your data retention policies

Next Steps

Custom Properties

Add user context with properties

Session Tracking

Track per-user sessions

Request Logging

View individual user requests

Rate Limiting

Set per-user rate limits