LLM observability installation (beta)

🚧 Note: LLM observability is currently considered in beta. To access it, enable the feature preview in your PostHog account.

We are keen to gather as much feedback as possible so if you try this out please let us know. You can email tim@posthog.com, send feedback via the in-app support panel, or use one of our other support options.

LLM observability enables you to capture and analyze LLM usage and performance data. Specifically, it captures:

• Input and output content and tokens
• Latency
• Model
• Traces
• Cost of each generation

All of this data is captured into PostHog to be used in insights, dashboards, alerts, and more. LLM events are charged at the same rate as other PostHog events, and so in practice are roughly an order of magnitude cheaper than most other LLM providers.

Observability installation

Setting up observability starts with installing the PostHog Python SDK.

pip install posthog

The rest of the setup depends on the LLM platform you're using. These SDKs do not proxy your calls, they only fire off an async call to PostHog in the background to send the data.

pip install openai

from posthog.ai.openai import OpenAI
import posthog

posthog.project_api_key = "<ph_project_api_key>"
posthog.host = "https://us.i.posthog.com"

client = OpenAI(
    api_key="your_openai_api_key",
    posthog_client=posthog
)

response = client.chat.completions.create(
    model="gpt-3.5-turbo",
    messages=[
        {"role": "user", "content": "Tell me a fun fact about hedgehogs"}
    ],
    posthog_distinct_id="user_123", # optional
    posthog_trace_id="trace_123", # optional
    posthog_properties={"conversation_id": "abc123", "paid": True} # optional
    posthog_groups={"company": "company_id_in_your_db"}  # optional 
    posthog_privacy_mode=False # optional
)

print(response.choices[0].message.content)

response = client.embeddings.create(
    input="The quick brown fox",
    model="text-embedding-3-small",
    posthog_distinct_id="user_123", # optional
    posthog_trace_id="trace_123",   # optional
    posthog_properties={"key": "value"} # optional
    posthog_groups={"company": "company_id_in_your_db"}  # optional 
    posthog_privacy_mode=False # optional
)

Privacy mode

To avoid storing potentially sensitive prompt and completion data, you can enable privacy mode. This excludes the $ai_input and $ai_output properties from being captured.

This can be done either by setting the privacy_mode config option in the Python SDK like this:

import posthog

posthog.project_api_key = "<ph_project_api_key>"
posthog.host = "https://us.i.posthog.com"
posthog.privacy_mode = True

It can also be on at the request level by including setting the privacy_mode parameter to True in the request. The exact setup depends on the LLM platform you're using:

client.chat.completions.create(
    messages=[...],
    posthog_privacy_mode=True
)