Token Meter for Free AI Model Tiers: A Practical Guide

Keep the job deliberately small

Start by writing down the contract before you touch code. A tiny meter can do three things:

1. Forward a request to one upstream model endpoint.
2. Record the provider's prompt and completion token counts in a JSON Lines ledger.
3. Reject new requests when the day's recorded total reaches a configured cap.

Keeping it narrow makes it easier to test. You're not building a billing platform, a general-purpose API gateway, or a per-user analytics system - you're trying to answer one question: whether this application is drifting toward the free-tier limit.

Make the data easy to inspect. A single ledger row per call can include the date, timestamp, prompt tokens, completion tokens, and total tokens. JSONL works well because each line is a standalone JSON object. You can open it in a text editor, parse it with Python, or load it into a spreadsheet.

Treat "free" as a changing constraint

A free-tier allowance is a product term, not a promise. Providers can change quotas, model availability, rate limits, and the rules around what counts as "free." A local meter helps you manage your app's behavior, but it can't guarantee the upstream will stay free - or even available.

This matters when you move from "experiment" to "feature." A prototype depending on a free endpoint is often fine as an experiment. A customer-facing feature that quietly assumes the same allowance will remain forever is a different risk. Before you build on a free tier, check the current terms and decide what your application should do when the quota is exhausted: fail clearly, switch models, queue work, or hand off to an operator.

Make the test path deterministic

One of the most valuable features in a small proxy is a stub mode. Instead of calling a live endpoint, return a fixed response with known usage - say, 12 prompt tokens and 8 completion tokens. Then you can test the proxy without network access, provider credentials, or the risk of consuming the free quota.

Run the local server with the stub enabled and a separate ledger file. Send a request with curl, then check both the response and the ledger. The expected result should be repeatable: the response includes the stub content and the ledger records a total of 20 tokens. Send a second request and confirm the ledger now has two rows.

A deterministic test is better than a quick manual check because it gives you a baseline. If the count changes, your test should fail for a reason you can investigate - not because the provider changed something overnight.

Keep the ledger readable, but protect it

A JSONL ledger is a practical first storage option for one process and low volume. It's also easy to inspect when something breaks. But a local file isn't automatically durable or safe.

On some hosting platforms, the filesystem can be replaced when the service restarts. If the ledger matters, check how your platform persists storage, and use persistent storage or an external database if needed.

Also, don't write credentials, full prompts, or sensitive user data to the ledger by default. Token counts, timestamps, and a request identifier are often enough to diagnose quota usage. Keeping full request/response bodies can create privacy and security obligations your meter wasn't built to handle.

If you run multiple processes or instances, you'll also need to think beyond "a thread lock." A lock can protect writes inside one process, but it won't coordinate separate workers or servers. At that point, a single JSONL file usually stops being the right tool.

Missing usage must not become zero usage

Update the stub response so it has no usage field. A naive implementation might default missing prompt and completion counts to zero, return success, and write a "0 tokens" row. That's worse than an obvious error - your ledger now looks like proof the request cost nothing, and your cap may never fire.

Treat usage data as untrusted input. Decide what the meter should do when usage is absent or malformed:

• Reject or flag the response. This is the safest option if enforcing the quota matters more than returning an answer.
• Estimate and label it. You can use a character-based estimate to keep a prototype running, but the ledger must mark it as estimated so nobody mistakes it for provider-reported usage.
• Allow the response but stop trusting the cap. This can be acceptable for a non-critical demo, as long as you log and surface that "missing usage" situation clearly.

Whichever policy you choose, don't silently turn missing usage into "trustworthy zero." Then compare your ledger to the provider dashboard on a regular schedule. If numbers diverge, figure out whether it's missing usage, cached inputs, different counting rules, or requests that bypassed the proxy.

Test the cap, including the boundary

Set the cap below your known stub cost. If the first call records 20 tokens and the cap is 10, the next call should be rejected with a clear response like HTTP 429, including fields that show the recorded spend and the configured cap.

Then test the exact boundary. For example, "reject when spent is greater than or equal to the cap" is different from "reject when spent is greater than the cap." Decide what your cap means:

• Do not start a request once you've reached this value, or
• Do not allow total usage to go above this value.

Those two policies behave differently under load.

Concurrency makes it even trickier. Imagine the ledger shows 20 tokens, the cap is 25, and two requests arrive at the same time. If both check the ledger before either one records usage, both can pass. If both then cost 10 tokens, you end up at 40.

A lock around file writes doesn't fix the logic race. You have to coordinate the check + admission decision. One approach is to serialize the whole check-call-record sequence. That's simple, but it reduces throughput because you hold the lock during the network request.

Another approach is to reserve capacity before sending the call, then reconcile with actual usage later. Reservations scale better, but they add state and require careful handling for failed or timed-out requests.

For a single-user script, a small overshoot might be an acceptable trade-off. For a strict quota, you need the lock or reservation to cover the full admission decision - not just the file write.

Test upstream errors and retries

Since the proxy sits in the request path, its error handling becomes part of your app's behavior. Test timeouts, non-2xx status codes, invalid JSON, and responses that don't include usage. Decide which upstream errors you pass through, and which you translate into a stable error format your client can understand.

Retries deserve special attention. Retrying every non-200 response can amplify an outage - and it can burn quota if the request actually reached the provider but the response got lost. Use a bounded retry policy, add backoff, and clearly separate "may be transient" from "don't retry."

A meter can show what traffic you generated, but it can't magically make an unbounded retry loop safe.

For every test, verify both sides of the contract: what the client receives, and what the ledger records. An error response shouldn't create a normal successful usage row unless you intentionally chose to count it that way. On the flip side, if the provider may charge a request that times out before returning usage, your ledger can't know the final charge with certainty. In that case, mark the uncertainty instead of pretending you have a precise number.

Know when the small design is no longer enough

A standard-library proxy is useful because it does very little. That also means it has clear limits:

• No streaming support: buffering the whole response won't work well for clients that expect tokens as a stream. A streaming version has to preserve the stream and still capture usage, which may arrive only at the end - or not at all.
• No per-user quota: one global daily ceiling can't tell you which user or client caused the spend.
• Limited persistence and concurrency: a JSONL file and an in-process lock aren't shared accounting across multiple workers or hosts.
• No guarantee of exact accounting: your meter can only record what the provider returns. Missing fields, cached-token rules, and provider-side adjustments can make local totals differ from billed usage.
• A new point of failure: if the proxy is down, the dependent clients are down too. In production you'll need monitoring, health checks, and a defined recovery path.

If your provider already gives you reliable, sufficiently detailed spend controls - and your app doesn't need an extra hop - skip the proxy. For an offline batch job, a usage export or a spreadsheet might be simpler. And if your gateway rejects the proxy's request format, validate compatibility before you route important traffic through it.

In the end, this is a trade-off between control and operational cost. For one developer testing a free model, a tiny meter can be enough. For a team, a customer-facing product, or a hard spend limit, you'll want a shared store, authentication, concurrency-safe admission control, and monitoring - or you'll use the provider's built-in controls where they meet your needs.