Local LLM Inference: What Four CUDA Streams Reveal12 min read2,424 words

Local LLM Inference: What Four CUDA Streams Reveal

See what TensorFold’s Spark-X2.6-4B CUDA tests reveal about concurrent throughput, long-context latency, GPU memory, and the limits of local LLM benchmarks.

local LLM inferenceCUDA LLM servingLLM concurrent throughputlong-context inferenceGPU inference benchmarks
Local LLM Inference: What Four CUDA Streams Reveal

Local LLM Inference: What Four CUDA Streams Reveal

One GPU, Four Requests: What TensorFold's CUDA Test Actually Shows

A local model can feel fast when one person is using it. Once you have several requests arriving at the same time, the GPU has to split its attention. At that point, users care about waiting time just as much as output speed - and a long prompt can make the first response feel slow even if generation later runs quickly.

TensorFold's CUDA test for Spark-X2.6-4B is a handy snapshot of those trade-offs. On an RTX 5090, the author reports up to four concurrent streams and measures generation throughput after a long input. A separate RTX 4090 test looks at how input length affects time to first token. The measurements are useful, but they're not the same question - and they don't remove the usual limits of benchmarking.

This article walks through what the results do and do not show, how to interpret aggregate tokens per second, why long-context latency needs its own metric, and what to measure before you pick a local serving setup. The goal isn't to crown a "winner." It's to help you check whether these numbers match your workload.

What do four concurrent streams actually show?

The headline result is straightforward: one GPU can produce much more combined output when it runs multiple requests at once. On an RTX 5090 running Linux, TensorFold reports:

  • 191-204 tokens per second for one stream
  • 375-389 combined tokens per second for two streams
  • 716-729 combined tokens per second for four streams

That's about the total generation rate across concurrent requests - not about what each person experiences.

The four-stream number is the sum across all four requests. If you divide it by four, you get an average of roughly 179-182 tokens per second per stream, but that's just arithmetic on the total. The report doesn't tell you how evenly the work gets distributed, or how request latency changes while the GPU is under load.

Capacity and experience are different measures

For planning, aggregate throughput can answer a capacity question: "How much total output can this server produce when work is available?" What it doesn't answer is whether users wait longer before the first token, whether some requests get stuck behind others, or whether latency stays steady during a busy period.

Those details matter depending on how you'll use the model. If you're running batch jobs - say, summarizing lots of internal documents - you might care more about total throughput than the speed of any single job. If you're running an interactive assistant for support, first-token wait and response consistency usually matter more than "tokens per second across all users."

The benchmark setup matters

In the RTX 5090 test, the author uses an 8-bit conversion and a 32,768-token context setting. The rates are medians of three greedy replies, each 256 tokens long, per prompt.

That's a clearly defined test shape. Real traffic is messier: prompt lengths vary, output lengths vary, and requests don't necessarily arrive as a fixed batch of concurrent responses. A fixed "four replies at once" test doesn't show what happens when users arrive continuously or when some requests are much longer than others.

So treat the four-stream result as evidence that this recipe can sustain substantial aggregate decoding under the stated conditions - not as a universal service-level guarantee.

How should you interpret the throughput results?

First, separate three things: aggregate throughput, per-request throughput, and latency. They're related, but they answer different questions.

  • Aggregate throughput: total output rate from the whole serving setup
  • Per-request throughput: how fast one request produces tokens
  • Latency: how long a request waits, including the time before generation starts

TensorFold's RTX 5090 results show combined throughput rising as concurrency increases in this specific test. What they don't prove is that every request gets faster as you add streams.

In fact, you can raise total completed work while individual users still see worse latency or a lower effective token rate. Whether that trade-off is acceptable depends on your product and your traffic pattern.

Read the measurement as a controlled snapshot

The test uses greedy 256-token responses and reports medians from three replies per prompt. A median reduces the impact of one weird fast or slow run, but three observations is still a small sample. Use the results as a directional snapshot, not as a promise for every prompt or every session.

The setup also locks in the model conversion, GPU, operating system, context setting, and response length. Change quantization, switch software builds, alter the prompt mix, or move to different hardware, and you're effectively running a new experiment.

That's why comparing "tokens per second" from two posts can lead you astray when the test conditions don't match your situation.

Do not infer answer quality from speed

The report includes consistency checks: 12 of 12 concurrent greedy replies matched their solo runs, and 12 of 12 replies with copy drafts disabled matched the drafted runs. That's helpful for output consistency under the tested conditions.

It doesn't measure correctness, task-completion rates, or whether the model produces better answers under concurrency.

If you're evaluating for a business use case, quality needs its own test set based on what the model will do for you. That might mean policy-reviewed support answers, code changes verified with tests, or structured extraction measured against labeled examples. The serving benchmark can tell you about capacity; it can't replace an evaluation of whether the output is useful and safe enough for your actual task.

What changes when the prompt gets longer?

Long prompts don't just "slow things down." They change what part of the user experience becomes slower first.

Before the model can generate tokens, it has to process the input. That means time to first token becomes its own part of the experience - especially for retrieval, document analysis, and workflows where each request includes a long conversation history.

TensorFold reports two different views of long-input behavior. On the RTX 5090, generation rates for four prompt types stay within a similar range after a 28,000-token input. On an RTX 4090 running Windows, the time to first token increases as input length grows. These measurements aren't from the same hardware or conditions, so you can't treat them as a controlled "4090 vs 5090" comparison.

Decode after a long input

For the RTX 5090 test, the reported generation rates before and after the long input were:

Prompt type Short input (tokens/s) After 28k input (tokens/s)
Prose 199 199
Code 191 198
JSON 203 197
List 204 200

The narrow takeaway is that decode speed stayed in a similar range for these prompt types after the long input had been processed. That doesn't mean the long input was "free." These rates describe output generation after input processing; they don't report the total request time, the time spent processing the prompt, or whether outputs were better or worse.

First-token wait grows with input size

The separate RTX 4090, Windows, 8-bit test reports time to first token of:

  • 0.24 seconds at 2,000 input tokens
  • 0.93 seconds at 8,000
  • 1.93 seconds at 16,000
  • 4.65 seconds at 29,000

The pattern is clear in that report: larger inputs lead to a longer wait before generation starts.

But don't use those figures to declare which GPU is faster overall. The tests differ in hardware and conditions, and the report doesn't offer a controlled setup. Use the figures as a concrete reminder to measure both prompt processing and decode - rather than relying only on output tokens per second.

What are the limits of this CUDA experiment?

A benchmark is most useful when you understand its boundaries.

TensorFold's recipe reports that the RTX 5090 server used 9.47 GiB for the model plus 1.23 GiB per stream at the 32k context setting. That helps explain how memory usage changes with concurrency in this particular setup. It isn't a complete memory budget for a different model, different quantization, different context length, different runtime, or different GPU.

The arithmetic also helps frame capacity. The model footprint is shared, while the per-stream costs add up as more requests run concurrently. At four streams, the components implied by the figures add up to 14.39 GiB. That shouldn't be treated as "the total memory you need" for every deployment - this source describes model and per-stream usage, not every runtime detail and system allocation you might see elsewhere.

Context limits are operational limits

The recipe sets a 32,768-token context. A planned 30,720-token input was tokenized past that limit and refused. In other words: a "nominal prompt size" doesn't always match the final token count after tokenization in the real system.

If your workflow depends on long documents, test the exact inputs you'll use and check how they get tokenized before assuming they'll fit.

The CUDA path described in the recipe also doesn't serve a drafter and doesn't provide a structured-output mode. Those gaps matter if your application depends on draft-based decoding or constrained JSON generation. A fast general generation path isn't automatically a drop-in replacement for a serving stack that supports the features your app needs.

Hardware coverage is narrow

The recipe covers RTX 4090 and RTX 5090 configurations, but other CUDA GPUs weren't tested in the same way. The measurements are presented as the author's results, and the report notes that they're historical rather than independently rerun here.

The author labels its 4090 test as October 25 and its 5090 test as October 26, 2026, without specifying a timezone; the source revision was checked on November 13, 2026 UTC.

That provenance matters. Software components - kernels, drivers, and even model revisions - can change performance. If you plan to base a purchase or deployment decision on this, reproduce the recipe on the hardware and software you'll actually use. The charts summarize reported numbers; they're not runtime screenshots and not independent verification.

How should you evaluate local LLM serving?

Start with your workload, not the most impressive number in a chart. A useful evaluation makes assumptions explicit and measures the parts of the experience users actually feel.

For a local assistant, that usually means testing concurrency and first-token wait alongside generation rate. For offline processing, total jobs completed per hour may matter more than how fast any single response begins.

Use a workload-first checklist

Before you compare GPUs or serving recipes, write down what you're really trying to run:

  • Request shape: typical and worst-case input lengths, output lengths, and prompt types
  • Concurrency: how many requests arrive together, and whether it's steady or bursty
  • User-facing latency: time to first token and end-to-end completion time, measured separately
  • Capacity: aggregate tokens per second and completed requests over a representative period
  • Memory: model footprint, per-request cost, context setting, and headroom under peak load
  • Quality and features: task success, structured-output requirements, and any decoding features your app depends on
  • Reproducibility: model revision, quantization, software build, GPU, operating system, and measurement method

This is what makes results comparable later. If you test another GPU, you can compare like with like instead of chasing a new headline. It also helps you identify the real bottleneck: memory pressure, prompt processing, decode capacity, or a missing feature may each call for a different fix.

Turn benchmark results into a deployment decision

A practical approach is to start with one request, then scale up to the concurrency you expect in real use. Keep the prompt and output lengths representative, and record both per-request latency and combined throughput. Repeat enough times to see variability, and keep the model and runtime versions fixed during comparisons.

Then test the edges: longer inputs, peak concurrency, and the biggest outputs users are likely to request. If the system handles average traffic but fails on long documents, or if latency spikes during bursts, the average tokens-per-second number will hide the problem you actually care about.

Finally, set acceptance criteria before tuning. For example, decide what first-token wait is acceptable for your interactive use case, and what minimum batch throughput you need for background work. Those thresholds are product decisions, not properties of a GPU benchmark. The "right" configuration is the one that meets them with enough memory and operational headroom - not necessarily the one with the highest aggregate rate.

Conclusion

TensorFold's experiment suggests that Spark-X2.6-4B can deliver much higher combined decode throughput as concurrency increases on the reported RTX 5090 setup. It also shows why you need context: aggregate throughput doesn't tell you how users experience it, long inputs add first-token delay, memory rises with streams, and the CUDA recipe comes with feature and hardware limits.

Use the numbers as a starting point, not a deployment promise. Re-run the recipe using your prompt mix, measure both first-token and end-to-end latency, and validate answer quality separately. If you share results from another card, include the GPU, quantization, prompt length, concurrency, decode rate, and first-token wait together - then another builder can actually use what you found.

Frequently Asked Questions

No. The 716–729 tokens per second figure is combined throughput across four concurrent streams. Dividing by four gives a rough average, not a measured per-user rate. The report does not establish fairness, per-request latency, or how throughput is distributed between users.
In the reported RTX 5090 test, generation rates after a 28,000-token input stayed in a similar range for prose, code, JSON, and list prompts. Those rates measure decoding after input processing; they do not include the cost of reading the prompt or describe total request latency.
No. The throughput and long-prompt decode results use an RTX 5090 running Linux, while the time-to-first-token test uses an RTX 4090 running Windows. They have different hardware and conditions, so they are not a controlled GPU comparison.
No. The reported 12-of-12 matches are output-consistency checks between specified runs. They do not measure whether answers are correct, safe, or useful for a business task. Evaluate those outcomes separately with representative examples and task-specific criteria.

Related articles