Edge0-145B: Run a 145B AI Model Locally13 min read2,505 words

Edge0-145B: Run a 145B AI Model Locally

See how Edge0-145B runs a 145-billion-parameter AI model locally, what its benchmarks show, and the hardware, privacy, and cost trade-offs to consider.

Edge0-145Bon-device AIlocal AI modelAI inferenceopen-source AI
Edge0-145B: Run a 145B AI Model Locally

Edge0-145B: What It Takes to Run a 145B AI Model Locally

Most big AI models come with a trade-off: use the cloud and pay ongoing fees (plus deal with network and data-handling concerns), or run something smaller on your own machine and accept reduced capability. Edge0-145B tries to move that trade-off in the other direction. This open-source model is meant to run coding, reasoning, and agent-style tasks on a compatible personal computer - without relying on a remote server.

The headline is eye-catching, but "145B parameters" doesn't automatically tell you whether a local setup will feel usable. Memory, generation speed, power draw, software support, and - most importantly - task quality all matter. Edge0 says its 145-billion-parameter model hits 80+ tokens per second and uses 29GB of peak memory in its tested setup. Those are the company's numbers, not a promise that every laptop (or every workload) will land the same way.

In this article, we'll look at how the system is designed, what its benchmarks and demos can (and can't) show, and when local inference actually makes sense for business use. We'll also cover the limits: you still need compatible hardware, local AI doesn't eliminate operating costs, and some workloads will remain better suited to cloud services.

What does Edge0-145B offer on a personal computer

Edge0-145B is an open-source model with 145 billion total parameters. It was released alongside Edge0-120B and a unified inference framework. The point isn't just that you can download a large model. It's that a model this size can generate responses on a compatible laptop at speeds meant for interactive work.

Edge0 reports peak memory use of 29GB, prompt processing above 1,100 tokens per second, and generation above 80 tokens per second. It also reports typical power draw of about 120 watts in its test setup, along with support for a 128K-256K context window. Those measurements reflect different parts of the experience. "Prefill" is the work of processing your prompt before the model starts replying, while "decode" is the speed at which it produces output.

In practice:

  • A fast prefill rate helps when you're feeding large codebases or long documents.
  • A strong decode rate can make interactive coding feel less like you're waiting on a remote service.
  • A long context window helps keep more material in view, though it doesn't guarantee the model will use every detail correctly.
Edge0-145B reported measurements
Reported measure Edge0-145B result Why it matters
Peak memory 29GB Determines whether the model and its runtime fit alongside your OS and other apps.
Prompt processing 1,100+ tokens/s How quickly long prompts get processed before output begins.
Generation 80+ tokens/s How fast the model produces interactive output.
Typical power Approximately 120W Impacts sustained workloads, heat, and energy use.
Context window 128K-256K Supports longer inputs and longer tasks, depending on runtime and workload.

These are reported results on a compatible setup - not minimum specs for every device. Edge0 lists day-one support for macOS, Windows, and Linux, including Apple M-series systems and selected NVIDIA platforms. But "supported" doesn't mean "runs the same way on every Mac, every Windows laptop, or every NVIDIA rig." Available memory, hardware configuration, and the inference software stack all change the outcome.

Edge0 also compares its numbers to conventional deployments where memory requirements are often higher. In its comparison, Edge0-145B uses 29GB, while standard 4-bit deployment estimates for some comparison models range from about 70GB to 87GB. The Edge0 figure is measured internally; the others are estimates. So these comparisons shouldn't be treated as a controlled, like-for-like hardware test.

If you're deciding whether this is worth it, the question isn't "Can it run on a laptop?" It's "Can it fit my actual hardware, sustain the speed I need, and perform well on the tasks my team actually runs?" The published figures are a good starting point - but they don't answer the question for every configuration.

How does Edge0 run a 145B model locally

A big part of the engineering challenge is data movement. Large models have huge parameter sets, and inference repeatedly needs access to the model weights. Those weights compete for space in memory and bandwidth with the operating system and everything else running on your machine. Model compression can shrink the footprint, but the speed and output quality you get depend on the model, file format, hardware, and runtime.

Edge0 says it designed the model architecture and the inference engine together, instead of training a conventional cloud model first and only later compressing it for local use. The framework is described as four layers: architecture, training, loading, and scheduling.

Sharing work across sparse experts

Edge0 uses a mixture-of-experts design. In a typical setup, each expert has its own full set of large matrices. Edge0 says groups of 64 experts share two large matrices, while each expert still keeps a smaller set of parameters that captures its differences. The company reports that about 10 billion of the model's 145 billion total parameters participate in computation at each inference step.

That helps explain why "total parameter count" doesn't equal "how much work happens per token." The full model's data doesn't vanish, though. The system still has to manage access to whichever expert parameters it needs. The design goal is to reduce repeated storage and data movement while keeping a large pool of expert capacity available.

Loading across GPU memory, system memory, and SSD

The inference framework uses multiple memory tiers. Edge0 describes keeping frequently used experts in GPU memory, storing an expert library of roughly 40GB in system memory, and reading about 48GB of vocabulary data from an SSD as needed. Loading and computation are pipelined, so the system can overlap some data access with ongoing processing instead of handling each step strictly one after another.

This is a real trade-off. Using system memory and SSD storage can make a larger model possible without requiring everything to fit in GPU memory at once. But then storage bandwidth and access patterns start to matter. A configuration that feels fast in one setup doesn't automatically translate to the same experience on a laptop with a slower SSD, less memory, or different hardware.

Scheduling CPU and GPU work together

Edge0 says its runtime assigns cached-expert computation to the GPU while the CPU handles other work in parallel. It also uses speculative decoding, where it evaluates multiple candidate tokens together to spread the cost of reading weights across more generated output.

The broader takeaway is that local inference speed isn't dictated by model weights alone. Architecture, memory management, scheduling, numerical formats, and chip-specific optimizations all contribute. That's why the model release and the compatible inference framework matter together - and why throughput numbers should always be understood in the context of the hardware and testing conditions that produced them.

For teams evaluating this system, reproduce the workload that matters: similar prompt lengths, similar tools, similar concurrency, and a similar software environment to what you'll use in production. A short demo can show that the model runs. It can't reliably show sustained performance over a day of real use.

What can Edge0-145B do, and how strong is the evidence

Edge0 positions the model for coding, reasoning, and agent tasks - work where you may want the system to plan, call tools, inspect outputs, and revise its approach. The benchmark table includes coding and terminal-style evaluations, along with reasoning, knowledge, and instruction-following tests.

In Edge0's reported results, Edge0-145B scores 64.6 on DeepSWE 1.1 and 86.52 on Terminal-Bench 2.1. It scores 90.4 on GPQA-Diamond and 80.6 on IFBench. In the company's comparison table, it shows higher results than Qwen3.8-27B on DeepSWE and Terminal-Bench, though some other models score higher on specific tests. There isn't a single benchmark winner across every category.

Edge0 reports a mean score of 72.8 across six listed benchmarks, and a measured 29GB peak memory footprint. Its table estimates around 70GB for a standard 4-bit deployment of Qwen3.8-Flash-Next, whose reported mean is 71.7. That comparison can be useful, but it mixes Edge0's internally measured memory with estimated figures for other models. It's not a universal efficiency measure.

Benchmark provenance matters too. Edge0 says its own results and several competitor results come from internal evaluations. Some other model results come from third-party evaluations, and the table notes that missing scores aren't zero - they're just unavailable. The company says a technical report includes evaluation setup and reproduction instructions. Until you run the relevant tests yourself, it's safer to treat benchmark scores as a screening signal, not a production guarantee.

Agent workflows are more than a good answer

The demos focus on full workflows that combine planning, code generation, execution, debugging, and visual inspection. Edge0 describes using the model to create a Blender city scene, build playable games, generate a video, write software for a drone task, and design a PCB in KiCad before producing an assembly animation.

These examples matter because they show the difference between a chatbot response and an agent workflow. A chatbot can suggest code. An agent can run it, inspect the error, adjust the code, and check the result. That loop can save time when the environment and tools are well defined. It can also make mistakes faster - so you still want permissions, sandboxing, and review, especially if the software can touch hardware or modify important project files.

Edge0 also describes some demos as finishing in under an hour, and it compares that time to human production estimates or cloud token costs. Those are still company-provided examples, not independent cost studies. Outcomes will vary with the prompt, the quality bar, the tool setup, and how much human review you include.

If you're evaluating this for business use, choose a task that looks like your real work and define what "done" means before you run it. Track the successful completion rate, how long it takes to correct wrong outputs, tool failures, and the cost of operating the hardware. A compelling demo is a starting point. Repeatability is what you need for deployment.

When does local AI make more sense than cloud AI

Local inference changes where computation happens. It can reduce dependence on a model-provider API, keep prompts and files on the user's machine, and let core model functions work without a network connection. For work involving source code, product plans, internal documents, or regulated information, that can simplify parts of data handling.

It doesn't automatically make things private, though. Teams still need to understand what the application logs, where model files and outputs are stored, which tools can access local data, and whether telemetry is enabled. Local inference only solves one part of the overall data path. Endpoint security, access control, retention rules, and user permissions still matter. In GDPR-regulated environments, running locally doesn't by itself settle questions about lawful processing or broader data governance.

Cost is similar: local inference avoids cloud token/API fees, which can matter when agent workloads involve many tool calls. But you still pay for hardware upfront, electricity over time, staff time to support it, and potentially upgrades as requirements change. If usage is occasional, a cloud API may be cheaper and simpler. If usage is frequent and you already have a suitable machine, local execution can make costs easier to predict.

A practical decision framework

Local inference is worth considering when most of these are true:

  • You run the task often enough that cloud usage and data-transfer overhead add up.
  • Your hardware can handle the workload with enough memory and acceptable sustained performance.
  • Keeping data on-device (or working offline) creates a real operational benefit.
  • You can evaluate model quality using your own representative tasks and review failures.
  • You're comfortable with the total cost of hardware, power, maintenance, and the engineering effort.

Cloud inference still makes sense when you need a model that's larger than your hardware can support, when you need access to up-to-date online information, when you need shared services across devices, or when you don't want the burden of maintaining local systems. Often, a hybrid setup is more practical than forcing everything into one place: run predictable or sensitive work locally, and send only the appropriate requests to a cloud service under a defined policy.

Edge0 itself describes the future as more on-device work with cloud support - not the end of cloud AI. Its 35B model is aimed at phones, while Edge0-145B targets computers that can handle the memory and performance requirements. Hardware constraints still drive what's feasible: battery-powered devices and laptops have different heat and power limits compared with workstations.

For developers, the open-source release offers another option: inspect the weights and inference stack, evaluate the model, and adapt it for a product. Open source doesn't remove licensing, security, or maintenance obligations. If you plan to build a commercial workflow around any model, check those details and test the exact release you intend to deploy.

Conclusion

Edge0-145B is interesting because it treats local large-scale AI as a full system problem, not just a model download. With 145 billion parameters, reported 29GB peak memory, and interactive generation speeds, it may make complex coding and agent workflows practical on compatible computers.

That said, you still need to read the evidence carefully. Benchmarks are task-specific, some comparisons mix measured and estimated memory figures, and demos don't guarantee day-to-day production reliability. The next sensible step is a controlled test against your own workloads, your own hardware, and your own quality bar.

Local AI is most useful when privacy needs, offline access, response time, or usage economics solve a real operational problem. If that matches your situation, Edge0-145B is worth evaluating. If it doesn't, cloud inference - or a hybrid setup - will likely be the more practical path.

Frequently Asked Questions

Not necessarily. Edge0 reports 29GB peak memory in its tested setup, so the computer needs sufficient available memory and compatible hardware. The company lists support for selected Apple M-series and NVIDIA systems, but performance varies by configuration. Check the current hardware guidance and test your own workload before relying on it.
No. Local inference avoids the corresponding cloud-model token or API fees, but you still pay for the computer, electricity, maintenance, and engineering time. Whether it is cheaper depends on usage volume, hardware you already own, and the cost of operating and supporting the local setup.
The published evaluations show competitive results against specific named models on selected benchmarks. They do not establish parity across every task or prove that it will outperform cloud models in your application. Test representative prompts, tools, and completion criteria before making a product or procurement decision.
No. Processing can remain on the computer, but privacy also depends on application behavior, logging, telemetry, file access, and endpoint security. Review the full data flow and configure permissions appropriately. Local execution can help meet data-handling needs, but it does not replace a privacy or security review.

Related articles