A couple of years ago, I did a really deep dive into understanding (as an amateur, of course) how the popular Large Language Models (LLMs), like ChatGPT, really worked. I was very surprised to learn that a workable model could be run on your local laptop, albeit perhaps not as quickly as would be the case with an online interface with OpenAI or another company. After playing around with ChatGPT, and seeing the amazing “conversations” I could have with it, I had just assumed that must require a supercomputer filling a warehouse to generate such answers to my prompts. But no, it was the training of the LLM—finding the “weights” on its billion-plus parameters—that was computationally expensive; running the model was relatively easy in comparison.
Now since that time, the cutting edge LLMs have grown in complexity, such that you really can’t literally run them on a laptop anymore. But even here, there are shortcuts available that can approximate decent versions of the leading models.
In the present post, I’ll offer a quick tutorial on the basic structure of a modern LLM, and then give some specifics on the computational power involved at various stages.
How an LLM Works
A large language model is, at its core, a very large collection of numbers. These numbers (“weights”) encode everything the model “knows” about language, reasoning, and the world. You can think of the weights as the distilled product of training: a snapshot of all the patterns the model has learned, stored as a giant file on a hard drive.
The model itself is a neural network, which is a layered mathematical structure loosely inspired by the human nervous system. When you type a prompt, your text gets converted into numerical inputs, which are then passed through layer after layer of the network. (Current models have 80 – 120+ layers.) At each layer, the inputs get multiplied by the weights, transformed by a simple mathematical function, and passed along to the next layer. By the time the signal has passed through all of the layers, the output is a prediction of which word (or token, to be more precise) should come next. The model samples from that prediction, appends the word to the (growing) answer to your prompt, and repeats the process until the response is complete.
That’s it. No magic, no lookup table of canned responses to popular inputs, no hidden internet search. Just a very large sequence of arithmetic operations, applied over and over again. When you understand just how simple the mechanical operations are “under the hood,” you can see why some critics maintain that LLMs will never truly be thinking or creative; there’s no genuine comprehension. (Here I don’t want to get philosophical, except to say: a materialist could use the very same approach to say that humans don’t actually comprehend questions and think about their answers to them, because after all if you zoom in on atoms in their brain, they just bounce around according to the laws of physics.)
The Hard Part: Training
Now here’s the key distinction. There are two completely different computational tasks involved in bringing an LLM to life: training and inference.
Training is the brutal part. To discover the weights in the first place, the developers feed the model staggering quantities of text—essentially a large fraction of the written internet—and repeatedly ask the model to predict the next word. When it gets it wrong (which it does, constantly, at first), an algorithm called backpropagation adjusts the weights ever so slightly in the direction that would have produced a better answer. This process is repeated billions of times across hundreds of billions of examples. Training GPT-4 is estimated to have cost tens of millions of dollars in compute and took months running on clusters of thousands of specialized chips. It is, without exaggeration, one of the most computationally intensive tasks humans have ever undertaken.
But the crucial point is that training only has to happen once (or once per major version). The result is a file containing the weights. Once that file exists, the hard work is done.
The Easy Part: Running the Model
Using the model (the technical term is inference) is a completely different story. Here, there is no backpropagation, no gradient computation, no iterative adjustment of weights. You simply load the weights into the working memory (RAM) and run each new prompt through the network in one forward pass. It’s still a lot of arithmetic—mostly matrix multiplication for those familiar with that part of mathematics—but it’s the kind of arithmetic that modern hardware handles with ease.
This is why the narrator of the YouTube video I saw two years ago was correct: In principle, someone could download the weights file and run the model locally. The weights are just data. The inference process is just computation. Neither requires a supercomputer.
So Why Does ChatGPT Run on Massive Servers?
Especially amidst the talk of data centers absorbing large fractions of the electricity supply, the claim above is probably surprising. If actually running an LLM day-to-day to answer specific prompts isn’t so challenging, why do the popular models run on massive servers?
There are two main reasons. First, OpenAI (to take a specific example) is serving millions of users simultaneously, which requires enormous infrastructure even if each individual query is cheap. Second, the flagship models like GPT-4 are genuinely huge—the weights file alone is estimated to require somewhere between 500GB and 1TB of memory to hold. That’s well beyond what any laptop can handle.
Ah, so even though a crude LLM could run on your laptop, to get a “conversational” LLM you still need a supercomputer? Eh, not quite, as we explain in the next section.
The Shortcuts That Make Local Inference Possible
For open-weights models, where the developers have publicly released the weights (such as Meta’s LLaMA family or Mistral), researchers and hobbyists have developed clever techniques to make local inference practical.
The most important is quantization. In their native form, the weights are stored as 32-bit or 16-bit floating-point numbers, which have high precision, but consume a large amount of computer memory. Quantization reduces this precision aggressively, down to 8-bit or even 4-bit integers. The memory footprint shrinks dramatically, with only modest degradation in output quality. A model that would require 40GB of RAM to run at full precision might need only 5GB at 4-bit quantization.
(A “bit” is a single binary digit—either 0 or 1—and a number’s precision is determined by how many bits are used to represent it in the computer’s memory. The weights in an LLM are typically small decimal numbers, and quantization doesn't change their rough magnitude but rather how precisely they can be expressed [similar to rounding a measurement from four decimal places down to one]. A 16-bit number can distinguish among 2^16 = 65,536 distinct values, while a 4-bit number can only distinguish among 2^4 = 16, so quantization trades some of that fine-grained precision for a dramatic reduction in memory usage.)
A second technique is offloading to disk, where layers of the model that don’t fit in RAM are kept on the SSD (i.e. “hard drive”) and swapped in on demand. This works, but at a speed penalty, because generation slows considerably when the system is constantly reading from disk storage rather than working memory.
Finally, for users who need something even more compact, distilled models offer a smaller network that has been trained to mimic the behavior of a larger one. The result is a genuinely smaller and faster model, at the cost of some capability.
What Can a Laptop Actually Run?
Now that we’ve worked through the general framework and caveats, let’s return to the fundamental question of this blog post: What type of LLM can a laptop today actually run locally, with no connection to the internet?
Currently, the best consumer hardware for local LLM inference is the Apple MacBook Pro with the M4 Max chip and 128GB of unified memory. The key advantage of Apple Silicon is that the CPU and GPU share the same memory pool, so all 128GB is available to hold model weights (unlike a Windows laptop, where GPU memory and system RAM are separate).
With 128GB and 4-bit quantization, you can comfortably run a model with roughly 70 billion parameters. Models at that scale—such as Meta’s LLaMA 3.3 70B or Qwen 2.5 72B—score above GPT-3.5 on most standard benchmarks, and are competitive with lower-tier GPT-4 on many tasks. (People who are more expert in this space tell me the best such model right now is Qwen3.5-27B.) For everyday use cases like drafting, summarizing, coding assistance, and question answering, the gap between these local models and the frontier API models is real but not dramatic.
Speed is also surprisingly competitive. A top-of-the-line MacBook Pro running a 70B model at 4-bit quantization will generate roughly 15–30 tokens per second. This is noticeably slower than responses of 50–80 tokens per second you’d get from OpenAI’s servers, but fast enough that conversation feels natural. And for short responses, local inference can actually feel faster, because there’s no network round-trip time or server queue (where you are waiting your turn for the network to respond to your specific question).
Private vs. Public Weights
There is one important clarification to all of this: it only works if the weights are publicly available. OpenAI, Anthropic, and Google do not release the weights for their flagship models. GPT-4, Claude, and Gemini Ultra are closed systems—you must use their APIs, and the weights never leave their servers. The local inference ecosystem runs entirely on open-weights models, which are excellent but are not the same thing as the proprietary frontier models.
Conclusion
My original mental image two years ago—that answering a single AI question requires a supercomputer—was wrong in an important way. The supercomputer is needed to discover the weights. Once they exist, running the model is a much more modest undertaking. All you need is a file, some arithmetic, and increasingly, just a very good laptop. As the leading companies eventually pour billions of dollars into training ever more sophisticated versions of their proprietary LLMs, slimmed-down versions can be deployed locally in “talking” robots, drones, cars, and other machines. As the RAM and computational speed of even an isolated drone improves, eventually the drones of the future will appear as “intelligent” as today’s leading LLMs.
Dr. Robert P. Murphy is the Chief Economist at infineo, bridging together the dependability of Whole Life insurance policies with the benefits of blockchain-based finance.
Twitter: @infineogroup, @BobMurphyEcon
Linkedin: infineo group, Robert Murphy
Youtube: infineo group
To learn more about infineo, please visit the infineo website