Right now, you’re your agent’s context engine. You write the rules files, maintain the skills, and paste in the Slack thread it missed. But hand-curated context goes stale. And when there are conflicting answers, your agent can’t tell which is right. You pay for it in higher token bills and longer reviews. Join us live on Oct 28 (FREE) to see: The 8-levels of AI maturity and where teams get stuck How a context engine curates context and resolves conflicts before your agent writes a line of code Live demo: token savings in the same coding task, and use cases beyond code gen Bonus: 2 open source tools you can use today. Register now
A large language model (LLM) is a neural network. You can use it to process and generate language. How the model behaves is dependent on the numerical values you teach it during training. These numerical values are known as parameters. Apple fits a 3-billion-parameter model on your iPhone. What this means is that the model has roughly 3 billion numerical values that it needs to work properly. These values influence how the model makes sense of its input and how it produces output. How is Apple able to fit such a model on an iPhone without any adverse impact on the device? In this article, we’ll try to answer this question.
Here’s what we will cover: The problems of running an LLM on a phone How Apple makes the weights cheaper to store How Apple made repeated calculations faster How adapters help features share the foundation How Apple optimizes the hardware to run the models How does the system decide when to use the cloud
Running an LLM is also called inference. In this stage, the LLM uses its parameters to process a request. Before processing some text, the LLM has to divide it into tokens. You can think of a token as a word, part of a word, or even punctuation. The model takes these tokens and generates more tokens as part of the response. Within the model, these tokens pass through a bunch of processing layers. The first problem is about storing the parameters. At 16-bit precision, the 3 billion parameters we talked about earlier can occupy almost 6GB. But that’s not all. You need space for the input prompt, calculations, and temporary results that the model needs. Lastly, you also need to run the OS and other apps using the same memory. As you can see, storage is a big decision. The second problem is about moving the weights quickly from memory to the processor. During text generation, the processor has to repeatedly fetch numerical data from memory. Even if you have a powerful processor, it might spend a lot of time just waiting for that data to be fetched from memory. The Apple engineering team also had to deal with the challenges regarding memory capacity and memory bandwidth while trying to make their LLM run on the iPhone. On top of this, a mobile phone also has to take care of battery consumption, overheating, and other apps trying to use the same resources. You don’t want the LLM to make the rest of the device unresponsive. The most direct way of reducing the size of the model is to use fewer bits to store each weight. This technique is known as quantization. You can think of quantization as a way to represent numerical values using a smaller set of possibilities. The Apple engineering team uses a technique known as palettization. In this technique, you group similar weight values. A small lookup table stores representative values, and individual weights store short references to that table. For example, you can have a 2-bit reference select from four entries in a lookup table.
Source: ByteByteGo · Summarized by HeadlinesBriefing