A recent demonstration shows the Qwen3.8-27B model running on the iPhone 18 Neural Engine with speeds between 15 and 70 tokens per second. The model uses 1.5 bits per weight (bpw) quantization, making it feasible for on-device inference. This development highlights the growing capability of mobile hardware to handle large language models efficiently.
The performance metrics suggest that advanced AI can now run locally on smartphones, reducing reliance on cloud services. As edge computing evolves, such optimizations may become standard, enabling more private and responsive AI experiences for users.
Source: Hacker News · Summarized by HeadlinesBriefing