An order of magnitude less epochs Across a large range of scales and datasets, e-means reaches a test loss below Lloyd's at 25 epochs, in 1.2 to 3 epochs. Measured end to end, that is a 10-45x wall-clock reduction on GPU against the state-of-the-art Lloyd implementation (Flash-KMeans), and 8-15x on CPU against Faiss. Speed and quality usually trade against each other in k-means, while e-means achieves both better speed and loss.
The k-means problem Given a fixed dataset of n points X = {x1, ..., xn} subset Rd and an integer k, the k-means problem is to find k centroids C = (c1, ..., ck) minimizing the mean squared distance between each point and its nearest centroid. The e-means project The goal of e-means is to provide a modern optimization rule for the k-means problem. We started this project from the following observations: k-means is one of the rare problems in machine learning where full-batch optimization is still widely used.
Mini-batch k-means methods in the literature exist, but are mostly based on the Mac Queen update rule, released in 1967. Modern deep learning offers optimization tools that can enhance mini-batch methods. For e-means, we chose to do mini-batch updates with exponential moving averages, which we found to be most efficient for the k-means problem.
Contrary to previous mini-batch approaches, e-means requires no hyperparameter tuning because it automatically sets the batch size and step size with scaling laws. In addition, e-means improves the initialization and the dead-cluster revival strategy, adds a cosine-scheduled decay rate, and introduces a finalization stage that reduces the training loss for a negligible runtime penalty at the end of training. e-means is highly parallelizable, and comes with specialized kernels that translate its algorithmic advantage into wall-clock speedups. Moving-average updates Each mini-batch updates every cluster's count and sum.
The centroid is their ratio. No hyperparameter tuning Everything is derived from scaling laws. log n, log k, log T Start and finish Novel initialization and finalization methods. The run ends on each cluster's exact last-epoch mean.
Source: Hacker News · Summarized by HeadlinesBriefing