You could have invented transformers from memory

Large language models can recall obscure facts from every corner of the world, but how do they remember so much? This post explores linear maps, MLPs, and self-attention as different kinds of memory stores. The key property of a good memory store is its ability to recall what it stored. The more memories we store, the more they interfere with each other when we try to recall one. The interference-to-signal ratio measures how much the other memories get in the way during recall. The number of memories a store can hold before the interference grows as large as the signal is its capacity. We’ll show how you can go from a linear layer to something better, an MLP, which improves both capacity and recall, and then all the way to self-attention, which has very good recall and can store a potentially unbounded number of memories. With 64-dimensional keys, that’s a jump from about 64 memories, to about 2,700, to over a million.

A memory store is a function f that holds M memories, for i = 1, ..., M. A memory is one key-value pair: a key k_i ∈ ℝ^{d_k}, used for lookup based on similarity, and a value v_i ∈ ℝ^{d_v}, which is what we want to retrieve. To use f, we query it with an input x, and it recalls an output f(x). A good memory store recalls v_i when we query it with k_i, or with another vector similar to k_i. In the literature, a memory store that recalls by similarity is called an associative memory. An outer product memory store is the most basic memory store: it simply accumulates memories via outer products, W = ∑_{i=1}^{M} v_i k_i^T = V^T K, where the rows of K ∈ ℝ^{M×d_k} and V ∈ ℝ^{M×d_v} are the keys k_i^T and the values v_i^T. To store a new memory (k, v), we simply add vk^T to the existing weights. To query the memory store with an input x, we multiply the input by the matrix: f(x) = V^T K x = ∑_{i=1}^{M} v_i (k_i^T x). So recall is a lookup by similarity to the stored keys: we compare the query with every key, and return the sum of the values weighted by those similarities. Recall picks up interference: storing memories is easy, but what happens when we try to recall one? Let’s assume for simplicity that every key is normalized to have length 1, i.e. k_j^T k_j = 1. Then f(k_j) = v_j + ∑_{i≠j} v_i (k_i^T k_j). The first term is the signal we want, and the second term is the interference from other memories.

يمكن للمodels لغوية كبيرة استرجاع حقائق obscure من كل أنحاء العالم، ولكن كيف تتذكر هذه الكمية؟ يستكشف هذا المنشور الخرائط الخطية، MLP، والانتباه الذاتي كنوعين مختلفين من Stores الذاكرة.

الخاصية الرئيسية لذاكرة جيدة هي قدرتها على تذكر ما تخزنه. كلما زادت الذكريات التي نخزنها، زاد تداخلها عندما نحاول استرجاع واحدة. نسبة التداخل إلى الإشارة تقيس كم تتداخل الذكريات الأخرى عند محاولة استرجاع واحدة. عدد الذكريات التي يمكن للمخزن الاحتفاظ بها قبل أن تنمو التداخل كما تنمو الإشارة هو سعته. سنوضح كيف ننتقل من طبقة خطية إلى شيء أفضل، MLP، والذي يحسن كل من السعة والاسترجاع، ثم كله إلى الانتباه الذاتي، الذي لديه استرجاع ممتاز ويمكنه تخزين عدد غير محدود من الذكريات. مع مفاتيح بعدية 64،那是 قفزة من حوالي 64 ذكرى، إلى حوالي 2,700، إلى أكثر من مليون.

مخزن الذاكرة هو دالة f تحتفظ فيها M ذكرى، لـ i = 1, ..., M. كل ذكرى هي زوج مفتاح-قيمة: مفتاح k_i ∈ ℝ^{d_k}، يستخدم للاسترجاع بناءً على التشابه، وقيمة v_i ∈ ℝ^{d_v}، وهي ما نريد استرجاعه. لاستخدام f، نستفسر عنها بإدخال x، وتسترجع إخراج f(x). ذاكرة جيدة تسترجع v_i عندما نستفسر عنها بـ k_i، أو ب vecteur مشابه لـ k_i. في الأدبيات، ذاكرة تسترجع بواسطة التشابه تسمى ذاكرة جمعية. مخزن الذاكرة積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積積

Los grandes modelos de lenguaje pueden recordar hechos obscuros de cada rincón del mundo, pero ¿cómo recuerdan tanto? Esta publicación explora mapas lineales, MLP y atención propia como diferentes tipos de almacenamiento de memoria.

La clave de un buen almacenamiento de memoria es su capacidad para recordar lo que almacenó. Cuantos más recuerdos almacenemos, más se interferirán entre sí cuando intentemos recordar uno. La razón interferencia-señal mide cuánto otras memorias entorpecen el recuerdo de uno. El número de recuerdos que un almacén puede almacenar antes de que la interferencia crezca tanto como la señal es su capacidad. Mostraremos cómo pasar de una capa lineal a algo mejor, un MLP, que mejora tanto la capacidad como el recuerdo, y luego todo el camino a la atención propia, que tiene un excelente recuerdo y puede almacenar potencialmente un número ilimitado de recuerdos. Con claves de 64 dimensiones, eso es un salto de aproximadamente 64 recuerdos a aproximadamente 2.700, a más de un millón.

Un almacén de memoria es una función f que mantiene M recuerdos, para i = 1, ..., M. Cada recuerdo es un par clave-valor: una clave k_i ∈ ℝ^{d_k}, utilizada para búsqueda basada en similitud, y un valor v_i ∈ ℝ^{d_v}, que es lo que queremos recuperar. Para usar f, lo consultamos con una entrada x, y recuerda una salida f(x). Un buen almacén de memoria recuerda v_i cuando lo consultamos con k_i, o con otro vector similar a k_i. En la literatura, un almacén de memoria que recuerda por similitud se llama memoria asociativa. El almacén de memoria de producto externo es el tipo de almacén de memoria más básico: simplemente acumula recuerdos mediante productos externos, W = ∑_{i=1}^{M} v_i k_i^T = V^T K, donde las filas de K ∈ ℝ^{M×d_k} y V ∈ ℝ^{M×d_v} son las transpuestas de las claves k_i y los valores v_i^T. Para almacenar un nuevo recuerdo (k, v), simplemente agregamos vk^T a los pesos existentes. Para consultar el almacén de memoria con una entrada x, multiplicamos la entrada por la matriz: f(x) = V^T K x = ∑_{i=1}^{M} v_i (k_i^T x). Así, el recuerdo es una búsqueda por similitud a las claves almacenadas: comparamos la consulta con cada clave y devolvemos la suma de los valores ponderados por esas similitudes. El recuerdo captura interferencia: almacenar recuerdos容易，但当我们尝试回忆其中一条时会发生什么？假设简单起见，我们假设每个键都归一化为长度 1，即 k_j^T k_j = 1。那么 f(k_j) = v_j + ∑_{i≠j} v_i (k_i^T k_j). La primera término es la señal que queremos, y el segundo término es la interferencia de otras memorias.

बड़ी भाषा मॉडल दुनिया के हर कोने से अस्पष्ट तथ्य याद कर सकते हैं, लेकिन वे इतने कैसे याद करते हैं? यह पोस्ट रैखिक मानचित्र, MLP और स्व-ध्यान को अलग-अलग प्रकार की स्मृति भंडार के रूप में探讨 करता है।

एक अच्छी स्मृति भंडार की प्रमुख गुण यह है कि यह वह याद कर सकता है जिसे उसने संग्रहीत किया है। जितने अधिक यादें हम संग्रहीत करते हैं, उतनी ही अधिक हस्तक्षेप होता है जब हम एक को याद करने की कोशिश करते हैं। हस्तक्षेप-से-सिग्नल अनुपात मापता है कि अन्य यादें याद करते समय कितनी बाधा डालती हैं। एक भंडार कितनी यादें संग्रहीत कर सकता है जब तक हस्तक्षेप सिग्नल के बराबर न हो जाए, उसकी क्षमता। हम दिखाएंगे कि कैसे रैखिक परत से कुछ बेहतर, एक MLP में जाएं, जो दोनों क्षमता और याद करने में सुधार करता है, और फिर स्व-ध्यान तक जाएं, जिसका याद करना बहुत अच्छा है और संभावित रूप से अनंत संख्या में यादें संग्रहीत कर सकता है। 64-आयामी कुंजी के साथ, वह लगभग 64 यादों से बढ़कर लगभग 2,700, और एक मिलियन से अधिक है।

एक स्मृति भंडार एक फलन f है जो M यादें संग्रहीत करता है, i = 1, ..., M। प्रत्येक याद एक कुंजी-मान pair है: एक कुंजी k_i ∈ ℝ^{d_k}, जो समानता आधारित खोज के लिए उपयोग की जाती है, और एक मान v_i ∈ ℝ^{d_v}, जिसे हम पुनः प्राप्त करना चाहते हैं। f का उपयोग करने के लिए, हम इसे इनपुट x के साथ प्रश्न करते हैं, और यह एक आउटपुट f(x) याद करता है। एक अच्छी स्मृति भंडार k_i के साथ हमें प्रश्न करते समय v_i याद करता है, या k_i के समान एक अन्य वेक्टर के साथ। साहित्य में, समानता द्वारा याद करने वाले स्मृति भंडार को सहायक स्मृति कहा जाता है। सबसे बुनियादी स्मृति भंडार बाह्य गुणन स्मृति भंडार है: यह बाह्य गुणन के माध्यम से यादें संचित करता है, W = ∑_{i=1}^{M} v_i k_i^T = V^T K, जहां K ∈ ℝ^{M×d_k} की पंक्तियाँ k_i^T और V ∈ ℝ^{M×d_v} की पंक्तियाँ v_i^T हैं। एक नई याद (k, v) संग्रहीत करने के लिए, हम बस मौजूदा वज़न में vk^T जोड़ते हैं। इनपुट x के साथ स्मृति भंडार को प्रश्न करते समय, हम इनपुट को मैट्रिक्स से गुणा करते हैं: f(x) = V^T K x = ∑_{i=1}^{M} v_i (k_i^T x)। इसलिए याद करना stored keys के समानता द्वारा दिखता है: हम प्रश्न के साथ हर कुंजी की तुलना करते हैं, और समानता से भारित मानों का योग लौटाते हैं। याद करना हस्तक्षेप उठाता है: यादें संग्रहीत करना आसान है, लेकिन जब हम एक याद करने की कोशिश करते हैं तो क्या होता है? सरलता के लिए, हम मान लेते हैं कि हर कुंजी को लंबाई 1 के लिए सामान्यीकृत किया गया है, अर्थात k_j^T k_j = 1। तो f(k_j) = v_j + ∑_{i≠j} v_i (k_i^T k_j)। पहली項是我们想要的信号，第二项是其他 यादों का हस्तक्षेप।

大型语言模型可以回忆世界各地的冷知识，但它们究竟是如何记住这么多的？本文探讨线性映射、MLP和自注意力作为不同类型的记忆存储。

关键属性是一个好的记忆存储能够回忆起它存储的内容。我们存储的记忆越多，试图回忆其中之一时，它们之间的干扰就越多。干扰-信号比衡量在回忆过程中其他记忆干扰了多少。它能存储的记忆数量在干扰增长到与信号相当之前的容量。我们将展示如何从线性层开始，转向更好的 MLP，它能提高容量和回忆能力，然后转向自注意力，后者回忆能力极好，可以存储潜在无限数量的记忆。使用 64 维键，这是从大约 64 条记忆跃升到约 2,700 条，再到超过 100 万条的飞跃。

记忆存储是一个保持 M 条记忆的函数 f，其中 i = 1, ..., M。每条记忆都是一个键值对：一个用于基于相似性查找的键 k_i ∈ ℝ^{d_k}，以及一个我们想要检索的值 v_i ∈ ℝ^{d_v}。要使用 f，我们用输入 x 查询它，它会回忆出输出 f(x)。一个好的记忆存储在我们用 k_i 或与 k_i 相似的向量查询时会回忆出 v_i。在文献中，通过相似性回忆的记忆存储称为关联记忆。最基础的记忆存储是外积记忆存储：它仅通过外积累记忆，W = ∑_{i=1}^{M} v_i k_i^T = V^T K，其中 K ∈ ℝ^{M×d_k} 的行是键 k_i^T，V ∈ ℝ^{M×d_v} 的行是值 v_i^T。要存储一个新记忆 (k, v)，我们只需将 vk^T 添加到现有权重中。用输入 x 查询记忆存储时，我们将输入乘以矩阵：f(x) = V^T K x = ∑_{i=1}^{M} v_i (k_i^T x)。所以回忆是通过相似性查找存储的键：我们将查询与每个键比较，并返回相似度加权的值之和。回忆会拾取干扰：存储记忆很容易，但当我们尝试回忆其中一条时会发生什么？假设简单起见，我们假设每个键都归一化为长度 1，即 k_j^T k_j = 1。那么 f(k_j) = v_j + ∑_{i≠j} v_i (k_i^T k_j)。第一项是我们想要的信号，第二项是其他记忆的干扰。