On periodic distributed representations using Fourier embeddings Jakeb Chouinard ([email protected])1,2 1 Department of Systems Design Engineering, University of Waterloo 2 Centre for Theoretical Neuroscience, University of Waterloo
arXiv:2605.10818v1 [cs.LG] 11 May 2026
Abstract Periodic signals are critical for representing physical and perceptual phenomena. Scalar, real angular measures, e.g., radians and degrees, result in difficulty processing and distinguishing nearby angles, especially when their absolute difference exceeds 𝜋. We can avoid this problem by using real-valued, periodic embeddings in high-dimensional space. These representations also allow us to control the nature of their dot product similarities, allowing us to construct a variety of different kernel shapes. In this work, we aim of highlight how these representations can be constructed and focus on the formalization of Dirichlet and periodic Gaussian kernels using the neurally-plausible representation scheme of Spatial Semantic Pointers1 . Keywords: Distributed Representations; Fourier Features; Spatial Semantic Pointers; Periodic Embeddings
Introduction The natural world has no lack of periodic signals and repeating phenomena, such as orientation angles, circadian rhythms, and musical patterns. Critically, most approaches to distributed embeddings, such as Random Fourier Features (RFFs; Rahimi and Recht, 2007), focus on continuous feature representations, thereby mapping some infinite number line to some high-dimensional trajectory. Previous work has done little to characterize or analyse the periodic versions of these embeddings as well as their induced kernels (see Frady et al., 2021; Voelker, 2020). Compared to representing angles as scalar values, such as degrees or radians, a distributed embedding can avoid the sharp discontinuity than comes when comparing points near the limits of the repeating signal’s period. In this work, we focus on real-valued embeddings of the form 𝜙(𝒙) = F −1 𝑒 𝑗 𝐴𝒙 where 𝐴 ∈ R𝑑×𝑛 is conjugate symmetric, 𝒙 ∈ R𝑛 , and 𝑗 is the imaginary unit. These embeddings are known as Spatial Semantic Pointers (Komer et al., 2019). We focus on these representations and their approach to embeddings due to their neural feasibility. Additionally, their compositional structure allows for a natural extension of these periodic representations to cognitive models. While most 𝐴 sufficiently large matrices will not see repetition in their resultant kernels; it can be desirable to control the point at which a repetition would occur. As an example, head direction relative to one’s environment or body di1 Code implementing the discussed SSP representations is avail-
able in the § “sspspace” “sspspace” Python Python package package as kernel configurations of the “CyclicSSPSpace” encoder.
rection can be thought of as an angular measure—a signal with a period of 2𝜋. A periodic embedding and its corresponding periodic kernel can be constructed by ensuring that all entries of 𝐴 have some common multiple, 𝑡◦ ∈ R+ , such that mod(𝑘𝑡 ◦ 𝐴, 2𝜋) = 0 ∀ 𝑘 ∈ Z. Notably, the “distance” between two orientation states, 𝜖 and 2𝜋 − 𝜖 where 𝜖 is some small number, is poorly represented by their difference. That is to say, the actual distance between 𝜖 and 2𝜋 − 𝜖 is much smaller than |2𝜋 − 2𝜖 |. By contrast, distributed representations capture their adjacency using a dot product measure such that lim 𝜖 →0 𝜙(𝜖) · 𝜙(2𝜋 − 𝜖) = 1 and lim 𝜖 →0 ∥𝜙(𝜖) − 𝜙(2𝜋 − 𝜖) ∥ 2 = 0.
Methods To construct some phase matrix, 𝐴 ∈ R𝑑×𝑛 , with a featurespace period of 𝑡 ◦ ∈ R+ , all entries of 𝐴 must be integer multiples of period-scaled Fourier roots, 2𝜋𝑡 ◦−1 . The set of possible entries for 𝐴, S◦ , can be written as: S◦ (𝑡 ◦ ) = 2𝜋𝑛𝑡 ◦−1 |𝑛 ∈ Z (1) with a corresponding probability mass function (PMF) 𝑝 [𝜔]. Since the PMF is a positive finite Borel measure in Fourier space, Bochner’s theorem demonstrates that there exists a positive definite function that defines a shift-invariant kernel over the feature space (Bochner, 1959). Let 𝐴 be a phase matrix realizing a periodic embedding with period 𝑡 ◦ whose column entries are i.i.d. uniformly sampled up to some frequency band 2𝜋𝑛𝑡 ◦−1 𝐵: 𝐴 ∼ Ω = 2𝜋𝑛𝑡 ◦−1 |𝑛 ∼ U [−𝐵, 𝐵] (2) where 𝐵 ∈ Z+ and U [𝑎, 𝑏] is uniform sampling of integers between 𝑎 and 𝑏 inclusive. Let 𝜙 𝐴 (𝒙) and 𝜙 𝐴 ( 𝒚) be SSP embeddings of two distinct points. The inner product of these two vectors, 𝜙 𝐴 (𝒙) ⊤ 𝜙 𝐴 ( 𝒚), approximates the similarity kernel between two points in R𝑛 : 𝐾 (𝒙, 𝒚). Since the kernel is shift-invariant, 𝐾 (·) can be written as function of the vector from 𝒙 to 𝒚 (or vice versa): 𝐾 (𝒙 − 𝒚). Since the columns of A are i.i.d. and assumed to be from the same distribution, the R𝑑 embeddings of individual features in R𝑛 are independent, and their kernels multiply (Voelker, 2020): 𝑛 Ö 𝐾 (𝒙) = 𝑘 (𝑥 𝑖 ) (3) 𝑖=1
An expression for 𝑘 (·) can be found by taking the inverse Fourier transformation of the probability distribution function
(PDF) from which entries of 𝐴 were sampled:
resulting in the kernel:
−1
Í
𝑘 (𝑥) =F {𝑝(𝜔)} ∫ = 𝑒 − 𝑗 𝜔 𝑥 𝑝(𝜔)𝑑𝜔
𝑘 (𝑥) =
𝜔 ∈Ω
cos(𝜔𝑥) 𝑝 [𝜔]
(5)
𝜔 ∈Ω
and for 𝐴 as defined in Equation 2: 𝐵 ∑︁ cos(2𝜋𝑛𝑡 ◦−1 𝑥) 𝑘 (𝑥) = 𝑛=−𝐵
−1 −1 𝑛=−𝐵 cos 2𝜋𝑛𝑡 ◦ 𝑥 𝑓 𝜎 2 2𝜋𝑛𝑡 ◦ Í𝐵 −1 𝑛=−𝐵 𝑓 𝜎 2 2𝜋𝑛𝑡 ◦
Í𝐵 𝑘 (𝑥) =
(11)
This is a periodic Gaussian similarity kernel. Supposing a sufficiently large 𝐵, the numerator and denominator of Equation 11 approximate a ratio of theta functions where each function is defined as: ∞ ∑︁ 2 𝜃 (𝑧, 𝜏) = 𝜏 𝑛 𝑒 2 𝑗𝑛𝑧 (12) 𝑛=−∞
1 2𝐵 + 1
𝐵 ∑︁ 1 cos(2𝜋𝑛𝑡◦−1 𝑥) = 2𝐵 + 1 𝑛=−𝐵
such that:
(6)
This is a normalized Dirichlet kernel with period 𝑡◦ , equivalently written as 𝐷ˆ 𝐵 (2𝜋𝑡◦−1 𝑥). This is identical to a repeating sinc kernel—the non-repeating variation of which is the kernel induced by uniform sampling over a continuous distribution of phases. Returning to 𝐾 (𝒙): 𝑛 Ö 𝐾 (𝒙) = 𝐷ˆ 𝐵 (2𝜋𝑡◦−1 𝑥𝑖 ) (7) 𝑖=1
Unfortunately, this 𝑛-dimensional kernel ish not symmeti
ric. The behaviour of the kernel for 𝒙 = 𝑐
√1 , . . . , √1 𝑛 𝑛
⊤
is different from that of the kernel for 𝒙 = 𝑐 [1, 0, . . . , 0] ⊤ where 𝑐 ∈ R. In the first case, the Dirichlet approximation is sinc𝑛 (𝑐), whereas in the second case, it is sinc(𝑐). The product of sinc functions is highly oscillatory nature, as an individual sinc function changes signs several times near 0 and has pronounced, negative lobes. These attributes make it less ideal for representing continuous, multidimensional feature spaces where equidistant features should be equally similar. Given Equation 5 as a generalized form, we can consider alternate repeating kernel shapes as well. As an example, we consider Normal-like sampling of phases similarly constrained to some frequency band 𝐵: 𝐴 ∼ Ω = 2𝜋𝑛𝑡 ◦−1 |𝑛 ∼ N 0, 𝜎 2 , 𝐵 (8) where 𝜎 ∈ R+ and N [0, 𝑎 2 , 𝑏] is a centered, pseudo-normal distribution over the integers from −𝑏 to 𝑏 inclusive with shape parameter 𝑎 2 —the construction of which is explained as follows. We calculate the PMF of N 0, 𝜎 2 , 𝐵 , 𝑝 [𝜔], by evaluating a centred normal distribution’s PDF with variance 𝜎 2 , 𝑓 𝜎 2 (𝑥), for the possible entries of 𝐴 and normalizing: 𝑝 [𝜔] = Í
(10)
𝜔∈Ω 𝑓 𝜎 2 (𝜔)
or equivalently:
However, since Ω is a discrete set for cyclic embeddings, 𝑝(𝜔) transitions from a PDF to a PMF, 𝑝 [𝜔], over the entries of Ω. Notably, 𝐴 is conjugate symmetric to ensure 𝜙 ∈ R𝑑 , resulting in every phase being matched with its conjugate: ∑︁ 𝑘 (𝑥) = 𝑒 − 𝑗 𝜔 𝑥 + 𝑒 𝑗 𝜔 𝑥 𝑝 [𝜔] ∑︁
Í
(4)
Ω
=
𝜔 ∈Ω cos (𝜔𝑥) 𝑓 𝜎 2 (𝜔)
𝑓 𝜎 2 (𝜔) 𝜔 ∈Ω 𝑓 𝜎 2 (𝜔)
(9)
𝜃 𝜋𝑡 ◦−1 𝑥, 𝑓 𝜎 2 2𝜋𝑛𝑡◦−1 𝑘 (𝑥) = 𝜃 0, 𝑓 𝜎 2 2𝜋𝑛𝑡 ◦−1
(13)
and considering 𝐾 (𝒙) for the periodic Gaussian kernel: Î𝑛 −1 −1 𝑖=1 𝜃 𝜋𝑡 ◦ 𝑥 𝑖 , 𝑓 𝜎 2 2𝜋𝑛𝑡 ◦ 𝐾 (𝒙) = (14) 𝑛 𝜃 0, 𝑓 𝜎 2 2𝜋𝑛𝑡 ◦−1 For the 𝑛-dimensional product-of-Gaussians kernel, 𝐾 (𝒙) defines a lattice of Gaussian-like similarity curves. While this multidimensional Gaussian kernel is radially asymmetric, similar to its Dirichlet equivalent, it is non-negative everywhere and smoother than the product-of-Dirichlets kernel. Figure 1 provides a visualization of both the normalized Dirichlet kernel and the normalized repeating Gaussian kernel using uniform and normal sampling respectively. 0
0
4
4
4
0.00 2 0.25 0.50 0.75 1.00
2
3 4
4
0.00 2 0.25 0.50 0.75 1.00
2
3 4
3 4
3 4
± 1.00 0.75 0.50 0.25 0.00 0.25
3 4
±
2
4
Normal Sampling
0
4
2
3 4
Uniform Sampling
Figure 1: Average approximations of the normalized periodic Gaussian and normalized Dirichlet kernels using normal and uniform sampling for 𝐴 ∈ R𝑑×1 respectively (𝑁 = 500, 𝑑 = 100, 𝐵 = 5, 𝑡◦ = 2𝜋, and 𝜎 = 1).
Conclusion In this brief account, we have laid out a way by which highdimensional representations can be constructed such that they are cyclic. Further, we have characterized the repeating kernels for high-dimensional feature spaces and provided a novel insight into how one can sample different distributions to create varying kernel shapes. We hope to apply these techniques to cognitive models in the near future, using representations of direction, orientation, and even stimulus perception like colour to create models capable of exhibiting human-like behaviours.
Acknowledgements The author would like to acknowledge and thank Nicole Sandra-Yaffa Dumont and Terrence Stewart for their technical assitance. The author would also like to thank Chris Eliasmith, Michael Furlong, Anna Penzkofer, and Madeleine Bartlett—all of whom were involved in discussions that motivated further examination of periodic distributed representations.
References Bochner, S. (1959, September). Lectures on Fourier Integrals. Princeton University Press. Frady, E. P., Kleyko, D., Kymn, C. J., Olshausen, B. A., & Sommer, F. T. (2021). Computing on Functions Using Randomized Vector Representations [Version Number: 1]. https://doi.org/10.48550/ARXIV.2109.03429 Komer, B., Stewart, T. C., Voelker, A. R., & Eliasmith, C. (2019). A neural representation of continuous space using fractional binding. Proceedings of the Annual Meeting of the Cognitive Science Society, 41, 2038–2043. https : //escholarship.org/uc/item/3zz346g1 Rahimi, A., & Recht, B. (2007). Random Features for LargeScale Kernel Machines. In J. Platt, D. Koller, Y. Singer, & S. Roweis (Eds.), Advances in Neural Information Processing Systems (Vol. 20). Curran Associates, Inc. https : / / proceedings . neurips . cc / paper _ files / paper / 2007 / file / 013a006f03dbc5392effeb8f18fda755-Paper.pdf Voelker, A. R. (2020). A short letter on the dot product between rotated Fourier transforms [Version Number: 1]. https://doi.org/10.48550/ARXIV.2007.13462