This geometry matters because downstream behavior is often decided by simple operations on these representations, such as linear heads, cosine similarity, nearest-neighbor retrieval, or clustering. If the representation space collapses into a narrow cone, cosine similarity becomes less informative. If fine-tuning concentrates label information in a few principal directions, removing those directions can destroy performance. If LoRA introduces new singular directions, those directions can explain both adaptation and forgetting.

This post gives a compact geometric view of pre-training and fine-tuning, and briefly covers scalable validation.

To make the discussion concrete, I include a small case study comparing distilbert-base-uncased with distilbert-base-uncased-finetuned-sst-2-english on GLUE/SST-2. You can reproduce this experiment and generate the visualizations using the code in the accompanying GitHub repository.

Transformer representations have geometry

Let $\mathcal V$ be a vocabulary and let $d$ be the hidden dimension. The input embedding matrix is

so each token index is mapped to a vector in $\mathbb{R}^d$. For a sequence of length $T$, the hidden states at layer $\ell$ are

A self-attention head computes

where $Q$, $K$, and $V_{\mathrm{val}}$ are query, key, and value projections of the current hidden states.

The model also has an output unembedding or language-model head

whose rows act as token classifiers. Given a hidden state $h$, the logit for token $v$ is typically

Some architectures tie $U$ and $E_{\mathrm{in}}$; others keep them separate.

There are two related geometries worth separating:

1. Activation geometry, which is the point cloud formed by hidden states $h_i$ across tokens, positions, layers, and contexts.
2. Weight geometry, or the geometry of learned matrices such as $E_{\mathrm{in}}$, $U$, attention projections, MLP weights, and low-rank adaptation matrices.

Fine-tuning changes both. Full fine-tuning moves all weights, while PEFT methods such as LoRA move only a small number of parameters, making the resulting geometric deformation easier to inspect.

Isotropy and anisotropy

A basic geometric question is whether a representation uses its dimensions evenly.

Given centered hidden states $X\in\mathbb{R}^{N\times d}$, with rows

the empirical covariance is

The representation is approximately isotropic when

meaning variance is spread roughly evenly across directions. Equivalently, no small set of principal components dominates the representation.

If $\lambda_1,\ldots,\lambda_d$ are the covariance eigenvalues, a useful summary statistic is the participation-ratio effective rank

This quantity is scale-invariant. It equals $d$ when all eigenvalues are equal and approaches $1$ when one direction carries almost all variance.

Transformer representations are often anisotropic, meaning that hidden states can share a large common component, occupy a narrow cone, or have a covariance spectrum dominated by a few directions. These are related but not identical phenomena. A large mean vector affects raw cosine similarity; a concentrated covariance spectrum affects effective rank and explained variance. A representation can look good under one diagnostic and degenerate under another.

This phenomenon of anisotropy in transformer representations is referred to as representation degeneration and has been observed in contextual word representations and language generation models^{[5], [6], [7]}. Similar effects have also been reported outside natural language text, including in protein language models, suggesting that anisotropy is a broader interaction among architecture, data distribution, objective, normalization, and optimization^[8].

A common symptom is high average cosine similarity between unrelated hidden states. If

and the common component $m$ is large relative to the variation $\varepsilon_i$, then many pairwise cosines become artificially high. This makes angular distance less useful for semantic discrimination.

However, it’s important to note that low average cosine is not evidence of isotropy. A simple counterexample makes this clear. Suppose

with roughly balanced signs. The covariance is rank one, so the representation is maximally anisotropic in the sense that all variation lies along the single direction $v$, and $r_{\mathrm{eff}}=1$. But the average pairwise cosine is close to zero, because pairs with the same sign contribute $+1$, pairs with opposite signs contribute $-1$, and the two effects mostly cancel. If the signs are exactly balanced, the average over distinct pairs is $-1/(N-1)$, essentially zero for large $N$.

Why anisotropy can arise

One mechanism comes directly from the softmax objective.

For an autoregressive language model,

where $h_c$ is the hidden state for context $c$. For a single training example with target token $y$, the cross-entropy gradient with respect to output vector $u_v$ is

For a rare token $v$, most contexts are negative examples, in which $v\neq y$. Gradient descent therefore repeatedly updates $u_v$ in directions that reduce its logit on the bulk of hidden states.

Gao et al. describe a related degeneration mechanism^[7]. Suppose that there exists a direction $a$ such that

for all relevant hidden states $h_c$. Then moving a rare token’s output embedding in the $a$ direction lowers that token’s logit across those contexts. A separating-hyperplane argument can supply such a direction when the origin lies outside the convex hull of the hidden-state cloud.

This helps explain why rare-token output embeddings can acquire common directions under softmax training. If input embeddings and output unembeddings are tied, this output-side pressure can also leak into the input embedding geometry.

This is not the only source of anisotropy, but it is an important example showing that degeneration can be a natural byproduct of the optimization process rather than a mere numerical accident.

Fine-tuning reorganizes geometry

Fine-tuning should not be understood as simply repairing pre-training anisotropy. It often reorganizes the representation space around the downstream objective.

In full fine-tuning, the model is initialized from pre-trained weights $\theta_{\mathrm{pre}}$, and all parameters are updated on a downstream dataset $\mathcal D_{\mathrm{FT}}$ by the rule

At the activation level, fine-tuning changes the covariance of hidden states. For centered samples from layer $\ell$,

define

The eigenvalues of $C^{(\ell)}$ describe how concentrated the representation is. The eigenvectors describe where that variance lies.

Before fine-tuning, leading principal components may encode broad or nuisance statistics such as token frequency, punctuation, sentence length, and so on. This helps explain why post-processing methods such as All-but-the-Top, which center embeddings and remove leading principal components, can improve some static embedding and similarity benchmarks^[9].

After fine-tuning, however, dominant directions can become task-relevant. In BERT-like models, fine-tuning may concentrate information useful for natural language inference, semantic textual similarity, or classification into high-variance directions^[3]. The representation may become less isotropic while becoming better for the task.

This provides evidence that removing top principal components is not automatically a cleanup step. If the top components encode nuisance structure, removing them can help, but if they encode label-relevant information, removing them can destroy performance.

A task-aware ablation is better. If $V_k=[v_1,\ldots,v_k]$ contains the top $k$ principal directions of the centered representation, test the intervention

on the downstream metric, and also test removing low-variance directions. Then, report geometry and task performance together. Isotropy alone is not the objective.

Case study of fine-tuning on DistilBERT’s geometry

The claim above is easy to state but easy to misuse. So let us look at a small diagnostic experiment.

Experimental setup

I compared two public checkpoints:

• distilbert-base-uncased, the pre-trained DistilBERT backbone^[15];
• distilbert-base-uncased-finetuned-sst-2-english, the same architecture fine-tuned for SST-2 sentiment classification.

The evaluation set was the full GLUE/SST-2 validation split, $N=872$ examples^{[16], [17]}. The script did not train either model; rather, it extracted the $[\mathrm{CLS}]$ hidden state at every hidden-state index, which is the embedding output plus the six DistilBERT transformer layers.

For each layer, it computed:

• mean norm $|\bar h|_2$;
• average raw pairwise cosine;
• average centered pairwise cosine;
• covariance eigenvalue spectrum;
• participation-ratio effective rank;
• top-$k$ explained variance;
• pre-trained/fine-tuned PCA subspace overlap;
• task accuracy after ablating principal components from the fine-tuned final-layer $[\mathrm{CLS}]$ states.

The embedding-layer $[\mathrm{CLS}]$ vector is constant across examples in this architecture, so its covariance is zero.

Summary metrics

The final-layer $[\mathrm{CLS}]$ geometry changed sharply after fine-tuning.

The pre-trained backbone has no SST-2 classification head, so I do not report a task accuracy for it. The fine-tuned checkpoint reaches $91.1\%$ accuracy on these 872 validation examples.

Raw average cosine drops from $0.917$ to $0.230$, which by itself might suggest that the fine-tuned representation is less anisotropic. But the covariance spectrum says the opposite, as the final fine-tuned $[\mathrm{CLS}]$ states are much more spectrally concentrated. The top principal component alone explains $78.3\%$ of the variance, and the top ten explain $90.3\%$. The effective rank falls from $25.82$ to $1.62$ out of $768$ dimensions.

This is not surprising. A representation can have near-zero average centered cosine while being almost one-dimensional, as long as examples are distributed on both sides of the dominant axis. For a sentiment classifier, that is a plausible geometry, as positive and negative examples may separate along a high-variance direction, causing pairwise cosine contributions to cancel in the average. This demonstrates concretely why relying solely on average cosine to measure anisotropy can be misleading.

Figure 1. Covariance spectra before and after fine-tuning. Eigenvalues are normalized by total variance. A steeper curve means variance is more concentrated in a small number of directions.

The subspace-overlap diagnostic reinforces this finding. For top-$k$ PCA bases $A$ and $B$, I measured

This equals $1$ for identical $k$-dimensional subspaces and decreases as the subspaces diverge. Ignoring the degenerate embedding layer, the top-10 pre-trained/fine-tuned overlap was about $0.69$ after the first transformer layer, $0.49$ after the fourth, and $0.20$ in the final layer. The top-50 overlap in the final layer was $0.35$. Fine-tuning therefore did not merely rescale the old leading directions; in fact, it substantially reorganized the principal subspaces in later layers.

Principal-component ablation

The decisive question is whether the new dominant directions are useful or harmful.

To test this, I modified the fine-tuned final-layer $[\mathrm{CLS}]$ states by removing principal components and then passed the modified states through the frozen DistilBERT classification head via the intervention

This intervention was applied only at the final representation, without rerunning the transformer. It tests how much the trained classifier depends on particular representation directions.

Removing the top principal component alone drops accuracy from $91.1\%$ to roughly chance. Removing additional top components does not substantially worsen it, because the main decision-relevant direction has already been removed. By contrast, removing as many as 100 bottom principal components leaves accuracy unchanged.

Figure 2. Task performance after principal-component ablation. Removing the top principal components tests whether high-variance directions encode nuisance structure or task-relevant information. Removing bottom components serves as a control.

This is the main lesson of the case study. The final fine-tuned representation is highly anisotropic in the centered covariance sense, but that anisotropy is not merely collapse. At least for the SST-2 classifier, the dominant direction is task-critical.

There is one methodological caveat. In this small blog experiment, PCA directions were estimated on the same validation activations that were later ablated. The labels were not used to fit the PCs, but a stricter benchmarking evaluation should fit PCA directions on a separate calibration split and evaluate ablations on held-out examples. The result should therefore be read as an illustrative diagnostic, not as a universal claim about DistilBERT or SST-2.

Takeaways from the case study

The experiment illustrates three main points.

1. Fine-tuning changed geometry measurably. Effective rank, covariance spectra, cosine statistics, and PCA subspaces all shifted, especially in later layers.
2. Raw cosine alone was misleading. The fine-tuned representation had much lower raw average cosine but a far more concentrated centered covariance spectrum.
3. Task-aware ablation was essential. The top principal direction looked like a collapse under an isotropy metric, but removing it destroyed task performance. In this run, the dominant direction was not disposable nuisance variation; it was central to the classifier.

LoRA as an inspectable low-rank weight change

The case study above looked at activation geometry under a fully fine-tuned checkpoint. LoRA gives an especially clean way to study geometry on the weight side.

Parameter-efficient fine-tuning methods reduce adaptation cost by training only a small number of parameters. The most common example is LoRA^[1].

For a frozen weight matrix

LoRA learns

where

with

Thus

Geometrically,

The update first reads an $r$-dimensional sketch $Ax$ of the input, then writes back through the column space of $B$. This makes LoRA easier to inspect because the learned change is concentrated in a small number of input and output directions.

Using the singular value decomposition,

one can ask:

• Which input directions $q_i$ does the adaptation read?
• Which output directions $p_i$ does it write to?
• Are these directions aligned with the pre-trained matrix $W_0$?
• Do they correspond to task improvement, forgetting, or both?

Recent work comparing LoRA and full fine-tuning identifies intruder dimensions in some adapted matrices^[11]. These are high-ranking singular directions of the adapted matrix $W_0+\Delta W$ that have large singular values but weak alignment with the corresponding singular subspaces of $W_0$.

Mathematically, if $\hat u_i,\hat v_i$ are singular vectors of the adapted matrix, one can measure their alignment with pre-trained left and right singular subspaces using projection norms such as

Large singular value combined with weak pre-trained alignment is a warning sign. In reported interventions, damping singular values associated with intruder directions partially restored pre-training behavior while preserving much of the downstream gain^[11].

A related issue appears in multimodal fine-tuning. Vision and text encoders can occupy separated regions of a shared latent space, a phenomenon known as a modality gap^[12]. Under extreme low-rank constraints, such as rank-1 LoRA, a random direction is unlikely to align with a fixed high-dimensional gap direction. If $a$ is a random unit vector in $d$ dimensions and $g$ is a fixed unit direction, then

so the useful projection is typically very small.

This matters because standard LoRA often initializes one factor to zero. For example, if $B=0$ and $A$ is random, then $\Delta W=0$ initially. If $G=\nabla_W\mathcal L$, then at initialization

Thus, early gradient flow is shaped by the random row space of $A$. In rank-1 settings, poor alignment can slow or stall adaptation. Geometry-guided methods such as Gap-Init estimate the modality-gap direction and initialize the LoRA direction accordingly^[2].

Architecture and regularization

Anisotropy is common, but it is not inevitable. It depends on architecture, objective, normalization, data, and optimization. Recent analyses using model suites such as Pythia are useful because they allow comparisons across model scale and training checkpoints^{[13], [14]}.

Untied embeddings and unembeddings

Weight tying makes the same token vectors serve two roles:

1. input embeddings for computation;
2. output classifiers for the softmax objective.

This is parameter-efficient, but it couples input geometry directly to output-classification pressure. Rare-token and softmax effects on $U$ can affect $E_{\mathrm{in}}$ when the matrices are tied, as covered in the Gao et al. discussion.

Untying $E_{\mathrm{in}}$ and $U$ gives the model separate degrees of freedom for input representation and output classification. In the cited Pythia-related analysis, untied embeddings correlate with improved global isotropy in larger models, at the cost of additional parameters^{[13], [14]}.

Final LayerNorm geometry

Layer normalization also has a clean geometric interpretation. For a hidden state $h\in\mathbb{R}^d$, ignoring the usual numerical $\epsilon$, LayerNorm computes

where

and $g,b\in\mathbb{R}^d$ are learned gain and bias parameters.

Before gain and bias, the normalized vector

satisfies

So LayerNorm first maps each vector to a fixed-radius sphere inside the hyperplane orthogonal to $\mathbf 1$. The learned gain and bias then stretch and translate this set.

If

and all $g_i\neq 0$, then

or equivalently,

Thus post-LayerNorm states lie in an affine hyperplane. A large shared offset from the bias term can increase raw cosine similarity by adding a common component. Conversely, a final LayerNorm that avoids a large shared offset can improve apparent isotropy relative to preceding layers^[13].

Strictly, because the post-LayerNorm states lie in an affine hyperplane, their centered covariance is singular in the normal direction

So isotropy here should be interpreted relative to the allowed $(d-1)$-dimensional subspace.

Geometry-aware regularization

When architecture cannot be changed, one can regularize geometry during training or fine-tuning.

A covariance-based approach adds an isotropy-related penalty to the task loss. For example, I-STAR uses shrinkage covariance estimates to define a stable differentiable isotropy score^[4]. A typical objective has the form

Contrastive learning and related regularizers can also spread representations more uniformly in some settings^[10].

The important caveat is that maximal isotropy is not always best. As established before, fine-tuned models may intentionally concentrate useful task information in high-variance directions. In these cases, controlled anisotropy can be a feature, not a flaw.

Scalable geometric validation

If representation geometry matters, it should be monitored during training and fine-tuning. The challenge is scale.

A full pairwise distance or cosine matrix costs $O(N^2)$ memory and $O(N^2d)$ time for $N$ vectors in $d$ dimensions. This becomes infeasible quickly.

Fortunately, many useful diagnostics do not require all pairwise distances.

Streaming covariance

Mean and covariance can be estimated in a streaming or batched way. If a batch summary contains count $n$, mean $\mu$, and scatter matrix

then two summaries $(n_a,\mu_a,S_a)$ and $(n_b,\mu_b,S_b)$ can be merged by setting

and

Then

Forming the full covariance costs $O(Nd^2)$ time and $O(d^2)$ memory. If only the top $k$ directions are needed, randomized PCA or streaming sketching can often reduce the cost to roughly $O(Ndk)$.

Average cosine without all pairs

Even average pairwise cosine similarity can be computed without enumerating all pairs. If $u_i$ are normalized vectors, then

This provides an $O(Nd)$ diagnostic for one common symptom of anisotropy. However, this statistic is weak as a general anisotropy measure. As discussed previously, and as evidenced by the DistilBERT case study, a low average pairwise cosine similarity can occur simultaneously with highly concentrated covariance. This suggests the need for other methods of sublinear geometric validation, but covering these is beyond the scope of this post.

A practical checklist

A useful validation pipeline should track geometry and task performance together. Some useful metrics to cover include:

1. Mean shift: $|\bar h|_2$ by layer and checkpoint.
2. Cosine symptoms: average raw cosine and centered cosine.
3. Spectral concentration: top eigenvalue mass, effective rank, and explained variance curves.
4. Subspace drift: overlap between pre-trained and fine-tuned principal subspaces.
5. Task-aware ablations: remove top PCs, remove bottom PCs, and evaluate the downstream metric.
6. LoRA diagnostics: inspect singular values of $\Delta W$ and $W_0+\Delta W$ and measure alignment with pre-trained singular subspaces.
7. Multimodal alignment: in vision-language settings, measure whether low-rank updates align with modality-gap directions.

The DistilBERT case study used all but the LoRA diagnostics and multimodal alignment from the above checklist. For LoRA fine-tuning, the same activation diagnostics can be combined with singular-direction diagnostics on the learned weight updates.

The goal of geometric monitoring of fine-tuning is not to maximize isotropy blindly. The goal is to distinguish harmful degeneration from meaningful task-specific geometry. A good fine-tuned model may be anisotropic because it has organized the representation space around the task, while a bad one may be anisotropic because it has collapsed. Geometry is how we tell the difference.

References

1. Hu, E. J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., & Chen, W. (2022). LoRA: Low-Rank Adaptation of Large Language Models. ICLR 2022.
2. Zhao, H., Han, C., & Hovy, E. (2026). When Is Rank-1 Enough? Geometry-Guided Initialization for Parameter-Efficient Fine-Tuning. ICML 2026.
3. Rajaee, S., & Pilehvar, M. T. (2021). How Does Fine-tuning Affect the Geometry of Embedding Space: A Case Study on Isotropy. Findings of ACL: EMNLP 2021.
4. Rudman, W., & Eickhoff, C. (2024). Stable Anisotropic Regularization. ICLR 2024.
5. Godey, N., Villemonte de la Clergerie, E., & Sagot, B. (2023). Is Anisotropy Inherent to Transformers? arXiv:2306.07656.
6. Ethayarajh, K. (2019). How Contextual are Contextualized Word Representations? EMNLP-IJCNLP 2019.
7. Gao, J., He, D., Tan, X., Qin, T., Wang, L., & Liu, T.-Y. (2019). Representation Degeneration Problem in Training Natural Language Generation Models. ICLR 2019.
8. Hakim, S. A., Roy, K., & Rahman, M. S. (2025). A Look at the Isotropy of Pretrained Protein Language Models. OpenReview preprint.
9. Mu, J., & Viswanath, P. (2018). All-but-the-Top: Simple and Effective Postprocessing for Word Representations. ICLR 2018.
10. Qiu, R., Huang, Z., Yin, H., & Wang, Z. (2022). Contrastive Learning for Representation Degeneration Problem in Sequential Recommendation. WSDM 2022.
11. Shuttleworth, R., Andreas, J., Torralba, A., & Sharma, P. (2024). LoRA vs Full Fine-tuning: An Illusion of Equivalence. arXiv:2410.21228.
12. Liang, W., Zhang, Y., Kwon, Y., Yeung, S., & Zou, J. (2022). Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning. NeurIPS 2022.
13. Machina, A., & Mercer, R. E. (2024). Anisotropy is Not Inherent to Transformers. Preprint manuscript.
14. Biderman, S., Schoelkopf, H., Anthony, Q. G., Bradley, H., O’Brien, K., Hallahan, E., Khan, M. A., Purohit, S., Prashanth, U. S., Raff, E., Skowron, A., Sutawika, L., & Van Der Wal, O. (2023). Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling. ICML 2023.
15. Sanh, V., Debut, L., Chaumond, J., & Wolf, T. (2019). DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. NeurIPS EMC$^2$ Workshop.
16. Wang, A., Singh, A., Michael, J., Hill, F., Levy, O., & Bowman, S. R. (2019). GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding. ICLR 2019.
17. Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., & Potts, C. (2013). Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. EMNLP 2013.

Fast Exponentiation

Consider the following problem: given some integer base $x$ and some non-negative integer exponent $n$, efficiently compute the value of $x^n$, or $x$ raised to the power of $n$. Of course, the brute force solution involves $n$ multiplications by $x$, which will have $O(n)$ runtime efficiency. However, we can do better.

The core insight is that we can repeatedly halve the exponent rather than multiplying the base $n$ times, leading to logarithmic time complexity. We can summarize the fast exponentiation algorithm, also known as the method of repeated squaring, with the following recursion^[1]:

In each recursive call, we solve the subproblem with exponent $n/2$ when $n$ is even, or with exponent $(n-1)/2$, when $n$ is odd. This gives us the recurrence relation

where $O(1)$ represents the constant time operations (one or two multiplications per call). By the master method^[1], this yields a $O(\log_2 n)$ time complexity.

Below is an example of the algorithm implemented iteratively in C++:

int power(int base, int exp) {
    int result = 1;
    while (exp > 0) {
        if (exp & 1) result = result * base; // If exp is odd, multiply by the base
        base = base * base;                  // Square the base
        exp >>= 1;                           // Halve the exponent
    }
    return result;
}

In effect, we’re processing the binary representation of $n$ bit by bit, from least significant (rightmost) to most significant (leftmost). We can write any non-negative integer $n$ in binary by expressing it as a sum of powers of $2$:

where each $b_i \in { 0, 1 }$ is the $i$th binary digit of $n$. For example,

where the subscript-$2$ notation denotes a binary representation. With this in mind, we can write

This latter formula works as a neat summary of the fast exponentiation algorithm. At iteration $i$, the variable base holds $x^{2^i}$. If $b_i = 1$ (the exponent’s $i$th bit is 1), the algorithm multiplies this into result. If $b_i = 0$, the algorithm skips this multiplication. In both cases, it squares base to move from $x^{2^i}$ to $x^{2^{i+1}}$ for the next iteration.

To make this process more clear, consider the example when we want to compute $3^{13}$, so $x = 3$ and $n = 13$. When we write $n$ in binary, we are expressing it as a sum of powers of $2$, so $13$ is represented in binary as $1101_2$:

The walkthrough of the algorithm’s execution for this example is as follows:

The final result we end up with is $243 \cdot 6561 = 1594323$.

Modular Exponentiation

We can extend the above approach to instead raise one number to a power modulo another number, which is known as modular exponentiation. Formally, given a non-negative integer base $x$, a non-negative integer exponent $n$ and a positive integer modulus $m$, we want to efficiently compute the value $x^n \text{ mod } m$. Modular exponentation is crucial in many number-theoretic computations, including in primality-testing and in the RSA (Rivest–Shamir–Adleman) cryptosystem.

Practically, efficient modular exponentiation involves the same algorithm as fast exponentiation, but with modulo operations added at each step. Note that these modulo operations can safely be applied without threatening the correctness of the result because the modulo operation is distributive over multiplication, meaning that

for non-negative integers $a, b$ and positive integer $m$.

Below is an example of modular exponentation implemented in C++:

int power(int base, int exp, int mod) {
    int result = 1;
    base %= mod;                                           // Initial modulo operation
    while (exp > 0) {
        if (exp & 1) result = (1LL * result * base) % mod; // Modulo after multiplying
        base = (1LL * base * base) % mod;                  // Modulo after squaring
        exp >>= 1;
    }
    return result;
}

By performing modulo operations at each step, we keep intermediate values manageable, reducing the cost of calculation and preventing a potential overflow.

LeetCode 2438. Range Product Queries of Powers

Finally, we apply modular exponentiation to a the LeetCode problem 2438. Range Product Queries of Powers. The problem statement is as follows:

Given a positive integer n, there exists a 0-indexed array called powers, composed of the minimum number of powers of $2$ that sum to $n$. The array is sorted in non-decreasing order, and there is only one way to form the array.

You are also given a 0-indexed 2D integer array queries, where queries[i] = [left_i, right_i]. Each queries[i] represents a query where you have to find the product of all powers[j] with left_i <= j <= right_i.

Return an array answers, equal in length to queries, where answers[i] is the answer to the ith query. Since the answer to the ith query may be too large, each answers[i] should be returned modulo $10^9 + 7$.

The brute force solution would involve performing up to $32$ multiplications for each query, because the length of the powers array is bounded above by $32$, and then performing a modulo operation on the result we get at the end. Not only does this require a high constant factor to process each query, but it also requires multiplying very large numbers, which is costly and risks overflow.

We can use a prefix product array to achieve efficient processing of each query. However, the naive implementation of a prefix product array without performing intermediate modulo operations still requires multiplying large numbers and therefore risks overflow, and we can’t just perform intermediate modulo operations because the modulo operation is not distributive over division.

To solve this problem, we can apply Fermat’s Little Theorem. The statement is as follows:

Let $m$ be prime and $x$ be an integer such that $x$ and $m$ are coprime. Then \(\begin{equation*} x^{m-1} \text{ mod } m = 1. \end{equation*}\)

Readers well versed in number theory will recognize that this is a special case of Euler’s Theorem, because for any prime $m$, we have $\phi(m) = m - 1$. From Fermat’s Little Theorem, it immediately follows that

In other words, we know that modulo $m$, $a^{m-2}$ is the multiplicative inverse of $a$, so we can write that

This means that, for any integer $b$ which is coprime to $m$, we have

Observe that $10^9 + 7$ is prime (this can be verified by the reader). Thus, we can apply the above result, efficiently precomputing the multiplicative inverse for each element in the prefix array using modular exponentiation.

The solution in its entirety is below, implemented in C++:

// OJ: leetcode.com/problems/range-product-queries-of-powers/description/
// Author: github.com/theoteske
// Time: O(Q)
// Space: O(1)
class Solution {
private:
    const int MOD = 1000000007;
    int modPower(long long base, long long exp) {
        int result = 1;
        base %= MOD;
        while (exp > 0) {
            if (exp & 1) result = (1LL * result * base) % MOD;
            base = (1LL * base * base) % MOD;
            exp >>= 1;
        }
        return result;
    }
public:
    vector<int> productQueries(int n, vector<vector<int>>& queries) {
        // Construct powers array
        vector<int> powers;
        for (int i = 0; i <= 31; ++i) {
            if ((1 << i) & n) powers.push_back(1 << i);
        }

        // Construct prefix product array
        const int P = powers.size();
        vector<int> prefix(P + 1, 1);
        for (int i = 1; i <= P; ++i) {
            prefix[i] = (1LL * prefix[i - 1] * powers[i - 1]) % MOD;
        }

        // Precompute multiplicative inverses under mod for each value in prefix
        vector<int> inverse;
        inverse.reserve(P + 1);
        for (int num : prefix) {
            inverse.push_back(modPower(num, MOD - 2));
        }

        // Process queries
        const int Q = queries.size();
        vector<int> ans(Q);
        for (int i = 0; i < queries.size(); ++i) {
            ans[i] = (1LL * prefix[queries[i][1]+1] * inverse[queries[i][0]]) % MOD;
        }

        return ans;
    }
};

References

1. Cormen, T. H., Leiserson, C. E., Rivest, R. L., & Stein, C. (2022). Introduction to Algorithms (4th ed.). MIT Press.

Math with LaTeX

Inline Math

We can write inline math like $E = mc^2$ or $\sum_{i=1}^{n} x_i$ directly in our text.

Display Math

For larger equations, we use display math:

Complex Equations

Here’s the quadratic formula:

Here is an aligned set of equations:

And here is a matrix example:

Code Examples

Python Code

def fibonacci(n):
    """Generate Fibonacci sequence up to n terms."""
    if n <= 0:
        return []
    elif n == 1:
        return [0]
    elif n == 2:
        return [0, 1]
    
    fib_sequence = [0, 1]
    for i in range(2, n):
        next_num = fib_sequence[i-1] + fib_sequence[i-2]
        fib_sequence.append(next_num)
    
    return fib_sequence

JavaScript Code

// Implementing a binary search tree
class Node {
    constructor(value) {
        this.value = value;
        this.left = null;
        this.right = null;
    }
}

class BinarySearchTree {
    constructor() {
        this.root = null;
    }
    
    insert(value) {
        const newNode = new Node(value);
        
        if (this.root === null) {
            this.root = newNode;
            return this;
        }
        
        let current = this.root;
        while (true) {
            if (value < current.value) {
                if (current.left === null) {
                    current.left = newNode;
                    return this;
                }
                current = current.left;
            } else {
                if (current.right === null) {
                    current.right = newNode;
                    return this;
                }
                current = current.right;
            }
        }
    }
}

Bash/Shell Script

#!/bin/bash
# Script to backup important files

SOURCE_DIR="/home/user/documents"
BACKUP_DIR="/backup/documents"
DATE=$(date +%Y%m%d_%H%M%S)

# Create backup directory if it doesn't exist
mkdir -p "$BACKUP_DIR"

# Perform backup
tar -czf "$BACKUP_DIR/backup_$DATE.tar.gz" "$SOURCE_DIR"

# Keep only last 7 backups
ls -t "$BACKUP_DIR"/backup_*.tar.gz | tail -n +8 | xargs rm -f

echo "Backup completed: backup_$DATE.tar.gz"

Algorithm Analysis

Let’s analyze the time complexity of the Fibonacci function above. The recursive implementation has a time complexity of $O(2^n)$, while our iterative approach has:

• Time Complexity: $O(n)$
• Space Complexity: $O(n)$

We can improve the space complexity to $O(1)$ by only keeping track of the last two numbers:

def fibonacci_optimized(n):
    if n <= 0:
        return 0
    elif n == 1:
        return 1
    
    prev, curr = 0, 1
    for _ in range(2, n + 1):
        prev, curr = curr, prev + curr
    
    return curr

Combining Math and Code

Consider the following implementation of gradient descent:

import numpy as np

def gradient_descent(X, y, theta, alpha, iterations):
    """
    Performs gradient descent to learn theta.
    
    Args:
        X: input features (m x n matrix)
        y: target values (m x 1 vector)
        theta: parameters (n x 1 vector)
        alpha: learning rate
        iterations: number of iterations
    """
    m = len(y)
    
    for i in range(iterations):
        # Hypothesis: h = X * theta
        h = X.dot(theta)
        
        # Calculate the cost J(theta)
        cost = (1/(2*m)) * np.sum((h - y)**2)
        
        # Calculate gradient
        gradient = (1/m) * X.T.dot(h - y)
        
        # Update parameters
        theta = theta - alpha * gradient
        
        if i % 100 == 0:
            print(f"Iteration {i}: Cost = {cost:.4f}")
    
    return theta

The cost function being minimized is:

Where $h_\theta(x) = \theta^T x$ is our hypothesis function.