Parameter-efficient finetuning methods based on spectral decomposition have enabled progress in continual learning. In this paper we introduce Spectral-Tail, which utilizes the singular bases U and V of the pre-trained weights as a fixed reference frame to learn a low-rank update applied to the singular value matrix. A soft spectral penalty discourages updates aligned with the dominant singular directions, reducing interference while routing fine-grained adaptation into the long-tail coordinates.

1  Introduction

Large Language Models (LLMs) have achieved remarkable performance across diverse reasoning and generation tasks (Zhao et al., 2023; Minaee et al., 2024). However, adapting these models to new domains remains computationally expensive, as full fine-tuning requires updating billions of parameters.

Among PEFT approaches, Low-Rank Adaptation (LoRA) (Hu et al., 2021) has emerged as one of the most widely adopted. Motivated by the evidence that task-specific updates lie in a low-dimensional subspace (Li et al., 2018), LoRA freezes the pretrained weights and learns two trainable low-rank matrices.

Existing low-rank methods often suffer from interference between overlapping update directions, especially when models are adapted across sequential tasks. Since the largest singular values encode the most critical structure, modifications there disproportionately degrade prior knowledge.

To mitigate this, we propose a spectral regularization scheme that selectively penalizes updates to the dominant singular components while allowing greater flexibility in the lower-rank "tail". Our specific contributions are as follows:

• We introduce Spectral-Tail, a low-rank adaptation method operating over the singular values of a weight matrix, coupled with a soft regularization that steers updates toward the spectral tail.
• Different from existing continual PEFT methods (Das et al., 2026; Wang et al., 2023a), it requires no access to adapters from prior tasks, preserving the privacy of each user's task-specific data.
• We evaluate on a suite of continual learning tasks, matching state-of-the-art methods while increasing the stable rank of the weight matrix.

2  Related Work

Spectral LoRA variants. Leveraging the spectral properties of base weights W is a key strategy in PEFT. Many SVD-based approaches (Meng et al., 2024; Lingam et al., 2024) partition the spectrum to align trainable updates with the structure of pretrained matrices for more efficient tuning.