Dual-Manifold Variational Autoencoders

Foundation-Model-Free Framework: Internalizing Regularization via Smooth Low-Dimensional Skeletons for High-Dimensional Latent Generative Modeling

Hongxu Chen* Chun Wang* Yanghao Wang Hongxiang Li Zhen Wang Zhixiang Zhang Long Chen†
The Hong Kong University of Science and Technology (HKUST)
*Equal contribution  ·  †Corresponding author
0.12 rFID
ImageNet-1K 256×256 reconstruction (f16d64, 30.76 dB PSNR) without external foundation models
1.20 / 1.17 gFID
Foundation-model-free FID50K w/ CFG (f16d64 / f16d32), 1.28 / 1.25 w/o CFG
1.12 OOD rFID
Zero-shot Anime-Faces-256 (32.87 dB PSNR), outperforming RAE (9.85) & VA-VAE (2.05)
rFID vs gFID scaling comparison
Overcoming the reconstruction-generation trade-off. On DiT-XL (80 epochs), scaling from d32 to d64 severely degrades SD-VAE ($10.59 \to 17.24$), whereas DM-VAE + $\text{IG}_{\text{TS}}$ reaches 0.12 rFID and 5.44 gFID10K without foundation models.
Qualitative comparison of OOD image reconstruction
Zero-shot OOD reconstruction (trained solely on ImageNet-1K): frozen external priors in RAE degrade severely on OOD domains (16.32 dB PSNR), whereas DM-VAE preserves crisp details (27.19 dB low branch, 30.10 dB high branch).

Abstract

In latent diffusion models, expanding the latent dimension of variational autoencoders (VAEs) is essential to push the upper bound of reconstruction quality. However, without explicit structural constraints, such unregularized expansion easily yields a rugged and ill-conditioned latent manifold, causing generated latents to drift off the manifold during generation (i.e., off-manifold problem). Recent works smooth this manifold by using external foundation models (e.g., DINOv2) as a topological regularizer. However, we argue that tying the latent space to frozen external priors inherently restricts generalization: bound by their pre-training distributions and semantic biases, they may fail to capture faithful visual details on out-of-distribution domains. Observing that low-dimensional latents naturally form a smoother, generation-friendly manifold, we propose dual-manifold VAE (DM-VAE) to leverage internal representations as a topological regularizer. Specifically, DM-VAE pairs a high-dimensional reconstructive branch with a low-dimensional branch that extracts a clean topological skeleton, anchoring the high-dimensional manifold to this smooth subspace. Building upon this framework, we further introduce topological representation alignment (TRA) to supervise intermediate generator features during training, and internal guidance with topological skeleton ($\text{IG}_{\text{TS}}$) to guide structural components while preserving high-frequency details at inference. On ImageNet-1K ($256\times256$), our foundation-model-free DM-VAE achieves a state-of-the-art rFID of 0.12 and a generation $\text{FID}_{50\text{K}}$ of 1.28 (w/o CFG) and 1.20 (w/ CFG), and further reaches 1.11 when orthogonally combined with DINOv2.

Comparison of reconstruction paradigms and DM-VAE framework
Comparison of reconstruction paradigms in LDMs. Unlike (a) vanilla VAEs that suffer from unconstrained, rugged high-dimensional manifolds, or (b) foundation-model-aligned and (c) foundation-model-based VAEs tied to frozen external priors (e.g., DINOv2) with limited out-of-distribution generalization, (d) our foundation-model-free DM-VAE constructs an internal topological regularizer. Compressing $z$ via $\mathcal{P}_{\text{down}}$ into a smooth low-dimensional branch ($z_{\text{low}}, \mathcal{D}_{\text{low}}$) and aligning $z$ with lifted $\hat{z} = \mathcal{P}_{\text{up}}(z_{\text{low}})$ compels the high-dimensional manifold to inherit a smooth, generation-friendly geometry (e).

Methodology & Latent Geometry Analysis

1. Dual-Manifold Alignment

Expanding channel capacity $C$ exponentially lowers the reconstruction distortion bound $\epsilon_{\text{recon}} \ge \alpha \exp(-\gamma \cdot C)$, yet embedding an intrinsically $d$-dimensional image manifold into $\mathbb{R}^{C\times H\times W}$ introduces exploding codimension $c = CHW - d$, scattering high-frequency energy across normal directions.

DM-VAE decomposes $z = z^{\text{base}} + z^{\text{detail}}$ via orthogonal $1\times 1$ projections ($\mathcal{P}_{\text{down}}, \mathcal{P}_{\text{up}}$), supervising $z_{\text{low}}$ solely with $\mathcal{L}_{\text{rec}}$ (without GAN loss) while minimizing $\mathcal{L}_{\text{DM}} = \mathbb{E}_x[\|z - \hat{z}\|_2^2]$. This concentrates $\mathcal{M}_Z$ within a narrow tubular neighborhood around smooth $\mathcal{M}_{\text{base}}$ ($|\cos(z^{\text{base}}, z^{\text{detail}})| = 0.0047$) and decouples temporal SNR transition windows ($t_{\text{base}}^* = 0.43$ vs. $t_{\text{detail}}^* = 0.91$).

Spectral Decomposition and Empirical Temporal SNR Decoupling
Empirical verification of spectral decomposition and temporal SNR decoupling on 5K ImageNet validation images (f16d64, $C_{\text{low}}=32$). (a) 2D Fourier PSD confirms skeleton $\mathcal{D}(\hat{z})$ preserves global low-frequency structures ($k \le 8$), while detail residual $\mathcal{D}(z) - \mathcal{D}(\hat{z})$ rises by over 44 dB toward high frequencies. (b) While SD-VAE suffers heavy base-detail overlap ($t_{\text{detail}}^* = 0.68$), DM-VAE pushes the detail transition peak to $t_{\text{detail}}^* = 0.91$, isolating structural velocity learning within $[0, 0.6]$.

2. TRA & Skeleton Guidance ($\text{IG}_{\text{TS}}$)

Training (TRA): Intermediate DiT hidden features $h_t$ are mapped via $\mathcal{P}_{\text{align}}$ to $\tilde{z}_{\text{low}}$ and aligned with the clean skeleton $z_{\text{low}} = \mathcal{P}_{\text{down}}(z_1)$ via cosine similarity $\mathcal{L}_{\text{TRA}}$, establishing early structural velocity priors free from high-frequency gradient variance.

Inference ($\text{IG}_{\text{TS}}$): Within a single forward pass (zero doubled-NFE overhead), $\text{IG}_{\text{TS}}$ separates $\hat{z}_1 = \hat{z}_1^{\text{base}} + \hat{z}_1^{\text{detail}}$, guides only the skeleton with norm-calibrated $\tilde{z}_1^{\text{base}}$, and adds unperturbed $\hat{z}_1^{\text{detail}}$ back: $$\hat{z}_1^{\text{guided}} = \hat{z}_1^{\text{base}} + (\omega - 1)(\hat{z}_1^{\text{base}} - \tilde{z}_1^{\text{base}}) + \hat{z}_1^{\text{detail}}$$

Overview of TRA and IG_TS
Overview of TRA and $\text{IG}_{\text{TS}}$. Aligning intermediate features to $z_{\text{low}}$ enables single-pass skeleton guidance while preserving high-frequency details $\hat{z}_1^{\text{detail}}$.
Algorithm 1: Internal Guidance with Topological Skeleton
Algorithm 1 ($\text{IG}_{\text{TS}}$). Single-pass velocity and skeleton prediction, norm calibration, and detail-preserving latent update.

Experimental Results

1. In-Domain & Zero-Shot Out-of-Distribution (OOD) Reconstruction

Method In-Domain rFID↓ PSNR↑ OOD rFID↓ OOD PSNR↑
RAE0.5719.209.8517.53
RAEv20.3123.295.0922.20
MAE-VAE (f16d32)0.2828.332.3829.78
VA-VAE (f16d32)0.2827.962.0529.33
SD-VAE (f16d32)0.2628.592.0830.19
DM-VAE (f16d32, Ours)0.2128.501.9330.30
DM-VAE (f16d64, Ours)0.1230.761.1232.87

2. Class-Conditional Generation on ImageNet-1K ($256\times256$)

Method Epochs #Params FID50K↓ (w/o CFG) sFID↓ (w/o CFG) IS↑ (w/o CFG) FID50K↓ (w/ CFG) sFID↓ (w/ CFG) IS↑ (w/ CFG)
Latent Diffusion with External Self-Supervised Representation Models
REPA800675M5.905.69157.81.424.70305.7
REG480677M2.204.67219.11.404.24296.9
VA-VAE (LightningDiT)800675M2.174.36205.61.354.15295.3
RAE (DiTDH)800839M1.605.40242.71.284.72262.9
REPA-E800675M1.834.17217.31.264.11314.9
DDT-XL400675M6.27–154.71.26–310.6
IG (VA-VAE)680675M1.343.94229.31.194.11269.0
DiT-XL (DM-VAEDino, f16d32)60675M2.264.81200.31.364.12276.3
DiT-XL (DM-VAEDino, f16d32)400675M1.294.28253.31.114.03297.4
Foundation-Model-Free Latent Diffusion (No External Self-Supervised Priors)
DiT-XL/21400675M9.626.85121.52.274.60278.2
MaskDiT1600675M5.6910.34177.92.285.67276.6
SiT-XL/21400675M8.616.32131.72.064.50270.3
TREAD740675M–––1.694.73292.7
MDTv2-XL/21080675M–––1.584.52314.7
DiT-XL (DM-VAE, f16d32)64675M2.624.62190.11.674.12253.4
DiT-XL (DM-VAE, f16d32)400675M1.254.16250.81.174.02283.0
DiT-XL (DM-VAE, f16d64)400675M1.284.16245.71.204.10274.8

BibTeX

@article{chen2026dmvae,
  title   = {Dual-Manifold Variational Autoencoders},
  author  = {Chen, Hongxu and Wang, Chun and Wang, Yanghao and Li, Hongxiang and Wang, Zhen and Zhang, Zhixiang and Chen, Long},
  journal = {arXiv preprint},
  year    = {2026}
}