Retrieve What's Missing: CovRAG (NeurIPS 2026)

01

Abstract

Long autoregressive video models forget what they saw. CovRAG retrieves the past frames that cover what the temporal context cannot see.

Maintaining long-term geometric consistency remains challenging for long-horizon autoregressive video generation. Memory-augmented generative models address this by retrieving historical frames, but their effectiveness depends on two key design choices: what 3D-geometric evidence should represent past observations, and how memory frames should be selected from this evidence. Existing methods often rely on camera poses or field-of-view overlap, which are lightweight but too coarse to reason about pixel-wise visibility, or use explicit 3D reconstruction, which provides fine-grained evidence but is costly to maintain over long rollouts. We propose Coverage-Maximizing Retrieval-Augmented Generation (CovRAG), a depth-based memory retrieval framework that uses pretrained 3D priors to construct a target-view coverage map as lightweight 3D memory evidence. For frame selection, CovRAG maximizes residual coverage gain, iteratively retrieving frames that explain target-view regions not covered by the current context or previously selected memories. To improve scalability in long-video generation, we introduce sliding-window depth caching for efficient geometry estimation. Experiments on RealEstate10K and DL3DV10K show that CovRAG improves long-horizon geometric consistency while maintaining low latency compared to baselines.

Figure 1Consistency under occlusion and out-of-frame motion. Scrub the timeline or jump to the marked moments. Content that disappears and later reappears is preserved, with a better consistency–efficiency trade-off than FoV-based retrieval (WorldMem) and explicit 3D memory (VMem).

02

Method

Before each frame is generated, CovRAG marks which target-view pixels the temporal context already supports, then retrieves the memories that cover the rest.

i

Target-view coverage maps

Every past frame is warped into the target view with VGGT depth and pose. A target pixel is covered if any source pixel lands on it; context maps are merged with a logical OR.

$$\mathbf m^{\mathrm{ctx}}_t=\bigvee_{i\in\mathcal C_t}\mathbf m_{i\to t}$$

ii

Residual coverage gain

Each candidate is scored only by the uncovered pixels it would explain; the best one is retrieved, its coverage added, and the search repeats. Redundant frames score zero.

$$g_j=\sum_{\mathbf v}\bigl(1-\mathbf m^{\mathrm{ctx}}_t(\mathbf v)\bigr)\,\mathbf m_{j\to t}(\mathbf v)$$ $$j^\star=\arg\max_{j\in\mathcal M_t\setminus\mathcal H_t} g_j$$

iii

Sliding-window depth caching

VGGT runs only on the latest L frames; each new frame’s depth is cached with the memory and reused for warping, so geometry cost per step stays bounded by L.

$$\{\mathbf d_i\}_{i\in\mathcal W_t}=\mathrm{VGGT}\bigl(\mathbf x^0_{\mathcal W_t}\bigr)$$ $$\mathcal W_t=\{t-L+1,\dots,t\}$$

03

Results

More consistent long-horizon rollouts on both datasets. Every method shares the same backbone and memory module; only the retrieval rule differs.

Method	Consistency				Quality
Method	MEt3R ↓	LPIPS ↓	PSNR ↑	SSIM ↑	FVD ↓	FID ↓
RealEstate10K · loop closing, T = 40
DFoT (no memory)	0.415	0.588	11.7	0.372	265.03	29.88
+ WorldMem	0.144	0.197	18.2	0.555	242.36	24.77
+ VMem	0.120	0.174	18.6	0.568	231.77	24.05
+ Spatia	0.140	0.204	17.9	0.575	228.68	22.60
+ CovRAG (ours)	0.102	0.154	18.9	0.576	226.40	23.00
DL3DV10K · original trajectories, T = 100
DFoT (no memory)	0.430	0.529	14.5	0.394	706.70	88.01
+ WorldMem	0.316	0.401	16.2	0.436	428.49	58.24
+ VMem	0.315	0.380	16.3	0.447	394.79	57.61
+ Spatia	0.327	0.416	15.9	0.434	439.64	56.10
+ CovRAG (ours)	0.224	0.308	17.9	0.504	321.81	46.92

Table 1Long-horizon revisitation consistency (MEt3R, LPIPS, PSNR, SSIM) and generation quality (FVD, FID). Best in burgundy.

DL3DV10K · 100-frame rollouts

Long, spatially extended camera paths through diverse scenes. Baselines drift and deform structures; CovRAG keeps the layout.

RealEstate10K · loop closing

The camera returns to its first view after a long gap. CovRAG reproduces the revisited room; baselines deform or shift it.

The white sofa seen at frame 0 is deformed by WorldMem on revisit (frame 40) and preserved by CovRAG. Drag the divider to compare the revisited view.

The doorway structure shifts with VMem on revisit, while CovRAG keeps the geometry of the first view. Drag the divider to compare.

More CovRAG rollouts

100-frame autoregressive generations along original DL3DV10K trajectories.

Plug-in retrieval beyond DFoT

CovRAG replaces the memory-selection rule of existing pipelines without retraining.

Dynamic scenes · WorldPlay-8B on SpatialVID

Method	PSNR ↑	SSIM ↑	LPIPS ↓	FVD ↓
WorldPlay-8B	14.3	0.390	0.373	418.44
+ CovRAG	14.9	0.411	0.333	376.83

3D conditioning · GEN3C on RealEstate10K

Method	PSNR ↑	SSIM ↑	LPIPS ↓	MEt3R ↓
GEN3C	14.2	0.572	0.519	0.377
+ CovRAG	17.6	0.675	0.349	0.251

04

Analysis

Near-constant retrieval cost, and both design choices matter: fine-grained evidence and redundancy-aware selection.

Figure 3Retrieval latency per step. Explicit 3D memory grows with the rollout; CovRAG stays nearly constant thanks to sliding-window depth caching. End to end, a 100-frame video takes 8.70 s with CovRAG, 8.43 s with WorldMem and 45.40 s with VMem.

ID	Evidence	Selection	MEt3R ↓	LPIPS ↓
(a)	FoV overlap	Independent	0.141	0.198
(b)	FoV overlap	Residual	0.130	0.187
(c)	Coverage	Independent	0.149	0.210
(d)	Coverage	Residual	0.100	0.156

Table 2Ablation on RealEstate10K. Coverage evidence alone is redundant under independent top-n selection (c); combined with residual gain it is best (d).

Figure 4Retrieval budget on RealEstate10K. CovRAG is strong with a single retrieved frame (MEt3R 0.107 vs. 0.413 for WorldMem) and saturates at n = 2, since residual gain avoids redundant memories.

05

Citation

@inproceedings{joo2026covrag,
  title     = {Retrieve What's Missing: Coverage-Maximizing Retrieval
               for Consistent Long Video Generation},
  author    = {Joo, Minseok and Park, Dogyun and Lee, Taehoon and
               Lee, Kyujin and Kim, Hyunwoo J.},
  booktitle = {Advances in Neural Information Processing Systems, NeurIPS},
  year      = {2026}
}