Breaking the Model Forgetting Cycle in
Long-Incremental 3D Object Detection

Peisheng Qian^1,2, Jie Xu¹, Xulei Yang^2,*, Na Zhao^1,*

¹Singapore University of Technology and Design ²Institute for Infocomm Research, A*STAR, Singapore

^*Corresponding authors

ECCV 2026

Long-incremental 3D object detection task illustration and result comparison — **Task illustration and result comparison.** (a) Long-incremental 3D object detection has the continuous incremental learning stages, where novel class annotations are available at the current stage; objects from previously learned old classes are unlabeled in the later stages. (b) Performance comparison over multiple long-incremental 3D detection tasks, showing our method consistently outperforms baseline approaches.

Abstract

Incremental 3D object detection requires a detector to learn novel object classes while remembering previously learned ones over sequentially arriving data. Previous methods, primarily based on pseudo-labeling, perform reasonably in short-incremental stages but still suffer from severe model forgetting when dealing with long-incremental sequences. We investigate this failure and reveal a detrimental self-reinforcing cycle: data distribution shift of novel classes causes model forgetting on old classes, which further produces accumulated error in pseudo-labeling that exacerbates model degradation. To address this issue, we draw inspiration from the human learning process and propose the Learning-Dynamics-driven Memory and Review (LDMR) framework. LDMR monitors per-class detection quality at periodic training checkpoints and uses these learning-dynamics signals to drive two innovative mechanisms, namely (i) human-like intra-stage review that divides each incremental stage into multiple sub-stages' training and concentrates on remembering the most-forgotten objects, and (ii) scene-aware cross-stage memory evolution that evolves a memory bank to transfer knowledge between two consecutive stages by jointly considering scene learnability and diversity. Extensive experiments across multiple long-incremental protocols on indoor benchmarks SUN RGB-D and ScanNetV2 show that LDMR substantially mitigates the model forgetting and outperforms all baselines by a clear margin. Code is available at https://github.com/qianpeisheng/LDMR.

Method

The forgetting cycle. In long-incremental 3D object detection, each stage introduces a new set of classes and the old classes are left unlabeled. Our analysis shows that the training distribution shifts substantially from stage to stage, and that this shift raises the empirical risk of the detector on old classes. Because pseudo-labeling methods supervise old classes with the predictions of the previous-stage model, a degraded model produces degraded pseudo-labels, which in turn degrade the next model. Distribution shift, model forgetting and pseudo-label error therefore form a self-reinforcing cycle that short-incremental protocols never expose.

Human-like intra-stage review. LDMR splits each incremental stage into I sub-stages and, at each sub-stage checkpoint, evaluates the current model on the memory bank to record its learning dynamics: the per-class recall Rⁱ_t(s, c) of every class in every memory scene. Comparing consecutive checkpoints gives a scene-level forgetting score F_s = Σ_c log(1 + n_s,c) · max(0, Rⁱ_t − Rⁱ⁺¹_t), which weights each class by how many of its objects the scene contains. The sampling weight of a memory scene in the next sub-stage is then w_s = 1 + η · F_s, so training time is redirected towards the objects the model is currently forgetting most, exactly as a student reviews the material they just failed.

Scene-aware cross-stage memory evolution. At the end of a stage, the same learning-dynamics signals decide which scenes are carried forward under a fixed memory budget. A scene's learnability aggregates, over its classes, a class under-learning weight ω_t(c), the log object count log(1 + n_s,c), and the learning gain G_t(s, c) observed across the sub-stage checkpoints. Its diversity is measured in a two-dimensional, L2-normalised learning-dynamics embedding e_t(s), so that the bank does not collapse onto scenes that behave alike. The two scores are combined as (1 − λ) · learnability + λ · diversity; the highest-scoring candidate enters the bank and the lowest-scoring resident leaves it. The bank thus keeps evolving into a compact set of scenes that is both informative and varied, and knowledge is transferred between consecutive stages without storing the full history.

Empirical analysis of long-incremental 3D object detection on SUN RGB-D — **Empirical analysis of L-I3DOD on SUN RGB-D (40 classes).** (a) Training data variation indicates the distribution shift over long-incremental stages. (b) Training loss or empirical risk evaluated on old classes at each stage. (c) The quality of pseudo-labels generated by each stage's model.

Results

Table 1. Incremental 3D object detection performance (mAP@0.25) over 3-/5-/10-incremental stages on SUN RGB-D and ScanNetV2 benchmarks.

Method	Backbone	SUN RGB-D (40 classes)			ScanNetV2 (35 classes)
Method	Backbone	3-stage	5-stage	10-stage	3-stage	5-stage	10-stage
Fine-tuning	VoteNet [19]	1.12	0.62	0.03	5.09	4.21	0.15
CPDet3D [43]	VoteNet	9.70	7.11	0.39	16.33	9.05	2.54
AIC3DOD [4]	VoteNet	10.73	7.32	0.85	19.21	15.51	1.60
SDCoT [40]	VoteNet	10.94	7.66	0.98	18.39	14.38	2.93
SDCoT++ [41]	VoteNet	11.39	7.86	1.67	18.51	16.19	3.38
SDCoT++ & Random Memory	VoteNet	11.52	7.93	3.53	21.19	19.46	7.25
LDMR (ours)	VoteNet	11.87	8.75	6.20	24.82	23.37	9.76
Joint Training (Upper Bound)	VoteNet	22.39	22.39	22.39	36.00	36.00	36.00
Fine-tuning	TR3D [22]	7.86	2.37	0.59	11.49	6.24	0.50
CPDet3D [43]	TR3D	8.71	7.55	3.32	18.76	7.26	0.66
AIC3DOD [4]	TR3D	22.43	15.37	9.11	32.69	20.11	8.66
SDCoT [40]	TR3D	22.10	13.77	8.72	34.65	20.86	6.93
SDCoT++ [41]	TR3D	22.92	14.28	8.45	35.15	20.61	7.06
SDCoT++ & Random Memory	TR3D	25.18	20.02	15.54	36.86	23.67	14.86
LDMR (ours)	TR3D	29.12	25.10	19.38	39.00	28.38	17.82
Joint Training (Upper Bound)	TR3D	33.26	33.26	33.26	51.52	51.52	51.52

Numbers are mAP@0.25 over all seen classes after the final stage. The gap to the previous best method widens as the sequence gets longer: with the TR3D backbone on SUN RGB-D, LDMR gains +3.94 / +5.08 / +3.84 mAP over SDCoT++ & Random Memory at 3 / 5 / 10 stages.

Stage-wise mAP and average forgetting on SUN RGB-D with TR3D — **Stage-wise experimental analysis on SUN RGB-D (with TR3D).** Left: Overall mAP on all seen classes after each stage. Right: Average forgetting computed by Eq. (15) of old classes at each stage.

Ablations

Table 2. Ablation study on our key modules of human-like intra-stage review (HR) and scene-aware cross-stage memory evolution (ME) (SUN RGB-D, TR3D, 3-/5-/10-stages' mAP@0.25).

Method	3-stage	5-stage	10-stage
LDMR (full)	29.12	25.10	19.38
w/o HR	26.67	23.47	17.51
w/o ME	27.44	23.17	18.70
w/o HR & ME	25.18	20.02	15.54

Table 3. Comparison of alternative memory-bank schemes in SDCoT++ & Random Memory (SUN RGB-D, TR3D, 10-stage's mAP@0.25).

Memory-bank scheme	mAP@0.25
Random	15.54
Reservoir	16.12
Max #objects	15.31
Lowest recall	14.10
LDMR	19.38

Table 4. Ablation on the scoring-function design choices (SUN RGB-D, TR3D, 10-stage mAP@0.25). Each variant alters one component of LDMR.

Component	Design choice	mAP@0.25
Count weight (Eqs. 4, 9)	linear: n_s,c	18.55
	sublinear: √n_s,c	19.01
	log: log(1 + n_s,c) (final design)	19.38
Learnability (Eq. 9)	w/o ω_t(c) (class under-learning weight)	19.13
	w/o log(1 + n_s,c) (object-count weight)	17.86
	w/o G_t(s, c) (learning gain)	18.88
Under-learning weight	low-recall only	18.45

Qualitative results

Qualitative comparison on SUN RGB-D over 10 incremental stages — Qualitative comparison of incremental 3D object detection performance over 10 incremental stages on SUN RGB-D (with TR3D).

Qualitative comparison on ScanNetV2 over 10 incremental stages — Qualitative comparison of incremental 3D object detection performance over 10 incremental stages on ScanNetV2 (with TR3D).

Per-class AP@0.25 across 10 incremental stages — Per-class incremental 3D object detection performance (AP@0.25) across 10 incremental stages.

Failure cases on ScanNetV2 and SUN RGB-D.

BibTeX

@inproceedings{qian2026ldmr,
  title     = {Breaking the Model Forgetting Cycle in Long-Incremental 3D Object Detection},
  author    = {Qian, Peisheng and Xu, Jie and Yang, Xulei and Zhao, Na},
  booktitle = {Proceedings of the European Conference on Computer Vision (ECCV)},
  year      = {2026},
  publisher = {Springer},
  % TODO: add pages and DOI once the ECCV 2026 proceedings are published
  % pages  = {--},
  % doi    = {10.1007/---}
}

Acknowledgement

This research is supported by the Agency for Science, Technology and Research (A*STAR) under its MTC Programmatic Funds (Grant No. M23L7b0021), and the Ministry of Education, Singapore, under its MOE Academic Research Fund Tier 2 (MOE-T2EP20124-0013).

Breaking the Model Forgetting Cycle inLong-Incremental 3D Object Detection