Diff2I2P: Differentiable Image-to-Point Cloud Registration with Diffusion Prior

作者：Juncheng Mu， Chengwei Ren，Weixiang Zhang， Liang Pan，Xiao-Ping Zhang，Yue Gao

单位：Tsinghua University, Shanghai AI Laboratory

会议：2025 ICCV

链接： https://github.com/mujc2021/Diff2I2P

研究动机

1.当前的Image-to-Point Cloud的方法主要依靠度量学习来强制对齐图像和点云的特征空间，而忽略了两个模态之间存在的模态差距

2.非对齐的图像-深度会导致预训练扩散模型的预测能力下降，产生严重的生成伪影

diffi2p-1

3.梯度无法反向传播至骨干网络

核心方法

diffi2p-0

1.可微标定

利用KPConv和ResNet分别提取特征，然后采用多尺度块匹配策略来缓解尺度不匹配问题。在选取相似度最高的前 K 个点-像素对之后，得到初始的对应点

可变性对应点调整：对于每一对初始的对应点，在gt的转换下，DCT都预测一个点偏移来对齐对应点：

$C = \{(x_i + \Delta p_i, y_i) \mid (x_i, y_i) \in C_{\text{init}}\}$

采用BPnP作为可微求解器

2.控制侧分数蒸馏

diffi2p-2

将点云转换到图像的平面并投影为深度图，然后用可微的致密化操作将稀疏的深度图致密化

$\mathcal{L}_{\text{Diff}}(z_t, D) = \mathbb{E}_{\epsilon \sim \mathcal{N}(0,I),\, t \sim \mathcal{U}(0,1)} \left[ w(t) \| m \circ (\hat{\epsilon}_\phi(z_t, D; t) - \epsilon) \|_2^2 \right]$ $\nabla_\theta \mathcal{L}_{\text{Diff}}(z_t, D(\theta)) = \mathbb{E}_{t,\epsilon} \left[ w(t) \cdot m \circ (\hat{\epsilon}_\phi(z_t, D) - \epsilon) \cdot m \circ \underbrace{\frac{\partial \hat{\epsilon}_\phi(z_t, D)}{\partial D} \frac{\partial D}{\partial d}}_{\text{size matched Jacobian}} \cdot \frac{\partial d}{\partial \theta} \right]$ $\nabla_\theta \mathcal{L}_{\text{CSD}}(x, D(\theta)) = \mathbb{E}_{t,\epsilon} \left[ w(t) \cdot m \circ (\hat{\epsilon}_\phi(x, D; t) - \epsilon) \frac{\partial d}{\partial \theta} \right]$

3.损失函数

偏移损失：

$L_d^i = \left\| \left( \bar{R}(x_i + \Delta p_i) + \bar{t} \right) - K^{-1}(y_i; K) \right\|_2^2$ $L_r^i = \|\Delta p_i\|_2^2, \quad L_o = \frac{1}{|C|} \sum_{i=1}^{|C|} (L_d^i + \mu L_r^i)$

度量学习损失：

$\mathcal{L}^i = \frac{1}{\delta} \log \left[ 1 + \sum_{d_j \in D_i^P} e^{\beta_{i,j}^p (d_j^i - \Delta_p)} \cdot \sum_{d_k \in D_i^N} e^{\beta_{i,k}^n (\Delta_n - d_k^i)} \right]$ $\beta_{i,j}^p = \delta \lambda_{i,j}^p \left( d_j^i - \Delta_p \right), \quad \beta_{i,k}^n = \delta \lambda_{i,k}^n \left( \Delta_n - d_k^i \right)$

总损失：

$\mathcal{L} = \alpha \mathcal{L}_{\text{feat}} + \beta \mathcal{L}_{\text{offset}} + \gamma \mathcal{L}_{\text{CSD}}$