NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis

Neural Radiance Field Scene Representation

输入：连续的5D坐标（空间位置+视角方向）

$$3 \mathrm{D ~ l o c a t i o n ~} \mathbf{x}=( x, y, z )$$ $$\mathrm{2 D ~ v i e w i n g ~ d i r e c t i o n ~} ( \theta, \phi)$$

输出：空间位置上的体密度和辐射强度

$$\mathrm{emitted color}~{\bf c}=( r, g, b )$$ $$\mathrm{v o l u m e \ d e n s i t y} \ \sigma$$

密度控制了穿过某一位置的光线在那个位置累积了多少辐射

方法简单总结：

1）将相机光轴穿过场景生成采样的三维点集

2）用三维点和对应的二维视角方向输入神经网络生成颜色和密度集

$$F_{\Theta} : ( {\bf x}, {\bf d} ) \to( {\bf c}, \sigma)$$

3）使用传统的体渲染技术将这些颜色和密度积分到二维图像

优化目标：减小所观察到的图像与渲染得到的对应视角图像之间的误差

神经辐射场表示的基本实现不能够达到足够高的分辨率，同时对样本的使用效率较低

将输入5D坐标转换为位置编码，是的MLP能够表示更高频率的函数；

采用分层采样，减少对高频场景表示进行充分采样所需的queries

确保多视角一致性

We encourage the representation to be multiview consistent by restricting the network to predict the volume density $σ$ as a function of only the location$ x$, while allowing the RGB color $c $to be predicted as a function of both location and viewing direction.

the MLP $F_Θ$ first processes the input 3D coordinate $ x$ with 8 fully-connected layers (using ReLU activations and 256 channels per layer), and outputs $σ$ and a 256-dimensional feature vector. This feature vector is then concatenated with the camera ray’s viewing direction and passed to one additional fully-connected layer (using a ReLU activation and 128 channels) that output the view-dependent RGB color.

图三所表达的含义

沿着半球改变视角时表面的连续变化

Volume Rendering with Radiance Fields

The volume density $σ(x) $can be interpreted as the differential probability of a ray terminating at an infinitesimal particle at location $x$.

定义为在空间中任意点 $x $处，光线终止于该点的微分概率。这意味着$ σ(x) $反映了光线在通过空间中特定点$ x $时，终止于该点的可能性。

$$C ( \mathbf{r} )=\int_{t_{n}}^{t_{f}} T ( t ) \sigma( \mathbf{r} ( t ) ) \mathbf{c} ( \mathbf{r} ( t ), \mathbf{d} ) d t \,, \mathrm{~ w h e r e ~} T ( t )=\operatorname{e x p} \biggl(-\int_{t_{n}}^{t} \sigma( \mathbf{r} ( s ) ) d s \biggr) \,. \tag{1}$$

The function $T_(t)$ denotes the accumulated transmittance along the ray from tn to $t$, i.e., the probability that the ray travels from tn to $t $without hitting any other particle.

stratified sampling approach

$$t_{i} \sim\mathcal{U} \left[ t_{n}+\frac{i-1} {N} ( t_{f}-t_{n} ), \ t_{n}+\frac{i} {N} ( t_{f}-t_{n} ) \right]. \tag{2}$$ $$\hat{C} ( {\bf r} )=\sum_{i=1}^{N} T_{i} ( 1-\operatorname{e x p} (-\sigma_{i} \delta_{i} ) ) {\bf c}_{i} \,, \mathrm{~ w h e r e ~} T_{i}=\operatorname{e x p} \left(-\sum_{j=1}^{i-1} \sigma_{j} \delta_{j} \right), \tag{3}$$

where $δ_i = t_i+1 − t_i $ is the distance between adjacent samples.

Optimizing a Neural Radiance Field

Positional encoding

$$F_{\Theta}=F_{\Theta}^{\prime} \circ\gamma$$ $$\gamma( p )=\big( \operatorname{s i n} \bigl( 2^{0} \pi p \bigr), \, \operatorname{c o s} \bigl( 2^{0} \pi p \bigr), \, \cdots, \, \operatorname{s i n} \bigl( 2^{L-1} \pi p \bigr), \, \operatorname{c o s} \bigl( 2^{L-1} \pi p \bigr) \bigr) \,. \tag{4}$$

Hierarchical volume sampling

free space and occluded regions that do not contribute to the rendered image are still sampled repeatedly.

we first rewrite the alpha composited color from the coarse network $ \hat{C}_c(r) $ in Eqn. 3 as a weighted sum of all sampled colors ci along the ray:

$$\hat{C}_{c} ( {\bf r} )=\sum_{i=1}^{N_{c}} w_{i} c_{i} \,, ~ ~ ~ ~ w_{i}=T_{i} ( 1-\operatorname{e x p} (-\sigma_{i} \delta_{i} ) ) \,. \tag{5}$$

Normalizing these weights as

$$\hat{w}_{i} \ =\ w_{i} \big/ \sum_{j=1}^{N_{c}} w_{j}$$

produces a piecewise-constant PDF along the ray.