NLCA-Net: a non-local context attention network for stereo matching

Abstract

Accurate disparity prediction is a hot spot in computer vision, and how to efficiently exploit contextual information is the key to improve the performance. In this paper, we propose a simple yet effective non-local context attention network to exploit the global context information by using attention mechanisms and semantic information for stereo matching. First, we develop a 2D geometry feature learning module to get a more discriminative representation by taking advantage of multi-scale features and form them into the variance-based cost volume. Then, we construct a non-local attention matching module by using the non-local block and hierarchical 3D convolutions, which can effectively regularize the cost volume and capture the global contextual information. Finally, we adopt a geometry refinement module to refine the disparity map to further improve the performance. Moreover, we add the warping loss function to help the model learn the matching rule of the non-occluded region. Our experiments show that (1) our approach achieves competitive results on KITTI and SceneFlow datasets in the end-point error and the fraction of erroneous pixels $({D_1})$; (2) our proposed method particularly has superior performance in the reflective regions and occluded areas.

I. INTRODUCTION

Stereo matching plays an essential role in computer vision tasks, including autonomous driving [1, 2], object detection and recognition [3, 4], and 3D reconstruction and understanding [5–7]. For a couple of rectified stereo images, disparity refers to the apparent pixel difference or motion between a pair of corresponding pixels on the left and right images [8, 9].

The dense disparity map estimation methods have been studied for many years. For the traditional stereo matching methods (e.g. semi-global matching (SGM) [9], non-local cost aggregation [10], second-order smoothness priors [11]), the classical pipeline involves the finding of corresponding points by matching cost, cost aggregation, optimization, disparity refinement, and post-processing based on the local or global features. In general, the traditional methods often focus on using the prior knowledge of images to construct the warping function through for improving matching accuracy.

For disparity prediction based on deep learning, recent efforts have yielded many high-quality outputs due to deep fully convolutional neural networks (FCN) [12, 13] and a large amount of training data [14–16]. Classically, mainstream methods contain a four-step pipeline, while each step is important to the overall matching performance: 2D feature extraction, cost volume construction, 3D feature matching, and disparity regression [13, 17, 18]. Zbontar and LeCun first calculated the matching costs by convolutional neural networks (CNNs) to improve the performance, and the result showed that CNNs could learn more robust features from images and produced reliable matching cost in this task [17]. Following this work, many researchers [13, 19–22] proposed several methods to improve matching accuracy [13, 22], reduce some parameters [18], or achieve self-supervision ability [21].

Albeit the above success, the stereo matching methods based on deep learning still exist some limitations. First, the prediction pixels have terrible performance in the occluded, repeated object, and reflective regions [23, 24] due to a lack of sufficient understanding of the scene. Second, it is difficult to improve further accurate correspondence estimation if solely applying the concatenation operation between different viewpoints in cost volume construction [13, 20]; it is caused by the concatenation operation that is lack of the physical meaning about similarity. Due to the above problems, the performance of used networks has encountered bottlenecks.

In this paper, we propose a novel non-local context attention network (NLCA-Net) to exploit the global context information for stereo matching. First, we utilize spatial pyramid pooling (SPP) and dilated convolutions to extract the semantic information. Next, we apply a variance-based method to build cost volume. Then, we use the hierarchical 3D convolution and non-local block [25] to set up the non-local attention matching (NLAM) module for regularizing the cost volume. Finally, we adopt a soft argmin operation to get the initial disparity map and then refine it via combining the semantic information. At last, we further improve the accuracy of the non-occlusion region by the warping loss function.

Our main contributions are listed below:

• • We design a non-local context attention module to exploit the global context information for regularizing the cost volume, thus improving the performance of the matching task, particularly on the occlusion.

• • We use a variance-based method instead of traditional concatenation operation to build cost volume, which provides the similarity information and reduces some memory.

II. RELATED WORK

To improve the accuracy of disparity map estimation in stereo matching, many researchers have tried to optimize cost volume or matching cost computation and got fantastic achievements. Interested readers are suggested to read the surveys to get an overview of the typical matching algorithms and different optimization methods [26–28]. In this section, we will focus on a brief discussion about the related methods, involving traditional methods, deep leaning matching methods, and semantic segmentation methods, respectively.

In general, traditional stereo matching methods care more about how to compute the matching cost accurately and how to apply local or global features to refine the disparity map [29, 30]. Guney and Geiger used inverse graphics techniques to integrate objects as a non-local regularizer, then applied the conditional random field (CRF) framework to refine the disparity map; its result showed the value of this method on the KITTI dataset [31]. Seki and Pollefeys developed deep neural networks based on SGM for predicting accurate dense disparity map and introduced a novel loss function that fully uses sparsely annotated disparity maps features; their method replaced manually-tuned penalties for regularization [23]. Moreover, Gidaris and Komodakis proposed a generic architecture that improved the labels by detecting incorrect labels, replacing incorrect labels with new ones, and refining the renewed labels (DRR); their method achieved a significant improvement surpassing prior approaches [32]. These methods used the ideas of traditional disparity map post-processing to reduce the mismatch in ambiguous regions and improve disparity estimation.

Recently, in stereo matching areas, the end-to-end networks have been developed to predict whole disparity maps without post-processing. Mayer et al. introduced two end-to-end networks for estimating disparity (DispNet) and optical flow (FlowNet), and created a large synthetic dataset called SceneFlow, which improved the performance [14]. Chang and Chen introduced PSMNet, an end-to-end network for feature fusion using SPP and dilated convolution architectures [20]. Zhong et al. used image warping error as the loss function to drive the learning process, achieving a self-improving ability [21]. Kendall et al. exploited the way of cost volume regularization and shown 3D convolutions' effect in the context learning of stereo matching [13]. Guo et al. divided left–right features into different groups to obtain multiple matching cost proposals for measuring feature similarities and reducing some parameters [18]. Yin et al. composed local matching distributions to form the global match density for lessening the disparity candidates [22]. The main idea of these methods was to construct the cost volume or use external information (e.g. optical flow or edge) to improve the accuracy of disparity estimation, ignoring the effect of global scene understanding.

In the field of semantic segmentation, how to fuse the context information is an important topic. Many researchers proposed different methods to exploit global context information and make substantial progress in recent years. Long et al. demonstrated the value of the FCN in the semantic segmentation, and the performance had been dramatically improved [33]. Chen et al. designed a DeepLab_v3 that could capture multi-scale context with further boost performance by adopting multiple atrous rates, and the system of DeepLab_v3 without DenseCRF post-processing [34]. Ranjan and Black proposed the SPyNet, which introduced image pyramids to predict optical flow by a coarse-to-fine approach [35]. The above approaches showed that the idea of multi-scale architecture was essential for exploiting global context information in the field of semantic segmentation.

In this work, we exploit the potential of the non-local attention mechanism to enhance the scene understanding at the global-scope level. Moreover, we construct the cost volume by the variance-based method to add the similarity information compared with traditional concatenation operation. As described above, we propose the NLCA-Net for improving the matching accuracy, especially in the occlusions and reflective regions.

III. OUR METHOD

In this section, we propose the NLCA-Net. The network architecture is illustrated in Fig. 1, and the detailed parameters are presented in Appendix A. Our model consists of five parts: feature extraction and fusion, cost volume construction, feature matching, disparity map regression, and refinement. The implementation detail is described in the following sub-sections, respectively.

Fig. 1. Our end-to-end deep stereo regression architecture, NLCA-Net (Non-Local Context Attention network). Our model consists of three modules: 2D geometry feature learning (GFL), non-local attention matching (NLAM), and geometry refinement (GR) module.

A) Feature extraction and fusion by 2D geometry feature learning module

Previous methods failed to predict an accurate disparity in ill-posed regions because the network did not understand the context well. On the other hand, semantic segmentation methods have a fantastic performance in understanding the context. Thus, we are inspired by many semantic segmentation methods and get a robust descriptor that determines the context relationship from the pixel (particularly for ill-posed regions) via the SPP struct. We design the 2D geometry feature learning (GFL) module, as shown in Fig. 2. For description convenience, we set the input resolution of the stereo pairs to be $H \times W$, and the basic number F of the convolution filter to be $32$.

Fig. 2. The 2D geometry feature learning module (GFL). $x \times x,s,f$ denote the size of the convolution kernel, stride, and the number of convolution filters respectively. ${\times} n$ denotes the block repeats n times.

In this module, we apply a series of 2D convolutional operations to extract the semantic information, and each convolutional operation is followed by a batch normalization (BN) layer and a rectified linear unit (ReLU) layer. The GFL module consists of the unary feature extraction part and the multi-scale feature fusion part.

(1) The unary feature extraction part

The unary feature extraction part contains the block 1–4 and SPP, which are the basic or residual unit for learning the unary feature. In block 3 and block 4, dilated convolution is applied to enlarge the receptive field further. In SPP, we use the multi-scale average pooling to compress features and a $1 \times 1$ convolution to reduce feature dimension; then, the feature maps are upsampled to the original size. Next, we concatenate the unary feature of block 2, block 4, and SPP. After the unary feature extraction part, we could obtain the aggregated unary feature volume with the size $H/4 \times W/4 \times 10F$.

(2) The multi-scale feature fusion part

The multi-scale feature fusion part is block 5, which is used for fusing the aggregated unary feature volume. To avoid losing the critical information, we first adopt $128$ convolutional filters with the size $3 \times 3$ to fuse them, then use the $32$ convolutional filters with the size $3 \times 3$ to reduce feature dimension. After the multi-scale feature fusion part, we could obtain the semantic information with the size $H/4 \times W/4 \times F$.

B) Cost volume construction by variance-based cost metric

In previous works [13, 19–21], the cost volume is the critical step which links 2D and 3D convolution. To achieve better performance, we aggregate the semantics feature of left and right image ${V_l},{V_r}$ to one cost volume C via variance-based cost metric (VBCM) ${\mathcal M}$. Let $W,H,D,F$ be the input image height and width, the disparity sample number, and the feature number. Thus, the size of the semantics feature is ${V_l} = {V_r} = (H/4) \times (W/4) \times F$, and the size of cost volume $C = (D/4) \times (H/4) \times (W/4) \times F$. We define the cost metric as the mapping ${{\mathcal M}}:\underbrace{{\{ {V_\textrm{l}},{V_{r,1}}\}, \ldots, \{ {V_\textrm{l}},{V_{r,(D/4)}}\} }}_{{D/4}} \to C$ that:

(1)\begin{equation}\begin{aligned} C& = {\mathcal M}\left( {\{{{V_l},{V_{r,1}}} \}, \cdots,\Big\{ {{V_l},{V_{r,\frac{D}{4}}}} \Big\}} \right)\\ & = stack\left( {\dfrac{{{{({{V_l} - \overline {{V_i}} } )}^2} + {{({{V_{r,i}} - \overline {{V_i}} } )}^2}}}{2}} \right), \end{aligned}\tag{1}\end{equation}

where ${V_{r,i}}$ means traversed right semantics feature with a preset disparity range i, $\overline {{V_i}}$ means the average of ${V_l}$ and ${V_{r,i}}$, and all operations above are element-wise.

Most traditional stereo matching methods aggregate the cost volume between the left and right images in a heuristic way. However, recent works apply the concatenation operation instead of the mean or subtraction operation [13, 36]. This is the way that depends on network learning entirely. Here we choose the variance-based operation instead of the concatenation operation, due to which provides no direction about what the networks should do in the feature matching module. In contrast, our variance-based operation explicitly measures the left–right feature difference, which reflects the similarities between them and saves about half of memory. The true matched pair should have the lowest cost value, whereas it should have a higher cost. The output size of variance-based cost volume is $D/4 \times H/4 \times W/4 \times F$.

C) Feature matching by non-local attention matching module

To regularize the matching cost volume along the disparity dimension as well as spatial dimensions, we propose a 3D CNN architecture for learning the matching feature: the NLAM module. In [25], the non-local block was designed to compute the response at a position as a weighted sum of the features at all positions, and it showed a significant improvement for video classification and poses estimation. However, the cost volume is too big for the non-local block, leading it cannot be directly applied to the matching task. From another point, the essence of the non-local block is the attention mechanism. Therefore, we could combine the non-local block and the hierarchical 3D convolution for setting up the NLAM module, as shown in Fig. 3.

Fig. 3. The non-local attention matching module (NLAM). The NLAM module consists of feature matching part and scale recovery part. Note that the feature maps are shown as feature dimensions, e.g. $D \times H \times W \times F$means a feature map with disparity number$D$, height$H$, width W, and feature number$F$. Here, $L\ast$denotes different scale levels of the feature maps.

In this module, we apply a series of 3D convolutional operations to obtain the matching volume, and each convolutional operation is followed by a BN layer and a ReLU layer. The NLAM module consists of feature matching part and scale recovery part.

(1) The feature matching part

The feature matching part contains $26$ convolutions with stride one or two for regularizing the variance-based cost volume. This part has four levels, and we pass the feature maps between the same level to form the residual architecture, avoiding losing the critical information. Each level consists of an up-sampling or a sub-sampling convolution, and a residual block. After the $L4$ level, we adopt a non-local block as an attention block for further improving global matching learning.

To further understand the non-local attention mechanism, our feature matching part could be viewed as the group of the hierarchical 3D convolution block and the non-local block. The hierarchical 3D convolution block is an encoder–decoder architecture; it encodes the feature map by sub-sampling and decodes the encoded feature by up-sampling, as shown in Fig. 4.

Fig. 4. The encoder–decoder architecture. The pink block means the encoding process. The green block means the decoding process.

As shown in Fig. 4, we define the encoder and decoder process, respectively as:

(2)\begin{equation}\begin{aligned} {E_{i + 1}} & ={{\mathcal F}_{{E_{i + 1}}}}({{E_i}} )\\ {D_i} & ={{\mathcal F}_{{D_i}}}({{D_{i - 1}}} )+ {E_{n - i}}, \end{aligned}\tag{2}\end{equation}

where n denotes the number of the encoder and decoder, i denotes the level of the encoder or decoder, and ${{\mathcal F}_E}$ or ${{\mathcal F}_D}$ denotes the process of each encoder or decoder.

In the encoder–decoder architecture, the worst drawback is that the convolution is a local operation. It causes the network to not further improve the receptive fields to evaluate the impact of global information on the current pixel. Thus, we use the non-local block as attention mode for promoting the understanding ability of global context, as shown in Fig. 3.

The non-local operation ${{\mathcal N}}({\cdot} )$ could be represented as:

(3)\begin{equation}{y_i} = \frac{1}{{{{\mathcal C}}(x )}}\sum\limits_{\forall j} f ({{x_i},{x_j}} )g({{x_j}} ),\tag{3}\end{equation}

where $x,y$ denotes the input and output, respectively, i denotes the index of an output position, j denotes the index of all possible positions, $f({\cdot} )$ denotes the response function of global influence on current position ${x_i}$, $g({\cdot} )$ denotes a representation of the input signal at the position ${x_j}$, and ${{\mathcal C}}({\cdot} )$ denotes the total influence for normalizing the response.

In our non-local block, we apply the Gaussian function to compute similarity in an embedding space. We set the response function $f({\cdot} )$ as:

(4)\begin{equation}f({{x_i},{x_j}} )= {e^{\theta {{({{x_i}} )}^T}\phi ({{x_j}} )}},\tag{4}\end{equation}

where $\theta ({{x_i}} )= {W_\theta } \cdot {x_i}$, $\phi ({{x_j}} )= {W_\phi } \cdot {x_j}$. Similarly, $g({\cdot} )$ could be set as $g({{x_j}} )= {W_g} \cdot {x_j}$. Thus, the ${{\mathcal C}}({\cdot} )$ could be set as ${{\mathcal C}}(x )= {\sum _{\forall j}}f({{x_i},{x_j}} )$. In this response function, the process of $(1/{{\mathcal C}}(x)){\sum _{\forall j}}f({{x_i},{x_j}} )$ could be viewed as a softmax operation. In addition, we add y and x for a residual learning. In our architecture, the non-local block could be defined as:

(5)\begin{equation}{D_0} = {{\mathcal N}}({{E_4}} )+ {E_4}.\tag{5}\end{equation}

After the non-local attention step, we feed the fused global feature into the decoding process as presented in equation (2) or Fig. 4. The non-local block could improve the performance of matching effectively. After this part, we could obtain the matching volume but in a low resolution $1/4H \times 1/4W \times 1/4D$. Thus, we should recover the scale to get the final matching volume.

(2) The scale recovery part

The scale recovery part contains one convolution and two de-convolutions for recovering the size of the input image. The output of our NLAM module is a final matching volume with size $D \times H \times W$ from the variance-based cost volume.

D) Disparity map regression by soft argmin

In this step, we will estimate the initial disparity map from the matching volume. Thus, we naturally embed our matching volume into a 3D to 2D process. The simplest way to recover the initial disparity map $\hat{d}$ from the matching volume M is the pixel-wise winner-take-all such as an argmax operation. However, this way is unable to predict sub-pixel estimation and less robust [5, 13]. Thus, we predict the disparity map by passing an argmin operation. First, we convert the matching volume M to the probability volume ${{\mathcal P}}$ via the softmax operation $\sigma ({\cdot} )$. Then, we take the sum of each disparity d weighted with its probability. The soft argmin process is defined as:

(6)\begin{equation}\hat{d} = \mathop \sum \limits_{d = 0}^{{D_{\max }}} d \times {{\mathcal P}}(d )= \mathop \sum \limits_{d = 0}^{{D_{\max }}} d \times \sigma ({ - {M_d}} ),\tag{6}\end{equation}

where ${{\mathcal P}}(d )$ denotes the probability estimation for all pixels of the image at disparity d. ${M_d}$ denotes all value of the $d$-th layer in the matching volume M.

The above method could accurately approximate the disparity d in the range from $0$ to ${D_{\max }}$. The output initial disparity map is the same size as the input image.

E) Disparity map refinement by geometry refinement module

The initial disparity map from the probability volume is a qualified output, but the boundaries may suffer from over-smoothing in the recovery size part of the NLAM module, or the completeness of the object suffers from the missing piece in the occlusion area. Notice that the input image contains complete boundary information, and the output of 2D GFL module contains the semantics feature, we thus use the input image and the semantics feature as guidance to refine the initial disparity map. Inspired by the recent multi-view stereo algorithm [5], we redesign a geometry refinement (GR) module at the end of NLCA-Net, as shown in Fig. 5.

Fig. 5. Geometry refinement module (GR). The initial disparity map, the left image, and the semantics feature are fed to the GR module. After this module, we get refined disparity map. Here, blue block means the 32 convolutions with the size$3 \times 3$, and green block means the $1$ convolution with the size$3 \times 3$.

In the GR module, the initial disparity map, the left image, and the semantics feature are concatenated as the input. Notice that the size of the semantics feature is only quarter in width and height dimension compared to input images. Therefore, we first upsample the semantics feature, then use the $1 \times 1$ convolutional filter to adjust it. Next, we concatenate them as a $38$-feature input and send them to eight-layer residual network, which consists of the $32$ convolutional filters with the size $3 \times 3$. After that, we apply one convolutional filter to obtain the difference, and the initial disparity map is added back to generate the refined disparity map. The last layer does not contain the BN layer and the ReLU layer. After the GR module, we could get the refined disparity map.

F) Loss function

The loss functions consider both the initial disparity map and the refined disparity map. We use the ${L_1}$ loss to evaluate the difference between the ground truth and the predicted disparity map. Due to the ground truth sparse disparity map, we only consider those pixels which are valid. The $Los{s_{{L_1}}}$ is defined as:

(7)\begin{align} Los{s_{{L_1}}} =\dfrac{1}{{{N_1}}}\sum\limits_{p \in {P_{valid}}} {||d(p )- {{\hat{d}}_i}(p )|{|_1}} + ||d(p )- {{\hat{d}}_r}(p )|{|_1},\tag{7}\end{align}

where ${P_{valid}}$ denotes the set of valid ground truth pixels, ${N_1}$ denotes the total number of valid ground truth pixels, $d(p )$ denotes the ground truth of pixel p, ${\hat{d}_i}(p )$ denotes the initial disparity map value of pixel p, and ${\hat{d}_r}(p )$ denotes the refined disparity map value of pixel p.

To further improve the robustness and performance of the non-occluded area, we apply the warping loss to our loss function. The warping function is widely used in the traditional methods, and the previous learning method utilizes the warping function to structure the warping loss which gives the network self-supervised ability [21]. To solve the different illumination between left–right images, we introduce a structural similarity (SSIM) term $S({\cdot} )$ [37] to improve the robustness. The $Los{s_w}$ is defined as:

(8)\begin{align} Los{s_w}({{I_L},I_L^\prime } ) =\dfrac{1}{{{N_2}}}\mathop \sum \limits_{p \in {\textbf{n}_{valid}}} {\lambda _1}\dfrac{{1 - S({{I_L},I_L^\prime } )}}{2} + {\lambda _2}|{{I_L} - I_L^\prime } |, \tag{8}\end{align}

where ${\textbf{n}_{valid}}$ denotes the set of pixels in the non-occluded area, ${N_2}$ denotes the total number of valid pixels in the non-occluded area, ${I_L}$ denotes the left image, ${I^{\prime}_L}$ denotes the warping image which is reconstructed from the right image and the refined disparity map, and ${\lambda _1},{\lambda _2}$ denote the balance between structural similarity and image appearance difference.

Our loss function for stereo matching is defined as:

(9)\begin{equation}Loss = \alpha Los{s_{{L_1}}} + \beta Los{s_w},\tag{9}\end{equation}

where $\alpha$ denotes the weight of $Los{s_{{L_1}}}$, and $\beta$ denotes the weight of $Los{s_w}$.

IV. EXPERIMENTS

In this section, we will evaluate the performance of our method on two widely used stereo datasets: SceneFlow and KITTI. First, we show our implementation details about the network setting and training method, as shown in Section A. Then, we compare the contribution of the different components in NLCA-Net, as shown in Section B. Finally, we quantize the performance of our method and compare it with the state-of-the-art methods on the KITTI 2012 and 2015, as shown in Section C.

A) Implementation details

In this section, we implement NLCA-Net by Tensorflow with $5.46$ M trainable parameters, and the code will be released at the Github website.¹ To obtain the final model, we should choose the hyper-parameters, train, and evaluate the model.

(1) Hyper-parameters and datasets

For the hyper-parameters in this network, we set the max disparity $D = 192$ to ensure all possible disparity values in the image could be detected. In the loss function, we initially apply ${\lambda _1} = 0$, ${\lambda _2} = 0$, $\alpha = 1$, and $\beta = 0$ and then empirically test the best parameters based on our experiments.

For the datasets, we will train and evaluate our approach on these stereo datasets as follows:

• • SceneFlow: a large synthetic dataset consists of $35454$ training and $4370$ testing images with the size $H \times W = 540 \times 960$, which provides dense and clear disparity maps as ground truth. It could help us to adequately assess the performance of different model variants without worrying about over-fitting, and to make the pre-trained model have better generalization performance.

• • KITTI 2012: a challenging and varied road scene dataset contains $194$ training and $195$ testing images with the size $H \times W = 376 \times 1236$, which only provides sparse disparity maps as ground truth for training images.

• • KITTI 2015: a real-world street views dataset contains $200$ training and $200$ testing images with the size $H \times W = 374 \times 1236$, which only provides sparse disparity maps as ground truth for training images.

(2) Training method

For the training process, our network can be trained from random initialization in an end-to-end way with the supervision of stereo pairs and optimized using Adam Optimizer with ${\beta _1} = 0.9$, ${\beta _2} = 0.999$, and a batch size of $1$ for each GPUs. Before training, we normalize stereo pairs with pixel intensities level ranging from $0$ to$1$, and randomly crop them into $256 \times 512$. During the training process, we adopt the multi-step training method with four Nvidia 1080Ti GPUs. Thus, the training process consists of two parts: the pre-training process and the fine-tuning process.

• • In the pre-training process on the SceneFlow dataset, the learning rate is initially set to $1 \times {10^{ - 3}}$ for $30$ epochs and obtains the pre-train model.

• • In the fine-tuning process on the KITTI dataset, the learning rate is set to $1 \times {10^{ - 3}}$ for $800$ epochs and then reduced to $1 \times {10^{ - 4}}$ for the other $100$ epochs. After the training process, we get the final model.

(3) Evaluating metric

For evaluating our model and comparing with the state-of-the-art methods published recently, we show our results' errors with the following metrics, which have been widely used in the website of KITTI dataset:

\begin{align*} MAE & = \frac{1}{{|N |}}\mathop \sum \limits_{i \in T} |{\hat{d} - d} |,\\ {D_{1 - bg}} & = \frac{{\mathop \sum \limits_{i \in bg} [{|{\hat{d} - d} |\gt m \wedge i = vaild} ]}}{{\mathop \sum \limits_{i \in bg} [{i = vaild} ]}},\\ {D_{1 - fg}} & = \frac{{\mathop \sum \limits_{i \in fg} [{|{\hat{d} - d} |\gt m \wedge i = vaild} ]}}{{\mathop \sum \limits_{i \in fg} [{d = vaild} ]}},\\ {D_{1 - all}} & = \frac{{\mathop \sum \limits_{i \in T} [{|{\hat{d} - d} |\gt m \wedge i = vaild} ]}}{{\mathop \sum \limits_{i \in T} [{d = vaild} ]}},\end{align*}

where d denotes the ground truth of disparity, $\hat{d}$ denotes the estimated depth, m denotes the threshold of error pixel, $[{\cdot} ]$ denotes the Iverson bracket, $bg$ represents the set of all points in background regions of the images, $fg$ represents the set of all points in foreground regions of the images, and T represents the set of all points in the images.

B) Ablation study for NLCA-Net

To verify our design's effectiveness, we conduct experiments with different settings to evaluate NLCA-Net on the SceneFlow dataset. We train different model variants like the pre-training process, as presented in Section A. We first compare the performance of different settings, including the 2D GFL module, VBCM, the NLAM module, and the GR module, as shown in Table 1. Then, we present the representative results of our model and ablation study of loss weight, as shown in Fig. 6 and Table 2. Moreover, we test the impact of the different numbers of the non-local on the model, as shown in Table 3.

Table 1. Evaluation of NLCA-Net with different settings.

Computed the percentage of three-pixel-error on the SceneFlow and KITTI 2015 test set.

Fig. 6. SceneFlow test data qualitative results. From left: left stereo input image, ground-truth, disparity prediction.

Table 2. Influence of weight values for ${\lambda _1}$, ${\lambda _2}$, $\alpha$, and $\beta$ on three-pixel-error.

We empirically found that 0.85/0.15/0.8/0.2 yielded the best performance on the SceneFlow test set.

Table 3. Influence of the different numbers of the non-local blocks on the model.

Here R denotes the number of the non-local blocks.

As shown in Fig. 6 and Table 1, it qualitatively demonstrates the benefits of using the modules which we proposed. First, the 2D GFL and NLAM modules show a significant performance improvement for the matching accuracy. For the GFL module, it enhances the scene understanding ability of the model effectively; for the NLAM module, it has a strong regularization ability to learning the matching rules and can facilitate the learning process. Second, the GR module and $Los{s_w}$ function improve the matching accuracy a little. On a good baseline, the GR module could further improve the performance and $Los{s_w}$ function could achieve $1\%$ improvement on whole pixels and $4\%$ on non-occluded pixels. Finally, the VBCM provides better testing accuracy compared with traditional concatenation way, and it saves half-memory of the cost volume, which reduces about 300 M running memory of the whole framework.

As shown in Table 2, we conduct experiments with various combinations of loss weights which have the relationship about ${\lambda _1} + {\lambda _2} = 1$ and $\alpha + \beta = 1$. For the baseline, we only use the ${L_1}$ loss function and then continuously adjust the loss weights to find the weights which yield the best performance. The result shows that the weight settings of $0.85$ for ${\lambda _1}$, $0.15$ for ${\lambda _2}$, 0.8 for $\alpha$, and 0.2 for $\beta$ obtain the best performance, which is a $2.87\%$ three-pixel-error rate on the SceneFlow test set and a $1.96\%$ three-pixel-error rate on the KITTI test set.

Finally, we also test our model with different non-local block (NL-block) numbers. In this part, we set a model without NL-block as the baseline. The impact of NL-block is tested by increasing the NL-block number in the model. As shown in Table 3, we list the results with different NL-blocks on the SceneFlow and KITTI 2015, indicating that performance can be improved by introducing NL-block into the model. It is also noticed that the improvement decays gradually as the number of NLB increases, showing the boundary effect.

C) KITTI 2012 and 2015 benchmark results

To evaluate the performance of our model, we compare the performance of NLCA-Net with other state-of-the-art methods on the KITTI dataset. We use the multi-step training method to train the model, as shown in Section A. We present the representative images of our model and other state-of-the-art methods, as shown in Figs 7 and 8. Then, we evaluate the performance of our model on the KITTI website,² as shown in Tables 4 and 5. Besides, we compare our model with other competing algorithms in the reflective regions, as presented in Table 6.

Fig. 7. KITTI 2012 test data qualitative results. We compare our approach with state-of-the-art methods (HD³-S and GwcNet), and we highlight our advantage in the error maps. Note that, in the error maps, the deeper red pixels mean higher error rate in the occluded regions and white pixels denote ≥5 pixels error in the non-occluded regions.

Fig. 8. KITTI 2015 test data qualitative results. From left: left stereo input image, disparity prediction, error map. Note that, in the error maps, depicting correct estimates (<3 px or <5% error) in blue and wrong estimates in red color tones.

Table 4. Results on KITTI 2012 stereo benchmark.

Average end-point-error (EPE) and the percentage of different pixel-error are used for evaluations on the KITTI 2012 test set. Compared with other algorithms, our approach achieves the best performance in most cases.

* The number of the non-local blocks is 3.

Table 5. Results on KITTI 2015 stereo benchmark.

Our approach achieves comparable performance to state-of-the-art methods.

* The number of the non-local blocks is 3.

Table 6. Comparisons of different state-of-the-art methods in the reflective regions.

Average end-point-error (EPE) and the percentage of different pixel-error are used for evaluations on the KITTI 2012 test set. Compared with other algorithms, our approach achieves the best performance in most cases.

* The number of the non-local blocks is 3.

As shown in Figs 7 and 8, the proposed method could produce dense and clear disparity maps. For the non-occluded region, the proposed method shows the powerful performance; the disparity maps predicted by NLCA-Net are sharper and more complete than other learning methods. Even if in the occluded region, the proposed method still provides high-level performance and significantly outperforms other state-of-the-art methods, as shown in Fig. 9. The results show that our method can notably reduce the error rate on the occluded area and handle well with the large textureless regions. It also means the semantic information plays an important role, which offers the boundary information to perfect the edge pixels of objects. Overall, the proposed method shows the incredible expressiveness in the matching task.

Fig. 9. Part zoom-up of the error maps on the occluded region. From left: original image, HD³-Stereo, GwcNet, and ours. The result shows that our method can notably reduce the error rate on the occluded area and handle well with the large textureless regions.

As shown in Tables 4 and 5, it qualitatively demonstrates the performance of the proposed method. Compared with other methods, the proposed method yields more precise and robust disparity maps, particularly in the non-occluded regions. For the KITTI 2012 dataset, our approach is very close to HD³-Stereo [22] on non-occluded (Out-Noc) in the error rate of 2 pixels; but for other quality indexes, our method all achieves the best performance as shown in the table. For the KITTI 2015 dataset, our approach shows comparable performance and markedly outperforms other competing algorithms, including the previous best result (HD³-Stereo [22] and GwcNet [18]). In short, the proposed approach is superior to state-of-the-art methods in most cases.

As shown in Table 6, it qualitatively shows the advantage of our model in the reflective regions. In our designs, we use attention mechanisms and semantic information to enhance the ability of scene understanding. Thus, in the reflective region, the proposed method achieves the best performance of all quality indexes and significantly outperforms other state-of-the-art methods. It indicates the effectiveness of our matching network based on the contextual attention mechanism, which exhibits robustness to the reflective regions.

V. CONCLUSIONS

In this work, we present a highly efficient network architecture for stereo matching. The proposed model can exploit the global context information to achieve superior performance in the matching task. The NLAM significantly enhances the ability of scene understanding to improve the accuracy in the challenging regions, such as occlusions and large textureless/reflective areas. The variance-based cost volume can provide the similarity information and reduces some memory, thus further improving the performance. The experiment shows that the proposed method can improve the performance in challenging regions and outperform state-of-the-art methods in most cases. For future work, we are interested in exploring the generative adversarial network's potential to achieve higher accurate semi-supervised or unsupervised stereo matching.