Wei Zhang1, Chenfei Qu1, Lin Ma2, Jingwei Guan3, and Rui Huang4
1School of Control Science and Engineering, Shandong University
2Huawei Noah's Ark Lab, Hong Kong
3Department of Electronic Engineering, The Chinese University of Hong Kong
4NEC Laboratories China
In this paper, we propose to learn the structures of stereoscopic images based on convolutional neural network (CNN) for no-reference quality assessment. Taking image patches from the stereoscopic images as inputs, the proposed CNN can learn the local structures which are sensitive to human perception and representative for perceptual quality evaluation. By stacking multiple convolution and max-pooling layers together, the learned structures in lower convolution layers can be composed and convolved to higher levels to form a fixed-length representation. Multilayer perceptron (MLP) is further employed to summarize the learned representation to a final value to indicate the perceptual quality of the stereo image patch pair. With different inputs, two different CNNs are designed, namely one-column CNN with only the image patch from the difference image as input, and three-column CNN with the image patches from left-view image, right-view image, and difference image as the input. The CNN parameters for stereoscopic images are learned and transferred based on the large number of 2D natural images. With the evaluation on public LIVE phase-I, phase-II, and IVC stereoscopic image databases, the proposed no-reference metric achieves the state-of-the-art performance for quality assessment of stereoscopic images, and is even competitive to existing full-reference quality metrics.
The contributions of this work:
The stereoscopic images differ from the 2D natural images, as the left and right views together can provide depth perception. Therefore, perceptual evaluation of the stereoscopic image needs to consider the information from both the left and right views. It is demonstrated that the difference image between the left view and right view is more important than the left and right views for quality assessment.
Proposed three-column CNN:
The three-column CNN consists three one-column CNNs, which takes full consideration of not only left view and right view in stereoscopic images, but also the difference image obtaining from the stereoscopic image. The difference image implicitly contains the disparity and depth information, which is proved to be very important for the stereoscopic image quality perception.
Each column in the three-column CNN has two layers of convolution and max-pooling. The three identical CNNs are used to learn the structures from the left view, right view, and difference image, respectively. With two layers of convolution and max-pooling processes, three different image patches are represented as three different vectors, which are expected to contain the structures of stereoscopic images from different viewpoints. These three vectors are concatenated together and fed into the upper multilayer perceptron (MLP) to generate the quality score.
SROCC of IQA metrics on LIVE phase-I dataset
SROCC of IQA metrics on LIVE phase-II dataset (with symmetric distortion)
SROCC of IQA metrics on LIVE phase-II dataset (with asymmetric distortion)
LCC and SROCC of IQA metrics on IVC dataset
Benoit et al. You et al. Gorley and Holliman. MS-SSIM Hewage and Martini Wang et al. Ma et al. Akhter et al. Shao et al. Ryu and Sohn Chen et al. SSIM FSIM GSMD Bensalma et al. Chen et al. Carnec et al. Skeikh et al. Venkata et al. Ryu et al. Campisi et al.
Update: Apr. 14, 2016
