本論文研究提出基於畫面內容之多運算子影像尺寸調整機制，希望在顯示畫面於不同輸出設備時仍能保持畫質，本研究亦提出適用於此應用的畫質衡量模型，合理評估原始影像與修改後影像的差異。首先我們改良多運算子畫面調整機制SCAN，它包含了圖縫裁減(Seam carving)、邊緣裁切(Cropping)、增加圖縫(Add seams)與畫面縮放(Normalization)。本研究主要改善邊緣裁切步驟，透過前景物偵測將影像分類，根據類別及畫面中的物體以不同的視覺顯著圖決定適當裁切位置。此外，我們加入人臉與建築物偵測，避免出現於畫面邊緣的人臉可能遭受不當裁切，並判斷建築物是否為畫面重要內容。實驗結果顯示所提出的改良式多運算子畫面調整機制在各式影像中能有效維持內容完整。在畫質衡量模型中，我們利用SIFT Flow比較原始影像及濃縮影像的內容差異，考量可能出現的幾何扭曲及線段扭曲，根據畫面顯著物及語意相關程度，以類神經網路迴歸分析找出平均意見分數(MOS)對每種屬性的依據，進而得到更貼近於人眼主觀感受的評估。實驗結果顯示，與其他評估方法相較，我們所提出的模型更貼近於MOS的結果。

This research proposes a content-based multi-operator image retargeting scheme, enabling the retargeted images to preserve its content after adaptation in various displays. Besides, a quality evaluation model is also proposed to compare original images and retargeted images. The proposed multi-operator retargeting scheme is termed “SCAN” as it contains Seam caving, Cropping, Adding seams and Normalization (scaling). This research mainly concentrates on improving the step of content-based cropping in SCAN. We classify images into two categories via foreground detection and adopt different types of visual saliency to determine appropriate cropping limits. The face detection is also introduced to protect face areas appearing at the edges of an image from being removed. A building detection mechanism is employed to determine whether a building in an image is significant or not. The experimental shows that the improved multi-operator retargeting scheme can effectively preserve the content and objects’ shape when dealing with various images. In the proposed quality evaluation model, we make use of SIFT Flow to compare the contents of original and retargeted images and identify possible geometric distortion and line distortion. We further consider salient objects and image semantics in the evaluation process. With these attributes, we utilize the neural network regression model to determine the weights of every feature in order to fit the Mean Opinion Score (MOS). The results show that the proposed model is closer to MOS than other evaluation methods.

[1] D. Vaquero, M. Turka, K. Pullib, M. Ticob, and N. Gelfandb, "A survey of image retargeting techniques," Proceedings of SPIE the International Society for Optical Engineering, vol. 7798, p. 779814, 2010.
[2] Ce Liu, Yuen, J. and Torralba, A. (2011). “SIFT Flow: Dense Correspondence across Scenes and Its Applications.” IEEE Transactions on Pattern Analysis and Machine Intelligence, 33(5), pp.978-994.
[3] F. Stentiford, "Attention based auto image cropping," The 5th International Conference on Computer Vision Systems, Bielefeld, 2007.
[4] I. S. Amrutha, S. S. Shylaja, S. Natarajan, and K. N. Murthy, "A smart automatic thumbnail cropping based on attention driven regions of interest extraction," Proceedings of the 2nd International Conference on Interaction Sciences: Information Technology, Culture and Human, ACM, pp. 957-962, 2009.
[5] P. Cheatle, "Automatic image cropping for republishing," IS&T/SPIE Electronic Imaging. International Society for Optics and Photonics, pp. 75400O-75400O-9, 2010.
[6] L. Itti, C. Koch, and E. Niebur, "A model of saliency-based visual attention for rapid scene analysis," IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. 20, no. 11, pp. 1254-1259, 1998.
[7] R. Gal, O. Sorkine, and D. Cohen-Or, "Feature-aware texturing," Proceedings of the 17th Eurographics conference on Rendering Techniques, Eurographics Association, June 2006.
[8] Y. S. Wang, C. L. Tai, O. Sorkine, and T. Y. Lee, "Optimized scale-and-stretch for image resizing," ACM Transactions on Graphics (TOG), vol. 27, no. 5, 2008.
[9] S. Avidan, and A. Shamir, "Seam carving for content-aware image resizing," ACM Transactions on graphics (TOG), vol. 26, no. 3, Aug 2007.
[10] M. Rubinstein, A. Shamir, and S. Avidan, "Improved seam carving for video retargeting," ACM Transactions on Graphics (TOG), vol. 27, no. 3, p. 16, 2008.
[11] M. Grundmann, V. Kwatra, M. Han, and I. Essa, "Discontinuous seam-carving for video retargeting," IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, June 2010.
[12] D. Domingues, A. Alahi, and P. Vandergheynst, "Stream carving: an adaptive seam carving algorithm," 17th IEEE International Conference on Image Processing (ICIP), pp. 901-904, 2010.
[13] S. Hua, G. Chen, H. Wei, and Q. Jiang, "Similarity measure for image resizing using SIFT feature," EURASIP Journal on Image and Video Processing, pp. 1-11, Jan 2012
[14] M. Rubinstein, A. Shamir, and S. Avidan, "Multi-operator media retargeting," ACM Transactions on Graphics (TOG), vol. 28, no. 3, 2009.
[15] W. Dong, N. Zhou, J. C. Paul, and X. Zhang, "Optimized image resizing using seam carving and scaling," ACM Transactions on Graphics (TOG), vol. 28, no. 5, p.125, 2009.
[16] Y.C., Chou, P. C., Su, “Toward More Efficient Multi-Operator Media Retargeting for Digital Images and Videos”, National Central University, 2016.
[17] S. Montabone, and A. Soto. “Human detection using a mobile platform and novel features derived from a visual saliency mechanism,” Image and Vision Computing, vol. 28, no. 3, pp391-402
[18] Ma, L., Lin, W., Deng, C. and Ngan, K. (2012). “Image Retargeting Quality Assessment: A Study of Subjective Scores and Objective Metrics.” IEEE Journal of Selected Topics in Signal Processing, 6(6), pp.626-639.
[19] Yabin Zhang, Weisi Lin, Qiaohong L, Wentao Cheng , and Xinfeng Zhang. “Multiple-Level Feature-Based Measure for Retargeted Image Quality” IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 27, NO. 1, pp 451-463, JANUARY 2018
[20] Yabin Zhang, Weisi Lin, Yuming Fang, Leida Li, “ASPECT RATIO SIMILARITY (ARS) FOR IMAGE RETARGETING QUALITY ASSESSMENT.”
[21] Yabin Zhang, Yuming Fang, Weisi Lin, Xinfeng Zhang and Leida Li, “Backward Registration-Based Aspect Ratio Similarity for Image Retargeting Quality Assessment,” IEEE TRANSACTIONS ON IMAGE PROCESSING, VOL. 25, NO. 9, pp. 4286-4297, SEPTEMBER 2016
[22] Hsu, C., Lin, C., Fang, Y. and Lin, W. (2014). “Objective Quality Assessment for Image Retargeting Based on Perceptual Geometric Distortion and Information Loss.” IEEE Journal of Selected Topics in Signal Processing, 8(3), pp.377-389.
[23] S. Ren, K. He, R. Girshick, and J. Sun, ‘‘Faster R-CNN: Towards realtime object detection with region proposal networks,’’ IEEE Trans. Pattern Anal. Mach. Intell., vol. 39, no. 6, pp. 1137–1149, Jun. 2017
[24] PASCAL VOC 2012 [Online] Available: http://host.robots.ox.ac.uk/pascal/VOC/
[25] Leeds Butterfly Dataset. Josiah Wang, Katja Markert, and Mark Everingham. Learning Models for Object Recognition from Natural Language Descriptions. In Proceedings of the 20th British Machine Vision Conference (BMVC2009) [Online] Avaliable: http://www.josiahwang.com/dataset/leedsbutterfly/
[26] 17 Category Flower Dataset. Maria-Elena Nilsback and Andrew Zisserman [Online] Available: http://www.robots.ox.ac.uk/~vgg/data/flowers/17/
[27] WIDER FACE: A Face Detection Benchmark [Online] Available: http://shuoyang1213.me/WIDERFACE/
[28] Pingping Zhang Dong Wang Huchuan Lu Hongyu Wang Xiang Ruan Dalian University of Technology, China Tiwaki Co.Ltd, “Amulet: Aggregating Multi-level Convolutional Features for Salient Object Detection”
[29] MSRA10k salient Object Database. Available: https://mmcheng.net/msra10k
[30] Hengshuang Zhao, Jianping Shi Xiaojuan Qi Xiaogang Wang, Jiaya Jia. “Pyramid Scene Parsing Network,” IEEE Conference on Computer Vision and Pattern Recognition (CVPR), IEEE, pp. 2881-2890, 2017.
[31] RetargetMe Benchmark [Online]. Available: http:// http://people.csail. mit.edu/mrub/retargetme/index.html
[32] NTHU Retargeting Image Dataset (NRID) [Online]. Available: http://www.ee.nthu.edu.tw/cwlin/Retargeting_Quality/NRID.html
[33] Zhibo Chen, Jianxin Lin, Ning Liao, and Chang Wen Chen. “Full Reference Quality Assessment for Image Retargeting Based on Natural Scene Statistics Modeling and Bi-Directional Saliency Similarity,” IEEE Transaction on Image Processing vol. 26, no. 11, pp 5138-5148, 2017