中文斷詞在中文的自然語言處理上，是個相當基礎且非常重要的前置處理工作。中文斷詞這個領域雖然已經研究了數十年，過去也有相當多的學者提出各種斷詞演算法，但至今解決中文斷詞問題的研究仍未中斷，並且越來越受到重視。近年來的斷詞系統則較傾向於使用統計式的機器學習演算法來解決中文斷詞的問題，例如隱藏式馬可夫模型。然而，標準的隱藏式馬可夫模型在解決中文斷詞的問題上，斷詞效能F-measure約只有80% 的結果，所以許多研究都是使用外部資源或是結合其他的機器學習演算法來幫助斷詞。本研究目的是希望使用最簡單的方法，並且毋須使用任何外部資源，來提升隱藏式馬可夫模型的準確率。我們的作法是應用特製化（specialization）的概念，將中文斷詞之歧義性及未知詞的資訊帶入隱藏式馬可夫模型中，於完全不修改模型之訓練及測試過程的前提之下，透過兩階段特製化的方式，分別以擴充「觀測符號」，以及擴充「狀態符號」的方式，大大地改善了隱藏式馬可夫模型的斷詞準確性。第一階段中，我們結合了長詞優先法以及遮罩方式（Mask method），將歧義性與未知詞的資訊帶入隱藏式馬可夫模型中，使得模型擁有更多的斷詞資訊做學習。於實驗結果得知，結合最簡單的長詞優先斷詞方法，確實能大幅地提升隱藏式馬可夫模型的效能，將F-measure由0.812提升至0.953的斷詞結果。而第二階段的特製化過程中，我們使用詞彙化（lexicalization）的方式分別對高頻率及高錯誤的觀測符號，來新增狀態符號，於實驗結果也證明了，透過此階段的改良能再次提升系統效能，將斷詞結果F-measure由0.953提升至0.963。

The first step in Chinese language processing tasks is word segmentation. Various methods have been proposed to address this problem in previous studies, e.g. heuristic-based approaches, statistical-based approaches, etc. HMM is a statistical machine learning approach that has been successfully applied in many fields, e.g. POS tagging, shallow parsing, and so on. However, we find that standard HMM achieved only 80% results in Chinese word segmentation. As is commonly known, segmentation ambiguity and unknown word occurrence are two main problems in Chinese word segmentation. In this paper, we proposed a two-stage specialized HMM by incorporating these information into the model. In the first stage, we combine the maximum matching heuristics to incorporate segmentation ambiguity and use a masking approach to handle unknown word information. By extending the observation symbols, the proposed M-HMM is improved from 0.812 to 0.953 in F-measure. At the second stage, we use lexicalization technique to further enrich HMM performance. The idea is to add new state symbols for high frequency characters or high tagging error symbols. Experimental results show that Lexicalized M-HMM is improved from 0.953 to 0.963 in F-measure.

1. M. Asahara, K. Fukuoka, A. Azuma, C. L. Goh, Y. Watanabe, Y. Matsumoto, T. Tsuzuki. Combination of Machine Learning Methods for Optimum Chinese Word Segmentation. In Proceedings of Fourth SIGHAN Workshop on Chinese Language Processing, pp. 134–137, 2005
2. M. Asahara, C. L. Goh, X. Wang and Y. Matsumoto. Combining Segmenter and Chunker for Chinese Word Segmentation. In Proceedings of Second SIGHAN Workshop on Chinese Language Processing, pp. 144–147, 2003
3. K. J. Chen and M. H. Bai. Unknown Word Detection for Chinese By a Corpus-based Learning Method. In Proceedings of ROCLING X, pp. 159–174, 1997
4. K. J. Chen and S. H. Liu. Word Identification for Mandarin Chinese Sentences. Proceedings COLING ''92, pp. 101-105, 1992
5. K. J. Chen and W. Y. Ma. Unknown Word Extraction for Chinese Documents. In Proceedings of COLING 2002, pp. 169–175, 2002
6. C. L. Goh, M. Asahara and Y. Matsumoto. Chinese Word Segmentation by Classification of Characters. International Journal of Computational Linguistics and Chinese Language Processing Vol. 10, No. 3, pp. 381-396, 2005
7. J. D. Kim, S. Z. Lee and H. C. Rim. HMM Specialization with Selective Lexicalization. In Proceedings of the joinSIGDAT Conference on Empirical Methods in Natural Lan-guage Processing and Very Large Corpora(EMNLP-VLC-99), pp. 121-127, 1999
8. S. Z. Lee, J. I. Tsujii and H. C. Rim. Lexicalized Hidden Markov Models for Part-of-Speech Tagging. In Proceedings of 18th International Conference on Computa-tional Linguistics, Saarbrucken, Germany, pp.481-787, 2000
9. M. Li, J. F. Gao, C. N. Huang and J. F. Li. Unsupervised Training for Overlapping Ambiguity Resolution in Chinese Word Segmentation. In Proceedings of Second SIGHAN Workshop on Chinese Language Processing, pp. 1–7, 2003
10. Y. Y. Li, C. J. Miao, K. Bontcheva and H. Cunningham. Perceptron Learning for Chinese Word Segmentation. In Proceedings of Fourth SIGHAN Workshop on Chinese Language Processing, pp. 154–157, 2005
11. X. Lu. Towards a Hybrid Model for Chinese Word Segmentation. In Proceedings of Fourth SIGHAN Workshop on Chinese Language Processing, pp. 189–192, 2005
12. X. Luo, M. Sun and B. K. Tsou. Covering Ambiguity Resolution in Chinese Word Segmentation Based on Contextual Information. In Proceedings of COLING 2002, pp. 598-604, 2002
13. W. Y. Ma and K. J. Chen. A Bottom-up Merging Algorithm for Chinese Unknown Word Extraction. In Proceedings of Second SIGHAN Workshop on Chinese Language Processing, pp. 31–38, 2003
14. C. D. Manning and H. Schutze. Foundation of Statistical Natural Language Processing. Chapter 9-10. pp. 317-380, 1999
15. A. Molina and F. Pla. Shallow Parsing using Specialized HMMs. Journal of Machine Learning Research 2, pp. 595–613, 2002
16. A. Molina, F. Pla and E. Segarra. A Hidden Markov Model Approach to Word Sense Disambiguation. In Proceedings of the VIII Conferencia Iberoamericana de Inteligencia Artificial, IBERAMIA 2002, pp. 1-9, 2002
17. F. Pla and A. Molina. Improving Part-of-Speech Tagging using Lexicalized HMMs. Natural Language Engineering, pp. 167-189, 2004
18. F. Pla and A. Molina. Part-of-Speech Tagging with Lexicalized HMM. In proceedings of International Conference on Recent Advances in Natural Language Processing(RANLP2001), 2001
19. L. R. Rabiner. A Tutorial on Hidden Markov Models and Selected Applications in Speech Recognition. Proceedings of the IEEE, Vol.77, No.22, pp. 257-286, 1989
20. H. H. Tseng, P. H. Chang, G. Andrew, D. Jurafsky, and C. Manning. A Conditional Random Field Word Segmenter for Sighan Bakeoff 2005. In Proceedings of Fourth SIGHAN Workshop on Chinese Language Processing, pp. 168–171, 2005
21. Y. C. Wu, C. H. Chang and Y. S. Lee. A General and Multi-lingual Phrase Chunking Model Based on Masking Method. Lecture Notes in Computer Science (LNCS): Computational Linguistics and Intelligent Text Processing, Vol. 3878, pp. 144-155, 2006
22. N. Xue. Chinese Word Segmentation as Character Tagging. International Journal of Computational Linguistics and Chinese, pp. 29–48, 2003
23. N. Xue and L. Shen. Chinese Word Segmentation as LMR Tagging. In Proceedings of Second SIGHAN Workshop on Chinese Language Processing, pp. 176–179, 2003
24. H. P. Zhang, Q. Liu, H. Zhang and X. Q. Cheng. Automatic Recognition of Chinese Unknown Words Based on Roles Tagging. In Proceedings of First SIGHAN Workshop on Chinese Language Processing, pp. 71-77, 2002
25. H. P. Zhang, H. K. Yu, D. Y. Xiong and Q. Liu. HHMM-based Chinese Lexical Analyzer ICTCLAS. In Proceedings of Second SIGHAN Workshop on Chinese Language Processing, pp. 187–187, 2003
26. J. H. Zheng and F. F. Wu. Study on segmentation of ambiguous phrases with the combinatorial type. Collections of Papers on Computational Linguistics. Tsinghua University Press, Beijing, pp. 129-134, 1999