|
Candidate Term Boundary Conflict Reduction Method for Chinese Geological Text Segmentation
点击次数：
发表刊物：Applied Sciences
关键字：Chinese word segmentation; information entropy; degree of freedom of terms; zero-sample; single geological text
摘要：Though Chinese word segmentation (CWS) relies heavily on arithmetic power to train huge models and human work to label corpora, models and algorithms are still less accurate, especially for segmentation in a specific domain. In this study, a high-degree-of-freedom-priority candidate term boundary conflict reduction method (HFCR) is proposed to solve the problem of manually setting thresholds on segmentation based on information entropy. We quantify the uncertainty of left and right character connections of candidate terms and then arrange them in descending order for local comparisons to determine term boundaries. Dynamic numerical comparisons are adopted instead of setting a threshold manually and randomly. Experiments show that the average F1-value of CWS for Chinese geological text is higher than 95% and the F1-value for Chinese general datasets is higher than 87%. Compared with representative tokenizers and the SOTA model, our method performs better, which solves the term boundary conflict problem well and has excellent performance on single geological text segmentation without any samples or labels.
合写作者：Zhiyong Guo
第一作者：Yu Tang
论文类型：期刊论文
通讯作者：Jiqiu Deng*
文献类型：J
卷号：13
期号：7
页面范围：4516
是否译文：否
发表时间：2023-04-02
收录刊物：SCI