TopWORDS can achieve efficient “word discovery” and “text segmentation” simultaneously for any text sequence encoded in utf-8. It's particularly useful for processing domain-specific Chinese texts which contains a large number of unknown or irregular words, phrases and technical terms. In case that the target texts are very different from available training corpus, TopWORDS often outperforms popularly used methods based on supervised learning. The method can also be readily applied to English and other alphabet-based languages for discovering regular expressions, idioms, and special names by treating each English word as a character, which can be more informative for downstream analyses.

The figure below illustrates the architecture of the TopWORDS algorithm. More details about the approach can be found in our paper.

Reference: Deng, K., Bol, P. K., Li, K. J., & Liu, J. S. (2016). On the unsupervised analysis of domain-specific Chinese texts. Proceedings of the National Academy of Sciences of the United States of America, 113(22), 6154–6159. http://doi.org/10.1073/pnas.1516510113