-
Notifications
You must be signed in to change notification settings - Fork 1
/
README-Chinese.txt
114 lines (78 loc) · 3.77 KB
/
README-Chinese.txt
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
Stanford Chinese Segmenter - v3.3.1 - 2014-01-04
--------------------------------------------
(c) 2003-2012 The Board of Trustees of The Leland Stanford Junior University.
All Rights Reserved.
Chinese segmenter by Pi-Chuan Chang, Huihsin Tseng, and Galen Andrew
CRF code by Jenny Finkel
Support code by Stanford JavaNLP members
The system requires Java 6 (JDK1.6) or higher.
USAGE
Unix:
> segment.sh [-k] [ctb|pku] <filename> <encoding> <size>
ctb : Chinese Treebank
pku : Beijing Univ.
filename: The file you want to segment. Each line is a sentence.
encoding: UTF-8, GB18030, etc.
(This must be a character encoding name known by Java)
size: size of the n-best list (just put '0' to print the best hypothesis
without probabilities).
-k: keep all white spaces in the input
* Sample usage: segment.sh ctb test.simp.utf8 UTF-8
* Note: Large test file requires large memory usage. For processing
large data files, you may want to change memory allocation in Java
(e.g., to be able to use 8Gb of memory, you need to change "-mx2g"
to "-mx8g" inside segment.sh). Another solution is to split the test
file to smaller ones to reduce memory usage.
* In addition to the command line scripts, there is a Java class
"SegDemo" which shows how to call the segmenter in Java code.
Usage:
java -mx2g -cp seg.jar:fastutil.jar:. SegDemo test.simp.utf8
SegDemo as supplied assumes that it is running in the home directory of the
installation, and to run anywhere else, you need to set the path to the
dictionaries.
SEGMENTATION MODELS
Two segmentation models are provided. The "ctb" model was trained with Chinese
treebank (CTB) segmentation, and the "pku" model was trained with Beijing
University's (PKU) segmentation. PKU models provide smaller vocabulary
sizes and OOV rates on test data than CTB models.
For both CTB and PKU, we provide two models representing slightly different
feature sets:
Models "ctb" and "pku" incorporate lexicon features to increase consistency in
segmentation.
DATA
[Segmentation standard]
(Chinese Penn Treebank)
The supplied segmenter segments according to Chinese Penn Treebank
segmentation conventions. For more information, see:
http://www.cis.upenn.edu/~chinese/segguide.3rd.ch.pdf
(Beijing University)
This segmenter segments according to the Peking University standard.
For more information, see:
http://sighan.cs.uchicago.edu/bakeoff2005/data/pku_spec.pdf
[Training data]
(Chinese Penn Treebank)
"data/ctb.gz" is trained with all data in the LDC Chinese Treebank 6
(prerelease).
(Beijing University)
"data/pku.gz" is trained with the data provided by Peking University
for the Second International Chinese Word Segmentation Bakeoff.
See:
http://sighan.cs.uchicago.edu/bakeoff2005/
MORE INFORMATION
The details of the segmenter can be found in this paper:
Huihsin Tseng, Pichuan Chang, Galen Andrew, Daniel Jurafsky
and Christopher Manning.
"A Conditional Random Field Word Segmenter."
In Fourth SIGHAN Workshop on Chinese Language Processing. 2005.
http://nlp.stanford.edu/pubs/sighan2005.pdf
(Notice that the training data, features and normalizations
used in this distribution are not exactly the same as the systems
described in the paper.)
The description of the lexicon features can be found in:
Pi-Chuan Chang, Michel Gally and Christopher Manning.
"Optimizing Chinese Word Segmentation for Machine Translation Performance"
In ACL 2008 Third Workshop on Statistical Machine Translation.
http://nlp.stanford.edu/pubs/acl-wmt08-cws.pdf
For more information, look in the included Javadoc, starting with the
edu.stanford.nlp.ie.crf.CRFClasifier class documentation.
Send any questions or feedback to java-nlp-support@lists.stanford.edu.