scikit-笔记10:用词袋模型进行文本特征解析

In the example illustrated in the figure at the beginning of this notebook, we used the so-called 1-gram (unigram) tokenization: Each token represents a single element with regard to the splitting criterion.

Entirely discarding word order is not always a good idea, as composite phrases often have specific meaning, and modifiers like "not" can invert the meaning of words.

A simple way to include some word order are n-grams, which don't only look at a single token, but at all pairs of neighborhing tokens. For example, in 2-gram (bigram) tokenization, we would group words together with an overlap of one word; in 3-gram (trigram) splits we would create an overlap two words, and so forth:

• original text: "this is how you get ants"
• 1-gram: "this", "is", "how", "you", "get", "ants"
• 2-gram: "this is", "is how", "how you", "you get", "get ants"
• 3-gram: "this is how", "is how you", "how you get", "you get ants"

Which "n" we choose for "n-gram" tokenization to obtain the optimal performance in our predictive model depends on the learning algorithm, dataset, and task. Or in other words, we have consider "n" in "n-grams" as a tuning parameters, and in later notebooks, we will see how we deal with these.

Now, let's create a bag of words model of bigrams using scikit-learn's CountVectorizer:

>>> create obj and conver to model by fit: this will create vocabulary and count the number of occurrence in each 'string' >>> compare bigrams(2-grams) with unigrams(1-gram)

# look at sequences of tokens of minimum length 2 and maximum length 2
bigram_vectorizer = CountVectorizer(ngram_range=(2, 2))

from sklearn.feature_extraction.text import CountVectorizer
vectorizer = CountVectorizer()

vectorizer.fit(X), bigram_vectorizer.fit(X)

(CountVectorizer(analyzer='word', binary=False, decode_error='strict',
dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
lowercase=True, max_df=1.0, max_features=None, min_df=1,
ngram_range=(1, 1), preprocessor=None, stop_words=None,
strip_accents=None, token_pattern='(?u)\\b\\w\\w+\\b',
tokenizer=None, vocabulary=None),
CountVectorizer(analyzer='word', binary=False, decode_error='strict',
dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
lowercase=True, max_df=1.0, max_features=None, min_df=1,
ngram_range=(2, 2), preprocessor=None, stop_words=None,
strip_accents=None, token_pattern='(?u)\\b\\w\\w+\\b',
tokenizer=None, vocabulary=None))

bigram_vectorizer.get_feature_names(), vectorizer.get_feature_names()

>>> what is the tokenization

(['end in',
'in fire',
'in ice',
'say in',
'say the',
'some say',
'the world',
'will end',
'world will'],
['end', 'fire', 'ice', 'in', 'say', 'some', 'the', 'will', 'world'])

>>> representation of samples(here, 1 row 1 sample)

bigram_vectorizer.transform(X).toarray(), X_bag_of_words.toarray()

(array([[1, 1, 0, 0, 1, 1, 1, 1, 1],
[0, 0, 1, 1, 0, 1, 0, 0, 0]]), array([[1, 1, 0, 1, 1, 1, 1, 1, 1],
[0, 0, 1, 1, 1, 1, 0, 0, 0]]))

>>> combine unigrams with bigrams This will give a lone vector representation for each sample(string).

Often we want to include unigrams (single tokens) AND bigrams, wich we can do by passing the following tuple as an argument to the ngram_range parameter of the CountVectorizer function:

gram_vectorizer = CountVectorizer(ngram_range=(1, 2))
gram_vectorizer.fit(X)

CountVectorizer(analyzer='word', binary=False, decode_error='strict',
dtype=<class 'numpy.int64'>, encoding='utf-8', input='content',
lowercase=True, max_df=1.0, max_features=None, min_df=1,
ngram_range=(1, 2), preprocessor=None, stop_words=None,
strip_accents=None, token_pattern='(?u)\\b\\w\\w+\\b',
tokenizer=None, vocabulary=None)

gram_vectorizer.get_feature_names()

['end',
'end in',
'fire',
'ice',
'in',
'in fire',
'in ice',
'say',
'say in',
'say the',
'some',
'some say',
'the',
'the world',
'will',
'will end',
'world',
'world will']

gram_vectorizer.transform(X).toarray()

array([[1, 1, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1],
[0, 0, 0, 1, 1, 0, 1, 1, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0]])

1. from sklearn.datasets import make_blobs
2. from sklearn.datasets import load_iris
3. from sklearn.model_selection import train_test_split
4. from sklearn.linear_model import LogisticRegression
5. from sklearn.linear_model import LinearRegression
6. from sklearn.neighbors import KNeighborsClassifier
7. from sklearn.neighbors import KNeighborsRegressor
8. from sklearn.preprocessing import StandardScaler
9. from sklearn.decomposition import PCA
10. from sklearn.metrics import confusion_matrix, accuracy_score
11. from sklearn.metrics import adjusted_rand_score
12. from sklearn.cluster import KMeans
13. from sklearn.cluster import KMeans
14. from sklearn.cluster import MeanShift
15. from sklearn.cluster import DBSCAN # <<< this algorithm has related sources in LIHONGYI's lecture-12
16. from sklearn.cluster import AffinityPropagation
17. from sklearn.cluster import SpectralClustering
18. from sklearn.cluster import Ward
19. from sklearn.metrics import confusion_matrix
20. from sklearn.metrics import accuracy_score
21. from sklearn.metrics import adjusted_rand_score
22. from sklearn.feature_extraction import DictVectorizer
23. from sklearn.feature_extraction.text import CountVectorizer *
24. from sklearn.feature_extraction.text import TfidfVectorizer *
25. from sklearn.preprocessing import Imputer
26. from sklearn.dummy import DummyClassifier

CountVectorizer implements both tokenization and occurrence counting in a single class