scikit-笔记22:超大数据集下的文本分类与情感分析

import os
train_path = os.path.join('datasets', 'IMDb', 'aclImdb', 'train')
test_path = os.path.join('datasets', 'IMDb', 'aclImdb', 'test')

Now, let's load them into our active session (load into memory by default) via scikit-learn's load_files function

from sklearn.datasets import load_files
train = load_files(container_path=(train_path),
                   categories=['pos', 'neg'])
test = load_files(container_path=(test_path),
                  categories=['pos', 'neg'])

NOTE: Since the movie datasets consists of 50,000 individual text files, executing the code snippet above may take ~20 sec or longer. The load_files function loaded the datasets into sklearn.datasets.base.Bunch objects, which are Python dictionaries:

for more information, see here sklearn.datasets.load_files()

train.keys()

dict_keys(['data', 'filenames', 'target_names', 'target', 'DESCR'])

In particular, we are only interested in the data and target arrays.

These two methods are very useful for get info of 'target' np.unique(data['target']) np.bincount(data['target'])

import numpy as np
for label, data in zip(('TRAINING', 'TEST'), (train, test)):
    print('\n\n%s' % label)
    print('Number of documents:', len(data['data']))
    print('\n1st document:\n', data['data'][0])
    print('\n1st label:', data['target'][0])
    print('\nClass names:', data['target_names'])
    print('Class count:',
          np.unique(data['target']), ' -> ',
          np.bincount(data['target']))

As we can see above the 'target' array consists of integers 0 and 1, where 0 stands for negative and 1 stands for positive.

from sklearn.utils.murmurhash import murmurhash3_bytes_u32
# encode for python 3 compatibility
for word in "the cat sat on the mat".encode("utf-8").split():
    print("{0} => {1}".format( word, murmurhash3_bytes_u32(word, 0) ))
    print("{0} => {1}".format(
        word, murmurhash3_bytes_u32(word, 0) % 2 ** 20))

This mapping is completely stateless and the dimensionality of the output space is explicitly fixed in advance (here we use a modulo 2 ** 20 which means roughly 1M dimensions). The makes it possible to workaround the limitations of the vocabulary based vectorizer both for parallelizability and online / out-of-core learning.

The HashingVectorizer class is an alternative to the CountVectorizer (or TfidfVectorizer class with use_idf=False) that internally uses the murmurhash hash function:

from sklearn.feature_extraction.text import HashingVectorizer
h_vectorizer = HashingVectorizer(encoding='latin-1')
h_vectorizer

HashingVectorizer(alternate_sign=True, analyzer='word', binary=False,
decode_error='strict', dtype=<class 'numpy.float64'>,
encoding='latin-1', input='content', lowercase=True,
n_features=1048576, ngram_range=(1, 1), non_negative=False,
norm='l2', preprocessor=None, stop_words=None, strip_accents=None,
token_pattern='(?u)\\b\\w\\w+\\b', tokenizer=None)

It shares the same "preprocessor", "tokenizer" and "analyzer" infrastructure:

analyzer = h_vectorizer.build_analyzer()
analyzer('This is a test sentence.')

['this', 'is', 'test', 'sentence']

We can vectorize our datasets into a scipy sparse matrix exactly as we would have done with the CountVectorizer or TfidfVectorizer, except that we can directly call the transform method: there is no need to fit as HashingVectorizer is a stateless transformer:

docs_train, y_train = train['data'], train['target']
docs_valid, y_valid = test['data'][:12500], test['target'][:12500]
docs_test, y_test = test['data'][12500:], test['target'][12500:]

The dimension of the output is fixed ahead of time to n_features=2 ** 20 by default (nearly 1M features) to minimize the rate of collision on most classification problem while having reasonably sized linear models (1M weights in the coef_ attribute):

h_vectorizer.transform(docs_train)

<25000x1048576 sparse matrix of type '<class 'numpy.float64'>'
with 3446628 stored elements in Compressed Sparse Row format>

Now, let's compare the computational efficiency of the HashingVectorizer to the CountVectorizer:

 h_vec = HashingVectorizer(encoding='latin-1')
 %timeit -n 1 -r 3 h_vec.fit(docs_train, y_train)

#The slowest run took 4.42 times longer than the fastest. This could mean that
#an intermediate result is being cached.

# 7.53 µs ± 5.13 µs per loop (mean ± #std. dev. of 3 runs, 1 loop each)

from sklearn.feature_extraction.text import CountVectorizer
count_vec =  CountVectorizer(encoding='latin-1')
%timeit -n 1 -r 3 count_vec.fit(docs_train, y_train)
# 2.95 s ± 6.17 ms per loop (mean ± std. dev. of 3 runs, 1 loop each)

As we can see, the HashingVectorizer is much faster than the Countvectorizer in this case.

• 7.53 µs ± 5.13 µs per loop (mean ± #std. dev. of 3 runs, 1 loop each)
• 2.95 s ± 6.17 ms per loop (mean ± std. dev. of 3 runs, 1 loop each)

Finally, let us train a LogisticRegression classifier on the IMDb training subset:

from sklearn.linear_model import LogisticRegression
from sklearn.pipeline import Pipeline
h_pipeline = Pipeline([
    ('vec', HashingVectorizer(encoding='latin-1')),
    ('clf', LogisticRegression(random_state=1)),
])
h_pipeline.fit(docs_train, y_train)

Pipeline(memory=None,
steps=[('vec', HashingVectorizer(alternate_sign=True, analyzer='word', binary=False,
decode_error='strict', dtype=<class 'numpy.float64'>,
encoding='latin-1', input='content', lowercase=True,
n_features=1048576, ngram_range=(1, 1), non_negative=False,
norm='l2', p...nalty='l2', random_state=1, solver='liblinear', tol=0.0001,
verbose=0, warm_start=False))])

print('Train accuracy', h_pipeline.score(docs_train, y_train))
print('Validation accuracy', h_pipeline.score(docs_valid, y_valid))

import gc
del count_vec
del h_pipeline
gc.collect()

101

But first, let us load the file names into a Python list:

train_path = os.path.join('datasets', 'IMDb', 'aclImdb', 'train')
train_pos = os.path.join(train_path, 'pos')
train_neg = os.path.join(train_path, 'neg')
fnames = [os.path.join(train_pos, f) for f in os.listdir(train_pos)] +\
         [os.path.join(train_neg, f) for f in os.listdir(train_neg)]
fnames[:3]

['datasets/IMDb/aclImdb/train/pos/5561_8.txt',
'datasets/IMDb/aclImdb/train/pos/8049_7.txt',
'datasets/IMDb/aclImdb/train/pos/9072_9.txt']

​

Next, let us create the target label array:

y_train = np.zeros((len(fnames), ), dtype=int)
y_train[:12500] = 1
np.bincount(y_train)

array([12500, 12500])

Now, we implement the batch_train function as follows, which return a SGD_classifier model.

from sklearn.base import clone
def batch_train(clf,           #<- classifier model
                fnames,        #<- array, filenames
                labels,        #<- array, labels
                iterations=25, #<- iteration times
                batchsize=1000,#<- size of each batch
                random_seed=1  #<- random seed
):
    # ---- do some configuration
    vec = HashingVectorizer(encoding='latin-1') #<- initial vectorizer model.
    idx = np.arange(labels.shape[0])            #<- create label array's index
    c_clf = clone(clf)                          #<-
    rng = np.random.RandomState(seed=random_seed)

    # ---- how many times you want to do batch learning
    for i in range(iterations):

        # each time randomly sample bathsize filenames indices
        # later will be used to index file content and related labels
        rnd_idx = rng.choice(idx, size=batchsize)

        # create an empty list, to save smapled file, used as dataset of SGD
        documents = []

        # combine all sample files' content into 'documents'
        for i in rnd_idx:
            with open(fnames[i], 'r', encoding='latin-1') as f:
                documents.append(f.read())

        # vectorize the sample files' inside 'documents'
        X_batch = vec.transform(documents)

        # index the related labels by indices array 'rnd_idx'
        batch_labels = labels[rnd_idx]

        # from classifier obj to classifier model
        c_clf.partial_fit(X=X_batch,
                          y=batch_labels,
                          classes=[0, 1])# Classes across all calls to
                                         # partial_fit. Can be obtained by via
                                         # np.unique(y_all), where y_all is the
                                         # target vector of the entire dataset.
                                         # This argument is required for the
                                         # first call to partial_fit and can be
                                         # omitted in the subsequent calls.
                                         # Note that y doesn’t need to contain
                                         # all labels in classes.

    return c_clf

Note that we are not using LogisticRegression as in the previous section, but we will use a SGDClassifier with a logistic cost function instead. SGD stands for stochastic gradient descent, an optimization alrogithm that optimizes the weight coefficients iteratively sample by sample, which allows us to feed the data to the classifier chunk by chuck.

And we train the SGDClassifier; using the default settings of the batch_train function, it will train the classifier on 25*1000=25000 documents. (Depending on your machine, this may take >2 min)

Eventually, let us evaluate its performance:

vec = HashingVectorizer(encoding='latin-1')
sgd.score(vec.transform(docs_test), y_test)

0.83176

The collision issues can be controlled by increasing the n_features parameters.

The IDF weighting might be reintroduced by appending a TfidfTransformer instance on the output of the vectorizer. However computing the idf_ statistic used for the feature reweighting will require to do at least one additional pass over the training set before being able to start training the classifier: this breaks the online learning scheme.

The lack of inverse mapping (the get_featurenames() method of TfidfVectorizer) is even harder to workaround. That would require extending the HashingVectorizer class to add a "trace" mode to record the mapping of the most important features to provide statistical debugging information.

In the mean time to debug feature extraction issues, it is recommended to use TfidfVectorizer(use_idf=False) on a small-ish subset of the dataset to simulate a HashingVectorizer() instance that have the get_feature_names() method and no collision issues.

EXERCISE: In our implementation of the batch_train function above, we randomly draw k training samples as a batch in each iteration, which can be considered as a random subsampling with replacement. Can you modify the batch_train function so that it iterates over the documents without replacement, i.e., that it uses each document exactly once per iteration?

1. from sklearn.datasets import make_blobs
2. from sklearn.datasets import make_moons
3. from sklearn.datasets import make_circles
4. from sklearn.datasets import make_scurve
5. from sklearn.datasets import make_regression
6. from sklearn.datasets import load_files *
7. from sklearn.datasets import load_iris
8. from sklearn.datasets import load_digits
9. from sklearn.datasets import load_breastcancer

---

For all Bunch object return by many load_xxx() is a dict-like obj, and you can:

• get all keys(attributes) by bunch_obj.keys()
• access all attributes by bunch_obj.[the key_name return by keys()]

---

1. from mpl_{toolkits.mplot3d} import Axes3D
2. from sklearn.model_selection import train_testsplit
3. from sklearn.model_selection import cross_valscore
4. from sklearn.model_selection import KFold
5. from sklearn.model_selection import StratifiedKFold
6. from sklearn.model_selection import ShuffleSplit
7. from sklearn.model_selection import GridSearchCV
8. from sklearn.model_selection import learning_curve
9. from sklearn.feature_extraction import DictVectorizer
10. from sklearn.feature_{extraction.text} import CountVectorizer
11. from sklearn.feature_{extraction.text} import HashingVectorizer *
12. from sklearn.feature_{extraction.text} import TfidfVectorizer
13. from sklearn.feature_selection import SelectPercentile
14. from sklearn.feature_selection import f_classif
15. from sklearn.feature_selection import f_regression
16. from sklearn.feature_selection import chi2
17. from sklearn.feature_selection import SelectFromModel
18. from sklearn.feature_selection import RFE
19. from sklearn.linear_model import LogisticRegression
20. from sklearn.linear_model import LinearRegression
21. from sklearn.linear_model import Ridge
22. from sklearn.linear_model import Lasso
23. from sklearn.linear_model import ElasticNet
24. from sklearn.neighbors import KNeighborsClassifier
25. from sklearn.neighbors import KNeighborsRegressor
26. from sklearn.neighbors.kde import KernelDensity *
27. from sklearn.preprocessing import StandardScaler
28. from sklearn.metrics import confusion_matrix, accuracy_score
29. from sklearn.metrics import adjusted_randscore
30. from sklearn.metrics.scorer import SCORERS
31. from sklearn.metrics import r2_score
32. from sklearn.cluster import KMeans
33. from sklearn.cluster import KMeans
34. from sklearn.cluster import MeanShift
35. from sklearn.cluster import DBSCAN # <<< this algorithm has related sources in LIHONGYI's lecture-12
36. from sklearn.cluster import AffinityPropagation
37. from sklearn.cluster import SpectralClustering
38. from sklearn.cluster import Ward
39. from sklearn.cluster import DBSCAN
40. from sklearn.cluster import AgglomerativeClustering
41. from scipy.cluster.hierarchy import linkage
42. from scipy.cluster.hierarchy import dendrogram
43. from scipy.stats.mstats import mquantiles
44. from sklearn.metrics import confusion_matrix
45. from sklearn.metrics import accuracy_score
46. from sklearn.metrics import adjusted_randscore
47. from sklearn.metrics import classification_report
48. from sklearn.preprocessing import Imputer
49. from sklearn.dummy import DummyClassifier
50. from sklearn.pipeline import make_pipeline
51. from sklearn.svm import LinearSVC
52. from sklearn.svm import SVC
53. from sklearn.svm import OneClassSVM *
54. from sklearn.tree import DecisionTreeRegressor
55. from sklearn.ensemble import RandomForestClassifier
56. from sklearn.ensemble import GradientBoostingRegressor
57. from sklearn.ensemble import IsolationForest
58. from sklearn.decomposition import PCA
59. from sklearn.manifold import TSNE
60. from sklearn.manifold import Isomap
61. from sklearn.utils.murmurhash import murmurhash3_bytesu32
62. from sklearn.base import clone *

Constructs a new estimator with the same parameters.

Clone does a deep copy of the model in an estimator without actually copying attached data. It yields a new estimator with the same parameters that has not been fit on any data.

from sklearn.base import clone
def batch_train(clf, fnames, labels, iterations=25, batchsize=1000, random_seed=1):
    vec = HashingVectorizer(encoding='latin-1')
    idx = np.arange(labels.shape[0])
    c_clf = clone(clf) #<- clone the classifier 'clf' passed to this function
    rng = np.random.RandomState(seed=random_seed)

    for i in range(iterations):
        rnd_idx = rng.choice(idx, size=batchsize)


        documents = []
        for i in rnd_idx:
            with open(fnames[i], 'r', encoding='latin-1') as f:
                documents.append(f.read())

        #
        X_batch = vec.transform(documents)
        batch_labels = labels[rnd_idx]
        c_clf.partial_fit(X=X_batch,
                          y=batch_labels,
                          classes=[0, 1])

    return c_clf

numpy.random.choice(a, size=None, replace=True, p=None) Generates a random sample from a given 1-D array

from sklearn.base import clone
def batch_train(clf, fnames, labels, iterations=25, batchsize=1000, random_seed=1):
    vec = HashingVectorizer(encoding='latin-1')
    idx = np.arange(labels.shape[0])
    c_clf = clone(clf)
    rng = np.random.RandomState(seed=random_seed)

    for i in range(iterations):
        rnd_idx = rng.choice(idx, size=batchsize) # <- randomly choose
                                                  # batch-size samples from idx
                                                  # with replacement
        documents = []
        for i in rnd_idx:
            with open(fnames[i], 'r', encoding='latin-1') as f:
                documents.append(f.read())

        #
        X_batch = vec.transform(documents)
        batch_labels = labels[rnd_idx]
        c_clf.partial_fit(X=X_batch,
                          y=batch_labels,
                          classes=[0, 1])

    return c_clf

SGDClassifier(loss=’hinge’,
              penalty=’l2’,
              alpha=0.0001,
              l1_ratio=0.15,
              fit_intercept=True,
              max_iter=None,
              tol=None,
              shuffle=True,
              verbose=0,
              epsilon=0.1,
              n_jobs=1,
              random_state=None,
              learning_rate=’optimal’,
              eta0=0.0,
              power_t=0.5,
              class_weight=None,
              warm_start=False,
              average=False,
              n_iter=None)

The ‘log’ loss gives logistic regression, a probabilistic classifier. ‘modified_huber’ is another smooth loss that brings tolerance to outliers as well as probability estimates. ‘squared_hinge’ is like hinge but is quadratically penalized. ‘perceptron’ is the linear loss used by the perceptron algorithm. The other losses are designed for regression but can be useful in classification as well; see SGDRegressor for a description.