scikit-笔记16:深入理解线性模型

The difference between the linear models for regression is what kind of restrictions or penalties are put on coef_ as regularization , in addition to fitting the training data well.

The most standard linear model is the 'ordinary least squares regression, often simply called 'linear regression'. It doesn't put any additional restrictions on coef_, so when the number of features is large, it becomes ill-posed and the model overfits.

Let us generate a simple simulation, to see the behavior of these models.

make_regression can create a data set.

from sklearn.datasets import make_regression
from sklearn.model_selection import train_test_split
X, y, true_coefficient = make_regression(n_samples=200, n_features=30, n_informative=10, noise=100, coef=True, random_state=5)
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=5, train_size=60, test_size=140)
print(X_train.shape) # (60,30)
print(y_train.shape) # (60,)

Regression fn model:

y_pred = x_test[0] * coef_[0] + … + x_test[n_features-1] * coef_[n_features-1] + intercept_

All linear models for classification learn a coefficient parameter coef_ and an offset intercept_ to make predictions using a linear combination of features:

Classification fn model:

y_pred = x_test[0] * coef_[0] + … + x_test[n_features-1] * coef_[n_features-1] + intercept_ > 0

As you can see, this is very similar to regression, only that a threshold at zero is applied.

Again, the difference between the linear models for classification what kind of regularization is put on coef_ and intercept_, but there are also minor differences in how the fit to the training set is measured (the so-called loss function).

The two most common models for linear classification are the linear SVM as implemented in LinearSVC and LogisticRegression.

A good intuition for regularization of linear classifiers is that with high regularization, it is enough if most of the points are classified correctly. But with less regularization, more importance is given to each individual data point. This is illustrated using an linear SVM with different values of C below.

In LinearSVC, the C parameter controls the regularization within the model.

Lower C entails more regularization and simpler models, whereas higher C entails less regularization and more influence from individual data points.

from figures import plot_linear_svc_regularization
plot_linear_svc_regularization()

Similar to the Ridge/Lasso separation, you can set the penalty parameter to:

• 'l1' to enforce sparsity of the coefficients (similar to Lasso)
• 'l2' to encourage smaller coefficients (similar to Ridge).

We can see,

. xxx regularization apply on yyy , yyy will change to zzz . | | | . v v v . l1 coef_ sparse coef_ . l2 coef_ small coef_

you can see if you want to : separte the 2 class, you should need 1 split line; separte the 3 class, you should need 3 split line; separte the 4 class, you should need 6 split line;

So the shape of coef_ should be 3 in this case.

You should note, why we use the code shown below, to plot split line. As we said that fn model for classification is :

y_pred = x_test[0] * coef_[0] + … + x_test[n_features-1] * coef_[n_features-1] + intercept_ > 0

We get 2d coef_ for each line(totally we have 3 lines), for each line we should make make one dimension of the 2d point(x1,x2), as x-axis and another as y-axis, to plot the line for example:

• x1 –> x-axis
• x2 –> y-axis

In another word, this means that we should make point

• x1 as 'x' the independent variable;
• x2 as 'y' the dependent variable;

x_test[0] * coef_[0] + x_test[1] * coef_[1] + intercept_ = 0

x_test[1] = - (x_test[0] * coef_[0] + intercept_) / coef_[1]

This is the origin of code below:

plt.plot(line, -(line * coef[0] + intercept) / coef[1])

plt.scatter(X[:, 0], X[:, 1], c=plt.cm.spectral(y / 2.))
line = np.linspace(-15, 15)
for coef, intercept in zip(linear_svm.coef_, linear_svm.intercept_):
    plt.plot(line, -(line * coef[0] + intercept) / coef[1])
plt.ylim(-10, 15)
plt.xlim(-10, 8);

Points are classified in a one-vs-rest fashion (aka one-vs-all), where we assign a test point to the class whose model has the highest confidence (in the SVM case, highest distance to the separating hyperplane) for the test point.

1. from sklearn.datasets import make_blobs
2. from sklearn.datasets import make_regression *
3. from sklearn.datasets import load_iris
4. from sklearn.datasets import load_digits
5. from sklearn.model_selection import train_test_split
6. from sklearn.model_selection import cross_val_score
7. from sklearn.model_selection import KFold
8. from sklearn.model_selection import StratifiedKFold
9. from sklearn.model_selection import ShuffleSplit
10. from sklearn.model_selection import GridSearchCV
11. from sklearn.model_selection import learning_curve *
12. from sklearn.linear_model import LogisticRegression
13. from sklearn.linear_model import LinearRegression
14. from sklearn.linear_model import Ridge *
15. from sklearn.linear_model import Lasso *
16. from sklearn.linear_model import ElasticNet *
17. from sklearn.neighbors import KNeighborsClassifier
18. from sklearn.neighbors import KNeighborsRegressor
19. from sklearn.preprocessing import StandardScaler
20. from sklearn.decomposition import PCA
21. from sklearn.metrics import confusion_matrix, accuracy_score
22. from sklearn.metrics import adjusted_rand_score
23. from sklearn.metrics.scorer import SCORERS
24. from sklearn.metrics import r2_score *
25. from sklearn.cluster import KMeans
26. from sklearn.cluster import KMeans
27. from sklearn.cluster import MeanShift
28. from sklearn.cluster import DBSCAN # <<< this algorithm has related sources in LIHONGYI's lecture-12
29. from sklearn.cluster import AffinityPropagation
30. from sklearn.cluster import SpectralClustering
31. from sklearn.cluster import Ward
32. from sklearn.metrics import confusion_matrix
33. from sklearn.metrics import accuracy_score
34. from sklearn.metrics import adjusted_rand_score
35. from sklearn.metrics import classification_report
36. from sklearn.feature_extraction import DictVectorizer
37. from sklearn.feature_extraction.text import CountVectorizer
38. from sklearn.feature_extraction.text import TfidfVectorizer
39. from sklearn.preprocessing import Imputer
40. from sklearn.dummy import DummyClassifier
41. from sklearn.pipeline import make_pipeline
42. from sklearn.svm import LinearSVC
43. from sklearn.svm import SVC

Generate a random regression problem.

The input set can either be well conditioned (by default) or have a low rank-fat tail singular profile. See make_low_rank_matrix for more details.

The output is generated by applying a (potentially biased) random linear regression model with n_informative nonzero regressors to the previously generated input and some gaussian centered noise with some adjustable scale.

make_regression(n_samples=100,      #<- number of samples.
                n_features=100,     #<- number of features.
                n_informative=10,   #<- number of truely useful features
                n_targets=1,        #<- the dimension of y output
                bias=0.0,           #<- bias term in underlying linear model
                effective_rank=None,#<- The approximate number of singular
                                    #vectors required to explain most of the
                                    #input data by linear combinations.
                tail_strength=0.5,
                noise=0.0,          #<- The standard deviation of the gaussian
                                    #noise applied to the output.
                shuffle=True,
                coef=False,         #<- whether return coef of this undelying
                                    #linear model
                random_state=None)

Learning curve.

Determines cross-validated training and test scores for different training set sizes.

A cross-validation generator splits the whole dataset k times in training and test data. Subsets of the training set with varying sizes will be used to train the estimator and a score for each training subset size and the test set will be computed. Afterwards, the scores will be averaged over all k runs for each training subset size.

training_set_size, train_scores, test_scores = learning_curve(
    est,  #<- the ML model used to predict
    X,    #<- dataset passed to this model
    y,    #<- labels passed to this model
    train_sizes=np.linspace(.1, 1, 20) #<- commonly usage way: linspace(.1,1, num)
)

0 - ee240ab5-f4df-4c6b-8fa9-dc13d1590680

Ridge is another linear_model, and has the identical inteface with linear_model on methods invocation

• ridge = Ridge()
• ridge.fit(X,y)
• ridge.predict(X)
• ridge.score(X,y) : model_name.score(X,y)

Lasso is another linear_model, and has the identical inteface with linear_model on methods invocation

• lasso = Lasso()
• lasso.fit(X,y)
• lasso.predict(X)
• lasso.score(X,y) : model_name.score(X,y)

The singular value decomposition of a matrix A is the factorization of A into the product of three matrices \(A = UDV^T\) where the columns of U and V are orthonormal and the matrix D is diagonal with positive real entries.

http://scikit-learn.org/stable/datasets/index.html#sample-generators

In addition, scikit-learn includes various random sample generators that can be used to build artificial datasets of controlled size and complexity.

Returns the indices that would sort an array, for multiple dimension array, sort and return each array('[]' denote an array) itself.

Perform an indirect sort along the given axis using the algorithm specified by the kind keyword. It returns an array of indices of the same shape as a that index data along the given axis in sorted order.

import numpy as np
arr = np.array([[4,2,1,0],[5,9,8,7]])
argarr = np.argsort(arr)
argarr

array([[3, 2, 1, 0],
[0, 3, 2, 1]])

specify the indexs of axes

import matplotlib.pyplot as plt
fig, ax = plt.subplots()
ax.set_xticks([0.15, 0.68, 0.97])
ax.set_yticks([0.2, 0.55, 0.76])
plt.show()