scikit-笔记15:性能矩阵与模型评价

threshold = 0.5

all predicted probability > threshold, predict it positive; all predicted probability < threshold, predict it negative;

this means that :

all red pixels to the right of the line are correct predictions; all blue pixels to the left of the line are correct predictions;

accuracy rate = correct predictions %

[真正经，原本正经，你判断他是正经的] TPR = (red region on right of threshold) / whole red region TPR = TP / (TP + NF) = 真正 / (真正+假负) TPR = recall

[假正经，原本不正经，你判断他是正经的] FPR = (blue region on right of threshold) / whole blue region FPR = FP / (FP + TN) = 假正 / (假正+真负)

np.bincount(y) / y.shape[0]

array([ 0.1,  0.1,  0.1,  0.1,  0.1,  0.1,  0.1,  0.1,  0.1,  0.1])

As a toy example, let's say we want to classify the digits three against all other digits:

X, y = digits.data, digits.target == 3

Now we run cross-validation on a classifier to see how well it does:

from sklearn.model_selection import cross_val_score
from sklearn.svm import SVC
cross_val_score(SVC(), X, y)

array([ 0.9,  0.9,  0.9])

Our classifier is 90% accurate. Is that good? Or bad? Keep in mind that 90% of the data is "not three". So let's see how well a dummy classifier does, that always predicts the most frequent class:

from sklearn.dummy import DummyClassifier
cross_val_score(DummyClassifier("most_frequent"), X, y)

array([ 0.9,  0.9,  0.9])

Also 90% (as expected)! So one might thing that means our classifier is not very good, it doesn't to better than a simple strategy that doesn't even look at the data. That would be judging too quickly, though.

Accuracy is simply not a good way to evaluate classifiers for imbalanced datasets!

np.bincount(y) / y.shape[0]

array([ 0.9,  0.1])

A much better measure is using the so-called ROC (Receiver operating characteristics) curve. A roc-curve works with uncertainty outputs of a classifier, say the "decision_function" of the SVC we trained above. Instead of making a cut-off at zero(one threshold) and looking at classification outcomes, it looks at every possible cut-off(every possible thresholds) and records how many true positive predictions there are, and how many false positive predictions there are.

The following plot compares the roc curve of three parameter settings of our classifier on the "three vs rest" task.

from sklearn.metrics import roc_curve, roc_auc_score
X_train, X_test, y_train, y_test = train_test_split(X, y, random_state=42)
for gamma in [.01, .05, 1]:
    plt.xlabel("False Positive Rate")
    plt.ylabel("True Positive Rate (recall)")
    svm = SVC(gamma=gamma).fit(X_train, y_train)
    decision_function = svm.decision_function(X_test)
    fpr, tpr, _ = roc_curve(y_test, decision_function)
    acc = svm.score(X_test, y_test) #<- accuracy_score
    auc = roc_auc_score(y_test, svm.decision_function(X_test)) #<- auc score
    plt.plot(fpr, tpr, label="acc:%.2f auc:%.2f" % (acc, auc), linewidth=3)
    plt.legend(loc="best");

With a very small decision threshold, there will be few false positives, but also few false negatives, while with a very high threshold, both true positive rate and false positive rate will be high.

So in general, the curve will be from the lower left to the upper right. A diagonal line reflects chance performance, while the goal is to be as much in the top left corner as possible. This means giving a higher decision_function value to all positive samples than to any negative sample.

In this sense, this curve only considers the ranking of the positive and negative samples, not the actual value. As you can see from the curves and the accuracy values in the legend, even though all classifiers have the same accuracy, 89%, which is even lower than the dummy classifier, one of them has a perfect roc curve, while one of them performs on chance level.

For doing grid-search and cross-validation, we usually want to condense our model evaluation into a single number. A good way to do this with the roc curve is to use the area under the curve (AUC). We can simply use this in cross_val_score by specifying scoring="roc_auc":

from sklearn.model_selection import cross_val_score
cross_val_score(SVC(), X, y, scoring="roc_auc")

array([ 1.,  1.,  1.])

R^2 affected by degrees of freedom, see this, and the LinTianXuan lecture

see this

see this

Can we infer a relationship between

Number of medas won by a country

and

1. the country's latitude
2. the country's average elevation
3. the country's population

number of medals = \(\beta_0+\beta_1(latitude_i)+\beta_2(elevation_i) + \beta_3(logpopulation_i)\)

see this

why we need SSR or SSE, they are concepts related with variance of a continuous random variable

related URL: https://stats.stackexchange.com/questions/133465/finding-the-mean-and-variance-from-pdf

Third, the definition of the variance of a continuous random variable \(Var(X)\) is \(Var(X) = E[(X-\mu)^2] = \int_{-\infty}^{\infty}{(x-\mu)^2 f(x) dx}\), as detailed here. Again, you only need to solve for the integral in the support. Alternatively, it is sometimes easier to rely on the equivalent expression \(Var(X) = E[(X-\mu)^2] = E[X^2] - (E[X])^2\), where the first term is \(E[X^2] = \int_{-\infty}^{\infty}{x^2 f(x) dx}\) (see the definition of the expectation in the second paragraph) and the second term is \((E[X])^2 = \mu^2\).

see the (x - mu)^2, this is the SST.

what we want to do is weigh the SST(which is fixed when given a dataset each y_i is fixed, the mean of dataset is fixed) coming more from the predict error, or the distance from mean

the larger the SSR, the smaller the SSE, the better the model is

separate the distance between true value and mean of true values (Y_i to mean(Y_i)) into two components: explained deviation and unexplained deviation:

• \(\hat{Y_i}-\bar{Y}\) predict - mean: explained deviation : sum square to SSR
• \(Y_i-\hat{Y_i}\) true - predict: unexplained deviation: sum square to SSE

\(Y_i\) : the true y-value of a point i

\(\bar{Y}\) : the mean of all true y-values of data points

\(\hat{Y_i}\) : the predict y-value of data point i

\(SSR=\sum(\hat{Y_i}-\bar{Y})^2\) : sum of square due to regression

\(SSE=\sum(Y_i-\hat{Y_i})^2\) : sum of square due to error

\(SST=SSR+SSE=\sum(Y_i-\bar{Y})^2\) : sum of square due to total

\(R^2=SSR/SST\)

example one:

Let's see the whole plot (not the up-left coner roc curve) for whether a paper admitted by journal:

• x-axis: predicted probabilities
• y-axis: count of observations
• pixel: each pixel represent a paper
• blue and red: are the true label distribution of a paper(rejected or admitted)

example one:

(0.1, 10) means :

1. (the axis shows) there are 10 papers which you predict an admission probability of 0.1
2. (the region where this point locate in shows) the true status for all 10 papers was negative

example two:

(0.5, 10) means :

1. (the axis shows) there are 20(10 blue, 10 red) papers which you predict an admission probability of 0.1
2. (the region where this point locate in shows) the true status is that 10 papers was negative, 10 was positive.

threshold = 0.5

all predicted probability > threshold, predict it positive; all predicted probability < threshold, predict it negative;

this means that :

all red pixels to the right of the line are correct predictions; all blue pixels to the left of the line are correct predictions;

accuracy rate = correct predictions %

[真正经，原本正经，你判断他是正经的] TPR = (red region on right of threshold) / whole red region TPR = TP / (TP + NF) = 真正 / (真正+假负) TPR = recall

[假正经，原本不正经，你判断他是正经的] FPR = (blue region on right of threshold) / whole blue region FPR = FP / (FP + TN) = 假正 / (假正+真负)

ROC curve is a plot of the TPR on the y-axis versus the FPR on the x-axis, for every possible threshold

One threshold only related to a fix TPR and FPR pair.

Each pair of true positive and true negative distribution form one roc curve;

Each choice of threshold on certain pair of distribution form a point of this roc curve;

The larger predicted probability distance (the x-axis shows the predicted probabilities by our model) between this pair distribution, ===> the more roc curve expand to up left coner, ===> the more area of AUC(area under the curve), ===> the better the model is.

Roc curve visualize ALL possible thresholds. misclassification rate only take s SINGLE threshold

1. better for imbalanced dataset. because it will doesn't change how the roc curve is generated. roc only care about the 'RATE'
2. can do multiple classification by 'one vs. all' approach, and you may should draw 3 curvers instead of 1.
   • 1st curve: class 1(possitive) vs classes 2 and 3(negative);
   • 2nd curve: class 2(possitive) vs classes 1 and 3(negative);
   • 3rd curve: class 3(possitive) vs classes 1 and 2(negative);
3. can do minimize FPR or maximize TPR
   1. minimize FPR: VIP clients admission
   2. maximize TPR: AIDS testing