post list
QuantDare
categories
artificial intelligence

Classification trees in MATLAB

xristica

artificial intelligence

Applying Genetic Algorithms to define a Trading System

aparra

artificial intelligence

Graph theory: connections in the market

T. Fuertes

artificial intelligence

Data Cleansing & Data Transformation

psanchezcri

artificial intelligence

Learning with kernels: an introductory approach

ogonzalez

artificial intelligence

SVM versus a monkey. Make your bets.

P. López

artificial intelligence

Clustering: “Two’s company, three’s a crowd”

libesa

artificial intelligence

Euro Stoxx Strategy with Machine Learning

fjrodriguez2

artificial intelligence

Visualizing Fixed Income ETFs with T-SNE

j3

artificial intelligence

Hierarchical clustering, using it to invest

T. Fuertes

artificial intelligence

Markov Switching Regimes say… bear or bullish?

mplanaslasa

artificial intelligence

“K-Means never fails”, they said…

fjrodriguez2

artificial intelligence

What is the difference between Bagging and Boosting?

xristica

artificial intelligence

Outliers: Looking For A Needle In A Haystack

T. Fuertes

artificial intelligence

Machine Learning: A Brief Breakdown

libesa

artificial intelligence

Stock classification with ISOMAP

j3

artificial intelligence

Sir Bayes: all but not naïve!

mplanaslasa

artificial intelligence

Returns clustering with k-Means algorithm

psanchezcri

artificial intelligence

Reproducing the S&P500 by clustering

fuzzyperson

artificial intelligence

Random forest vs Simple tree

xristica

artificial intelligence

Clasificando el mercado mediante árboles de decisión

xristica

artificial intelligence

Árboles de clasificación en Matlab

xristica

artificial intelligence

Redes Neuronales II

alarije

artificial intelligence

Análisis de Componentes Principales

j3

artificial intelligence

Vecinos cercanos en una serie temporal

xristica

artificial intelligence

Redes Neuronales

alarije

artificial intelligence

Caso Práctico: Multidimensional Scaling

rcobo

artificial intelligence

Confusion matrix & MCC statistic

mplanaslasa

24/06/2015

1
Confusion matrix & MCC statistic

In the field of predictive analytics, a confusion matrix is a table that allows the visualization of the performance of an algorithm whose objective is to predict the classes of a variable.

The name “confusion” comes from the fact that it makes it easy to see if the system is mislabelling one class as another.

The confusion matrix shows two rows and two columns that report the number of false positivesfalse negativestrue positives, and true negatives.

confusion

  • True Positive, TP: Number of data correctly predicted as a member of the class
  • True Negative, TN: Number of data correctly rejected as a member of the class
  • False Positive, FP: Number of data incorrectly identified as a member of the class
  • False Negative, FN: Number of data incorrectly rejected as a member of the class

This allows for a more detailed analysis than, for example, the mere proportion of correct guesses (accuracy). The confusion matrix is a more reliable metric for the real performance of a classifier because it won’t yield misleading results if the data set is unbalanced (that is, when the number of samples in different classes vary greatly). This is the main advantage of representing the results by using this structure, we must avoid statistics which return a misleading high success rate.

We could say that it may be easy to compare two confusion matrices, for example, you can say the confusion matrix M2 is better than confusion matrix M1, below:

matrix

Obviusly, two matrix only can be compared if they are based in the same data, so they have to sum the same.

Unfortunately, two confusion matrices are not always trivially comparable.  In fact, two confusion matrices M1 and M2 are trivially comparable if and only if:

trivially1

or

trivially2

In case two matrices are trivially comparable, the matrix with lower number of false positives and false negatives can then safely be called a better prediction.

So, what can we do if two confusion matrices are not trivially comparable? How can we compare them then? A good option is using the Matthew’s Correlation Coefficient (MCC), which helps us to represent the confusion matrix with a single value. While there is no perfect way of describing the confusion matrix of true and false positives and negatives with a single number, the Matthews correlation coefficient is generally regarded as being one of the best such measures. It is a measure of the quality of binary classifications and due to the fact that it takes into account the true and false positives and negatives, it can be used even if the classes are of very different sizes.

The MCC is basically a correlation coefficient between the actual and predicted series. It is bounded and returns values between −1 and +1. A coefficient of +1 represents a perfect prediction, 0 no better than random prediction and −1 indicates total disagreement between prediction and observation.

The MCC can be calculated directly from the confusion matrix by the formula:

mcc

For example, we have developed a methodology to predict trends of an index by using its corresponding Implied Volatility Index. We suppose that the implied volatility index can exhibit 3 possible scenarios: A, B and C:

scenarios

Our selected index in this test is the S&P 500 (so the Implied Volatility Index considered is VIX). We have identified its trends in a posteriori analysis and we have 3 real classes: upward trends, downward trends and ranged market. Remember that our prediction only predicts upward and downward trends.

The results of our predictions show that in essence the expected correspondence between the scenarios A and C and the actual market of the index come true:

legend

pie

But as the actual upwards trends predominate in the whole period, when we look directly at the frequencies, the results do not look as good:

freq

We are going to compare the MCC against the Accuracy statistic (defined as the number of correctly classified/total number).

The MCC’s values of the two classes we wanted to predict show superiority of the upward class but the two values are not very encouraging. The MCC mean of this test shows poor precision:

mcc_table

However, if we only had worked out the accuracy statistic, the value would have shown an acceptable result:

acc_table

In fact, the accuracy of the upward class would have been extremely good. This value is so high due to the predominance of actual upward trends of the index.

Therefore, the conclusion of all of this is clear: we have to take into account any existing bias in the actual data and choose good indicators to analyze our results in order to avoid making the wrong conclusions.

Tweet about this on TwitterShare on LinkedInShare on FacebookShare on Google+Email this to someone

add a comment

[…] Confusion matrix & MCC statistic [Quant Dare] In the field of predictive analytics, a confusion matrix is a table that allows the visualization of the performance of an algorithm whose objective is to predict the classes of a variable. The name confusion comes from the fact that it makes it easy to see if the system is mislabelling one class as another. The confusion matrix shows two rows and two columns that report […]

wpDiscuz