Financial data is one of kind: we think is non-stationary, and the samples are non-iid, which means one cannot simply apply common machine learning techniques. In this post, we briefly cover some pitfalls when using machine learning algorithms in a financial context.

1. Harry Potter and the model that predicts prices

Internet is full of articles claiming it is possible to predict prices using neural networks (often LSTM). They even show the final predictions of the models is the same than the test set. However, there is a catch: the model just learnt that a good prediction for the price at \( t \) is the price at \( t-1\). There is nothing meaningful in that prediction, since is just a confirmation that prices strongly follow a Martingale process [3], and it will probably lead to nowhere.

When working with machine learning models, it is often advisable to use returns as some properties are more stable and the magnitude of the returns actually is stationary.

2. Advanced Potion-Making and baseline models

Do you remember when Harry had a better book than Ron’s and Hermione’s in The Half Blood Prince? How did we know it was a better book (model)? Because we had another book that lead to worse results, even for Hermione!

Always use a baseline model to compare your model against. Prepare a set of metrics that cover the areas in which you are interested and compare both (or more) models.

For example, in the previous pitfall we can clearly see doing all the work with the model (data preparation and training) is a waste of time when we compare it with the shifted time series, which seems to be an appropriate baseline model for that case.

3. The polyjuice potion and predict _proba

Almost all machine learning libraries allow models created withing their frameworks to use a method called predict_proba to “predict probabilities”. Much like when someone uses the polyjuice potion in Harry Potter, what we see is not the truth.

These numbers are not probabilities or rather, they are not calibrated probabilities. To use these predictions as if they were probabilities, we need to calibrate them.

If you are using Python and Scikit-Learn, the library provides you with utilities to calibrate your models. Damien Martin has a great post about this topic.

4. Stationarity tests and the Erised Mirror

According to Dumbledore, the Erised mirror shows us the most deepest desire of our hearts. When analyzing a financial time series, I often desire it to be stationary. To test for stationarity, we have a bunch of tests, like Dickey-Fuller, that work based on strong assumptions about our data.

Mainly, two assumptions are typically made [2]:

• There is enough data for the test to work.
• The underlying process is autoregressive.

These tests are just mirrors. In [2] you can find empirical experiments to justify the aforementioned reasons.

5. Concept Drift and The Sorting Hat

We already saw that attempting to prove financial series are non-stationary is doomed to fail. However, the properties of the distribution do change over time, and models that do not take into account this phenomenon will surely end up in Azkaban.

Concept Drift encapsulates this problematic: it is the change in the relationship between input data and the target over time [1].

Some ways of dealing with concept drift are:

• Ensemble learning. A set of models will likely be more robust than a single one, specially when each model is trained to react for different conditions.
• Online learning. Incremental, or online, learning can be used to update the model every time a new observation is received. This is specially appropriate for streams of data, like financial time series, where we are constantly receiving new samples. The Kalman Filter can easily be used within this framework. For machine learning algorithms River lets you work in an online-fashion. River also includes concept drift detection algorithms that may be useful.
• Dynamic labels. Generating the supervisory signals based on the current conditions can help your models to adapt more quickly to a changing market. We have already discussed some solutions to this problem in Quantdate.

References

[1] R. C. Cavalcante and A. L. I. Oliveira, “An approach to handle concept drift in financial time series based on Extreme Learning Machines and explicit Drift Detection,” 2015 International Joint Conference on Neural Networks (IJCNN), Killarney, Ireland, 2015, pp. 1-8, doi: 10.1109/IJCNN.2015.7280721.

[2] Yves-Laurent Kom Samo – Stationarity and Memory in Financial Markets

[3] Grahan Giller – Why you Should Stop Predicting Prices if you want to Stand a Chance of Predicting Prices