Background:
Machine Learning is probably one of the most misunderstood concepts in the modern tech world. People often envision all these complicated computations happening inside a black box of a machine-learning based prediction algorithm. While some prediction algorithms are definitely more complicated than others, most algorithms generally boil down to some linear algebra and minimizing a loss function. In general, we know what is happening under the hood of these supervised learning tasks. However, there are some algorithms that even the data scientists and researchers who implement the algorithms have no idea what is going on.
These truly black-box algorithms generally involve using Neural Networks to solve various learning problems. And for some tasks they work absurdly well. Convolutional Neural Networks can achieve human-level accuracy in image classification and AlphaZero can destroy state-of-the-art programs and grandmasters at chess. This leads people to try and use Neural Nets to predict/classify everything from weather to Bitcoin prices.
So, I decided to try to apply a Neural Net to a problem I was working on: predicting Over/Under data for basketball games from only knowledge of previous games. My thought process was that Vegas might have a biased model which might overreact to Over/Under results from recent games. At first, I tried to exploit this with a Hidden Markov Model, but the results were inconclusive—frequencies of various patterns could not be distinguished from a random walk with any level of confidence. Then, I came across a paper that claimed very good prediction results for time-series data using an LSTM (Long Short-Term Memory) Neural Network. That got me pretty excited.
The Paper:
The link to the 2019 paper can be found here, and it generalized an approach to predicting US-INR currency values found here. What made me salivate was how good the paper’s final neural net (2 hidden layers of 50 nodes each) seemed to perform on the test data. Here was a graph that both papers presented as proof of “success” or their approach.
One thing someone should always be skeptical of is the very good r^2 value presented in these papers. Here they had an r^2 value of around 99.86%, which is very suspicious. It didn’t take me too long when trying to replicate their work (their source code can be found here btw) to figure out that their whole approach was a sham. Look at a zoomed in version of the previous graph to see if you can understand what I’m talking about.
What you can see a lot more clearly in this second graph is the relationship between the predicted value of price and the actual values: the predicted value is basically just a day-behind shifted value of the actual exchange rate. On one hand this is logical, you would expect the exchange rate not to fluctuate too much day-to-day, so predicting something close to the previous day’s close makes a lot of sense. The neural network, if anything, was able to learn this local relationship of slow price changes. Because there wasn’t a lot of variation in rates the r^2 value got really good.
Then, to see if Neural Networks have any advantage over common sense, I mapped the test data vs. the day-shifted test data and calculated the r^2 score for the result (graph below). I got an r^2 value of 99.88%: a value higherthan the one I got using a complicated Neural Network. In Data Science, if you have a simpler model that can be easily explained and performs better: there is no question that it’s the model you go with.
What led me to figure out that the LSTM Neural Net wasn’t that great at predicting this data in the first place was when I tried to use it to try to predict actual % changes of exchange rates day-to-day (something both studies conveniently excluded). When I tried this approach, I got an r^2 value of .0001. In other words, the Neural Network does no better than guessing. So much for that idea.
What’s shocking is that the 2019 paper actually got published in the International Journal of Innovative Technology and Exploring Engineering. This means the paper got through the peer-review process despite not providing any substantive results. All this goes to show how wide-open the field is for time-series prediction and Neural Network approaches in general.