SigOpt Recap with Anastasia AI: Democratizing Time Series Forecasting for Any Industry

Steven Stein
AI at Scale, Application, Applied AI Insights, Artificial Intelligence, Augmented ML Workflow, Hyperparameter Optimization, Intelligent Experimentation, Model Type, RNN, Time Series, Training & Tuning

Overview

In this blog post we are reviewing the  SigOpt Summit presentation by Anastasia on how they are democratizing AI for their customers. During this talk, Anastasia detailed how they used SigOpt to reduce error and randomness when training time series models using XGBoost by using a systematic approach to hyperparameter analysis. Using SigOpt’s dashboard and Intelligent Experimentation tools, Anastasia ran experiments to find why their models produced inconsistent results, used SigOpt to pinpoint which parameters were causing the issues, fixed the issues, then used SigOpt to validate the new models produced consistent quality for their customers.

Anastasia

Anastasia was founded in 2018 with the goal of making AI easier to deploy for small and medium sized businesses. Anastisia knows that all companies need AI, but not all companies have access to AI. Specifically, AI is complicated and difficult to apply to specific business use cases. Therefore, Anastasia’s mission is to democratize AI for smaller businesses by delivering pre-packaged products and services to help companies build solutions quickly for specific use business purposes. For companies that lack the internal resources to build AI products in-house, Anastasia offers products to easily build AI products while hiding the AI’s massive technical complexity.

Time Series Forecasting

Anastasia’s primary customers are in the retail and supply chain industries where time series prediction is critical. Today’s world of just-in-time manufacturing and supply chain delivery, being able to predict when and where a product is needed is of critical importance. Companies have developed massive inventory and supply chain systems to reduce carrying costs and ensure products arrive precisely when they are needed. Failure to predict when products are needed results in product shortages and customer dissatisfaction. And today, Covid unfortunately unscored the need to have robust and predictable supply chains. Anastasia’s solution is to deliver trained machine learning models to companies that need them to help them predict when and where to deliver and expect products using time series forecasting.

Forecasting needs to be precise, fast and cheap, but building models of future events is messy and the data used to build these models are often missing critical pieces of information. Therefore, models that predict time events output messy graphs with very large error bars.

In addition, building and training these time series models requires vast amounts of compute resources. These compute resources are expensive and complex to operate, especially for small and medium size businesses. Anastasia has built an expertise in delivering accurate and reliable time series models. For instance, when working with one of their customers, Anastasia produced astonishing results for their supply chain tracking systems:

Why using SigOpt

​​Anastasia chose to work with SigOpt to ensure the models they delivered to customers were of the highest quality. Anastasia knew that to be best in their space required validation and ensure their models produced the highest performance they could to remain competitive and win business. Also, the SigOpt Intelligent Experimentation platform integrated easily with their systems and produced clean and easy to use dashboards when running their experiments. Specifically, Anastasia was able to improve the error rates on their models by 11.79% when forecasting 113,337 SKUs due to the hyperparameter optimizations provided by SigOpt.

Time Series Model Quality Reproducibility

Being able to reduce the error rate for a supply chain system is one vector of value, but in time series forecasting, reducing randomness is just as important. To train their models, Anastasia uses the XGBoost framework. When training their models, they found there was a large amount of randomness built into the data used to train the model which caused the models themselves to vary widely in quality. Each model training session is very expensive computationally and thus reducing the amount of training runs to produce a high quality model is very important. ​​Randomness makes everything slower and increases costs. Lack of certainty on the results adds unwanted operational risks and makes it difficult for Anastasia to differentiate from competition since the quality of the results are random and cannot be directly assured.

When running their experiments in the SigOpt Intelligent Experiment Platform, Anastasia ran multiple experiments to determine how to create models with reduced randomness quicker. What Anastasia found by visualizing the runs and down-selecting on the metrics they wanted to track was that some runs produced very low randomized results. Unfortunately, it took them 600+ runs to find the optimal experiments.

When using Sigopt’s ability to order the parameters in terms of how they affect the models, Anastasia learned the initial conditions used to train the models greatly affected the randomness of the model quality. These random initial conditions mattered more than the data being used to train the model or the type of hyperparameters being selected. This visualization of ranking the order of the importance of parameters was easy to use, clear, and provided exactly the insight Anastasia needed to reduce randomness in their models.

Results

To fix the problem of initial conditions affecting the quality of models, Anastasia built their own recursive neural network which is able to better absorb initial conditions and therefore better control the quality of their models by eliminating randomness.

SigOpt was able to verify this new proprietary Recursive Neural Network was able to dramatically reduce the amount of randomness with much fewer training runs. In fact, it only took approximately 20 runs to reduce the randomness to more predictable levels; which is an order of magnitude lower than previous runs.

The SigOpt output of parameter importance verified that the new Recursive Neural Network was able to solve initial condition randomness by showing the most important parameter was now how past data affects the models. The new Recursive Neural Network is not tricked by the fuzziness of the data and SigOpt was able to validate this result.

Take Action

To try SigOpt today, you can access it for free at sigopt.com/signup.

PXL_20211015_211818478.PORTRAIT
Steven Stein Product Marketing Lead

Want more content from SigOpt? Sign up now.