This post is part of a five-part series. Follow these links to read any post in the series:
- Post 1 – Why Enterprise AI is Actually Three Markets in One
- Post 2 – Differentiated Models are Eating the World
- Post 3 – Technology Considerations for Machine Learning Operations
- Post 4 – Modeling with the Modern Machine Learning Stack
- Post 5 – How Teams Use SigOpt to Build Differentiated Models
Developing models requires a different workflow than developing traditional software. Between feature engineering, model training and everything in between, there are many variables that make modeling a messy, experiment-driven process. And if this work is unsupported with technology, it also requires a wide variety of skills, touching data science, domain expertise, devops and mathematical optimization.
This post discusses technology that can simplify this modeling process and how this technology is different depending on whether a team is developing basic or differentiated models. This is the third post in this series. In the first post, we discuss the Enterprise AI landscape and how it actually looks like three markets in one. In the second post, we explain why differentiated modeling – one of these three markets – represents the biggest opportunity in Enterprise AI.
The modeling steps required for developing basic and differentiated models are roughly the same. There is a data engineering step to prepare data, develop pipelines and engineer features. There is a model engineering step to train, tune and evaluate models. And there is a model deployment step to serve models, run inference on them and create processes for monitoring in production. This is an oversimplified version of this rather extensive workflow, but these rough components are represented in the visual below.
Summary of the modeling process and how SigOpt fits
Where basic and differentiated modeling differ are the requirements for technology supporting this workflow. There are three broad differences in the way software is designed to support these processes that further draw out this distinction.
Augment, not automate
In general, basic models seek technology that automates as broad a set of these tasks as possible, even if they could benefit from domain or modeling expertise. An example of this, most platforms designed for supporting basic models, such as end-to-end AutoML, will automate feature engineering. Building differentiated models, however, requires technology that augments the core domain and modeling expertise of modelers. So in the case of differentiated model development, modelers need access to supporting tools like a feature store, but do not need automated feature engineering. The feature store gives them access to past insights on features developed by them or their colleagues so they can more quickly understand and integrate these insights into their modeling workflow. This approach is designed to augment the expertise of the modeler rather than automate it away.
Design to fit workflow, not workflow fit to design
From three perspectives, it is important for differentiated modeling teams to acquire technology that fits any workflow. It frees modelers to apply the best modeling techniques to any given problem. It gives modeling teams a better chance of hiring and retaining the best talent. And it empowers the business to phrase as many problems in modeling terms as possible without being constrained by a tool set’s capabilities. If you have experts, you want to empower them with as many degrees of freedom as possible to do the best possible job. Alternatively, it is important for basic modeling to be guided and uniform. The purpose is to make simple problems easy to solve – even with a click of a button, where possible. If this requires putting datasets in the same format or applying the same modeling techniques to a wide range of simple problems, this is worth it if it makes it easier for a broader set of citizen data scientists to address the problem.
Navigate complexity, rather than avoid it
To build a differentiated model often requires running a variety of experiments across a wide range of models. These are often messy processes that require a combination of domain expertise and modeling techniques to arrive at a high-performing solution that makes accurate predictions in and out of sample. In the process, modelers usually need to embrace complexity, whether it comes in the form of data quality control, model comparisons, architecture search, training run analysis, hyperparameter tuning or cluster management that underpins all of it. Alternatively, to build a basic model often requires avoiding as much complexity as possible to quickly get to a viable solution. Create a quality tabular dataset, apply traditional machine learning models and run a grid search behind the scenes across a couple important parameters. Differentiated modeling processes require solutions that accommodate complexity and arm modelers with insights that help them navigate it. Basic modeling processes require solutions that largely ignore this complexity and arm modelers with tooling that helps them avoid it.
These broad differences translate into a tendency for basic model builders to need all-in-one solutions that do an 80% job across a variety of tasks and differentiated model builders to need best-in-class solutions that do a 100% job for their particular tasks.
So what does this mean for the ideal modeling process? In the next post, we will provide an overview of the modern machine learning stack and how it gives modelers the best chance to build models that impact the business.
We’d love to hear how it stacks up to your experience. Email us at [email protected], try our product or sign up for blog updates.