The Problem:
Defining, selecting, and optimizing with the right set of metrics is critical to every modeling process, but these steps are often hard to execute well.
Useful models require the modeler to select the right set of metrics, then maximize or minimize them during the model training and tuning process. But there are often a variety of metrics that matter and it can be tough to track all of these through the experimentation process.
These different metrics matter for different reasons, so need to be evaluated differently in training and tuning jobs. And even when you develop composite metrics, there are typically cases where you need to balance tradeoffs between a couple metrics, and potentially maximize or minimize both at the same time. Modelers often struggle with metrics because they have to piece together disparate tools to effectively manage them.
The Solution:
In any business it’s crucial to establish a guiding metric or metrics, and models are no different. Businesses often have shifting priorities, or regulatory constraints that might oppose other business objectives. For modelers making sophisticated, labor-intensive decisions about their model using a variety of metrics, SigOpt provides a metric management suite that helps you find production-ready models efficiently and confidently. Within our optimization platform, Metric Management provides analysis, optimizer guidance, and failure reporting for all of your metrics that help you explore, understand, and advance your model along the metrics you care about.
Within Metric Management, SigOpt specifically offers the following:
Stored Metrics:
Store up to 50 metrics for every training run, for later analysis
Metric Strategy:
Specify how to optimize a specific metric: minimize, maximize, apply a constraint, or simply store the metric
Metric Thresholds:
Establish minimum or maximum values for metrics you are optimizing
Metric Constraints:
Provide a bounding function around where you want SigOpt to search, and modify its parameters while your model is tuned
Multimetric Optimization:
Select two metrics to optimize against to understand trade-offs in one metric versus a second
Metric Failures:
Report failures to SigOpt to help guide the optimization process away from regions that are not feasible to train or evaluate a model
Alone, each feature serves a targeted purpose, but together, all of these are arrows in your modeling quiver, encouraging your team to arrive at just the right model for your current business needs, in a minimum of wall-clock time.
What Makes SigOpt’s Metric Management Unique:
SigOpt provides a “human-in-the-loop” process so that you can guide the platform’s discovery of a more effective set of parameters for your model. This process is called Metric Management, and is enabled by a variety of tools built into our API and web interface.
No other optimization solution, open source, off the shelf, or otherwise affords you this level of flexibility in assessing your model’s performance, and in allowing your data scientist to adjust and guide the tuning process to this extent. Although multiple metrics are possible with a few other solutions, setting detailed guardrails on your tuning process typically cannot even be found piecemeal in any other optimization platform on the market today.