Today, we are excited to announce the general availability of Metric Thresholds, a new feature that supercharges the performance of Multimetric optimization. Metric Thresholds helps you discover better models that meet your problem-specific needs by allowing you to define “thresholds” on your metrics. SigOpt will use this information to help guide our search to better target the results of greatest interest (those that satisfy the thresholds).
Modelers tackling problems ranging from speech recognition to quantitative trading utilize our existing Multimetric optimization feature to simultaneously optimize a model with regard to more than one metric. This allows them to align their modeling with real business objectives. This capability leverages cutting-edge research to explore the trade-off between 2 metrics and discover the efficient frontier of solutions.
However, we learned from our customers that there are situations in which only a subset of the full efficient frontier is relevant for their business problem. For example, a deep learning engineer tuning a computer vision model might optimize both validation dataset accuracy and inference time, maximizing the former and minimizing the latter. However, their application circumstances requires them to use a model that takes less than 30 milliseconds (ms) to perform inference. While Multimetric optimization enables the modeler to simultaneously optimize the two objectives, there are accurate but slow models that SigOpt will spend budget exploring. These accurate but slow models are irrelevant because their inference time is greater than 30 ms. With standard Multimetric optimization, there is no way to factor in this problem-specific criteria; this ultimately results in fewer models in the region of interest. Comparable examples exist in quantitative trading where a trader is maximizing return but has a threshold for their risk metric.
Metric Thresholds is a new feature that solves this problem by allowing modelers to define these unique problem-specific criteria as “thresholds” on one or both metrics in a Multimetric experiment. This criteria informs the SigOpt platform during the optimization process. It helps us both increase the number of models discovered during optimization that meet your problem-specific criteria, and it helps you better understand your metrics through the experiment insights dashboard.
In the previous example, the modeler would set a threshold of 30 ms on their inference time metric when creating the experiment through the API. The SigOpt optimization engine utilizes this information to focus on the region of interest during the intelligent optimization process. Additionally, thresholds are visualized in the web interface to aid with analysis.
Getting started with Metric Thresholds requires just a few lines of code. You can specify a threshold value for one (or both) metrics when creating the experiment. Thresholds can also be updated mid-way through an experiment from the API or through the properties page of the web dashboard. SigOpt will try to find observations that are greater than or equal to the specified threshold (or less than or equal to if you are minimizing the metric). Check out the documentation for example code!
Metric Thresholds is now in general availability for customers with access to Multimetric optimization. Please reach out to our customer success team to learn more.
Use SigOpt free. Sign up today.