In earlier blog posts we introduced Multimetric Optimization (here and here), an advanced feature to optimize a model with more than one metric. Our users often encounter scenarios where they need to solve for competing objectives, such as maximizing both accuracy and inference speed. Multimetric Optimization enables you to intelligently weigh these tradeoffs by discovering a frontier of the best possible models across these competing objectives.
In addition to the advanced algorithms powering Multimetric Optimization, we have built analysis tools into our web dashboard that allow you to consider these tradeoffs. We are excited today to announce improvements in analyzing and viewing Multimetric experiments with the Experiment Insights dashboard.
To that end, we have made the following changes:
You can now view parameter importances by metric in the analysis page for Multimetric experiments. Parameter importances represent the predicted complexity of the objective response relative to perturbations on this parameter and can be used to understand how parameters play different roles for each metric. The new parameter importances visualization tool allows sorting by name or importance value for each metric. Importances can also be retrieved from the API if you would like to perform a more detailed analysis.
Differing parameter importances between metrics can provide valuable insights into metric behavior.
The Best Metrics frontier has replaced the Experiment Improvement graph on the experiment summary page. The Best Metrics frontier visualizes the tradeoffs between your competing objectives by plotting the set of Pareto-efficient Observations. And now, you can click the points on the frontier to view detailed suggestion and observation information in a pop-up modal. Information in this modal, like model evaluation time and metadata, can aid you as you utilize your domain expertise to select between models. Combine this feature with dragging to zoom to perform fine-grained analysis of the frontier and select the model that delivers the greatest business outcome.
Drag, zoom and click to analyze the frontier of best models.
We have separated out some graphs that previously were combined on the Analysis Page. Experiment improvement graphs for both metrics have been separated for easier viewing. These show you how each of your metrics are increasing over the course of the experiment.
Experiment improvement is now shown for each metric.
Additionally, you can use the two separate Experiment History graphs (one for each metric) to understand how certain parameters (or metadata fields) affect metric values differently.
Notice that layer_1_nodes has a positive correlation with accuracy and an inverse correlation with negative_num_parameters.
We hope that these changes will further amplify the impact of customers running Multimetric experiments. If you have any feature requests, feedback, or want to run Multimetric experiments, please contact our Customer Success team.
Use SigOpt free. Sign up today.
