Here at SigOpt, your thoughts and problems matter to us. We’ve heard the struggles around building models for deployment and empathize with the need to trust the solutions you arrive at. After countless hours of research, we’ve implemented several changes that significantly impact the way Multisolution can be applied to an experiment.

Here’s what’s new:

1. More options: There are no longer hard limits on the number of solutions that Multisolution provides. You will be able to set the number of solutions SigOpt will return to any number lower than the total budget of your experiment. While a higher number of solutions doesn’t necessarily mean that the answers will be more useful (we recommend that you return no more than 20% of explored points to be returned solutions), our previous limits were too restrictive for regular application.
2. More exploration: Categorical parameters, as opposed to numerical or interval parameters, will now be supported with Multisolution. Leveraging categorical parameters for optimization can be a difficult task, and we hope that being able to include these parameters in your Multisolution search will provide more clarity into your experimental process.
3. Account for guardrails: Metric Constraints will now be supported to use with Multisolution. Applying metric constraints to non-optimized metrics can play a huge role in keeping your optimization experiment more focused on the things that matter to you. Whether it’s staying above a certain accuracy level, or below a specific inference time, metric constraints ensure that the solutions that SigOpt looks for meet the minimum requirements you’re looking for.

Pursuing confidence

If you have built any kind of model to predict or simulate a real-world scenario, then you know the familiar roller coaster feeling of seeing what you thought was a consistently performant model behaving in totally inconsistent ways once deployed. Modeling in the real world frequently poses a mismatch problem between the developing and production environments. This could be due to temporal differences from training and production (or unseen) data, or because there is an intrinsic difference between both settings. Either way, it is common to treat your development results with some healthy skepticism.

A modeler’s dilemma: finding out that the most promising model parameters in your development process don’t behave the way you expect in real life

In these cases it is often useful to consider multiple distinct and diverse parameter configurations that still have high performing values. You may look for parameter configurations and alternatives that are more “stable” —  representing areas where small changes in the parameters do not change drastically the metric values. And you may even end up selecting solutions that lie within this region in lieu of one with a higher performing metric result, knowing that this may not actually manifest in real life.

How can Multisolution help?

While there is no one-size-fits-all answer to this common modeling challenge, there are tools that can help you build confidence in the solutions you select. Our advanced feature Multisolution addresses this by searching for feasible regions of hyperparameter configurations that yield high-performing metric values and returning a set of solutions with diverse configurations to meet your business goals. While we can’t decide on the best model for your particular application, SigOpt Multisolution offers you the ability to explore and narrow down options to make the most informed choice you can.

The same modeler’s dilemma, but with alternative configurations provided by SigOpt’s Multisolution that highlight high-performing parameter regions that operate more consistently on unseen data

Great, when can I start?

Multisolution is available for all SigOpt users, big or small! If you don’t already have a SigOpt account, you can sign up here for free!

Check out our documentation here, and post your questions on the community forum for us to check out. Stay tuned for more posts and a deep dive into the intuition behind this feature!

Happy Modeling,
Joyce