What is Hyperopt? 

Hyperopt is a black-box optimization framework where users specify: 

• a search space for an optimization problem 
• an objective function to optimize 
• a database to store the optimization history 

The framework provides several search algorithms. Like SigOpt, Hyperopt’s algorithms determine which points are selected from the search space and then evaluated by the objective function at each optimization step. SigOpt intelligent experimentation offers a similar workflow where: 

• The optimization algorithms are dynamically and automatically selected according to the problem – users don’t need to worry about selecting the “best algorithm”. The SigOpt HPO engine has an ensemble of different optimization strategies that change based on the features of your optimization problem.  
• The algorithms are regularly updated and tested by experts in black-box optimization. The SigOpt Research team is constantly evaluating the HPO engine to ensure that it is state-of-the-art. 

Why use Hyperopt? 

In reality, it is not easy to answer whether one optimizer is better than another — every non-convex optimization problem is different and has its own challenges. So why, and when should you consider an alternative like Hyperopt over SigOpt?  

Typically, we expect SigOpt HPO to be more sample-efficient than HyperOpt; in short, you should get better results faster with SigOpt. However, for problems where the objective function is very “cheap” to evaluate (for example, less than half a second), or you wish to control your optimization search procedure, HyperOpt is a great tool. It allows searching over non-conventional, “awkward” search spaces, with arbitrary nested conditional parameters, whereas SigOpt allows for a more structured conditional search. 

Additionally, SigOpt offers a platform for experiment management. All progress and information are safely stored within SigOpt, which means you won’t accidentally overwrite your result files. Our RESTful API allows you to save your optimization history quickly. This includes the details of your favorite model, extra annotations, multiple performance metrics, and much more. On one hand, Hyperopt’s open-source nature provides excellent flexibility for specific applications. On the other, it requires users to maintain the HyperOpt platform and track necessary artifacts on their own. 

To combine Hyperopt’s HPO flexibility with SigOpt’s experiment management platform, we developed the sigopt.hyperopt integration. Look for more info soon on how to integrate HyperOpt into your experiments with SigOpt.