Blog | SigOpt

Modeling Managed

SigOpt’s Experiment Management solution helps you track, filter and reproduce your entire modeling workflow, with interactive visuals you can augment your intuition, understand your progress and explain your results.

Keeping track of it all: recording and organizing model training runs

This blog discusses the Runs feature and how to add it via API to track training runs as part of Experiment Management.

Establishing a robust pipeline for successful models requires more than a Jupyter notebook and a GPU instance.

Each new observation in the process of black-box optimization assumes there will be infinite possibilities to guess again, typically in multidimensional hyperparameter space.

When it comes to predictive analytics, there are many factors that influence whether your model is performant for the real-world business problem you are trying to address.

Experimenting with data and machine learning can prove useful, but productively collaborating with a team to hand off models from training to production is an entirely different process, and one that can lead to real business transformation.

SigOpt offers the most complete set of advanced experimentation features for model optimization

Metric Strategy enables you to track metrics, set them as constraints, or optimize across multiple at once

When designing models, it’s essential to track as many metrics as you find useful, and ultimately pick one or two to optimize when developing your model.

SigOpt offers our software free to academics, nonprofits, or labs who are performing research they plan to publish.

Experiments automates hyperparameter optimization with an ensemble of Bayesian and global optimization algorithms

The SigOpt Dashboard helps you organize model development and visualize training and tuning runs

SigOpt Runs allows you to track machine learning training runs with a few lines of code

SigOpt includes a unique combination of operational features and intelligent hyperparameter optimization

SigOpt partnered with MLconf on a webinar that focused on practical best practices for metrics, training, and hyperparameter optimization.

SigOpt Research Lead Michael McCourt discusses results from the NeurIPS Black Box Competition

SigOpt partnered with MLconf on a webinar that focused on practical best practices for metrics, training, and hyperparameter optimization.

How to think about the combination of metrics, training, and tuning for machine learning

This blog post will go over why you should use Bayesian optimization in your modeling process, the basics of Bayesian optimization, and how to effectively leverage Bayesian optimization for your modeling problems.

How B.Next applies SigOpt to optimize models in COVID-19 research

SigOpt partnered with MLconf on a webinar that focused on practical best practices for metrics, training, and hyperparameter optimization.

Learn the results from a recent set of polls during a SigOpt + MLconf Webinar

SigOpt collaborated with MLconf on a webinar to discuss best practices for metrics, training, and hyperparameter optimization for developing high-performing models.

In a talk at Ray Summit, Meghana Ravikumar discusses Efficient BERT

Modeling can oftentimes feel like a crapshoot.

AFWERX Selects SigOpt as a Machine Learning Operations Leader

This post walks through how to get Efficient BERT up and running in your modeling workflow.

When you’re trying to train the best model, especially on a deadline, there should be a method to the “madness.

This work was originally published here, and has been reproduced with the author’s permission.

Quasi-Monte Carlo (QMC) methods can sometimes speed up simple Monte Carlo (MC) calculations by orders of magnitude? What makes them work so well?

SigOpt is a proud sponsor of and contributor to QMCPy, an open source project and community dedicated to making quasi-Monte Carlo (QMC) methods more accessible that was launched last month.

Just as we’ve recently witnessed broader and broader use cases from some of our industrial customers, numerous academic groups are finding success using SigOpt to optimize and improve the accuracy of Natural Language models.

Models that were trained on pre-pandemic datasets and fine-tuned with pre-pandemic intuition may no longer make relevant predictions. Learn how how you can solve this challenge of model drift.

BERT performs well on question-answering tasks, but as a model, it is both structurally large and also resource-consuming.

As you build out your modeling practice, and the team necessary to support it, how will you know when you need a managed hyperparameter solution to support your team’s productivity? As you first start to optimize your business’s fraud detection algorithm or recommender system, you can tune simpler models with easy-to-code techniques such as grid […].

In this webinar, Machine Learning Engineer Meghana Ravikumar, shares how she used Experiment Management to guide her model development process for BERT.

This is the third post in this series about distilling BERT with Multimetric Bayesian Optimization.

This is the second post in this series about distilling BERT with Multimetric Bayesian Optimization.

This is the first post in this series about distilling BERT with Multimetric Bayesian Optimization.

I think we can all agree that modeling can often feel like a crapshoot (if you didn’t know before, surprise!).

BERT is a strong and generalizable architecture that can be transferred for a variety of NLP tasks (for more on this see our previous post or Sebastian Ruder’s excellent analysis).

How teams use SigOpt to explore modeling problems, understand model behavior, and advance models to production at scale

Researchers Rafael Gomez-Bombarelli from MIT and Simon Axelrod from Harvard collaborated on training and tuning models that generate three-dimensional geometric data based on 2D geometric definition strings for molecules.

Essential capabilities for any modeling team to scale machine learning development and production

This blog post presents some results presented by myself and my Ph.

In this discussion, Fay Kallel, Head of Product, and Jim Blomo, Head of Engineering, joined Product Marketing Lead Barrett Williams to demo Experiment Management, the latest solution from the SigOpt team.

Modeling can be a messy ordeal.

Srinivas, Krause, Kakade and Seeger's work on UCB in BO has received ICML's 2020 Test of Time award.

Considerations and requirements for any team building their machine learning technology stack

If you’re a SigOpt veteran, you’ll know that we provide best-in-class optimization for a wide variety of machine learning models.

This is the second of a three-part series covering different practical approaches to hyperparameter optimization.

Software is still eating the world and modeling is eating software development.

Efficient BERT uses Multimetric Bayesian Optimization to explore the tradeoff between model size and model accuracy

This blog discusses the Runs feature and how to add it via API to track training runs as part of Experiment Management.

Defining Enterprise AI in terms of the modeling approach companies adopt, and the outcomes the expect

Dr Michael McCourt and Dr Alexander Wong share their experience working on COVID-Net, a neural network to detect COVID-19

SigOpt has expanded to include a number of advanced features to help you save time while increasing model performance through experimentation.

When it’s time to ensure that your model, whether that’s a recommendation system, computer vision classifier, or trading strategy is performing as well as it possibly can, you’ll need to choose the right hyperparameter optimization strategy.

Defining, selecting, and optimizing with the right set of metrics is critical to every modeling process, but can be tough to get right.

SigOpt customers can now pass Prior Beliefs to experiments to help inform SigOpt about their expertise.

In Case You Missed It: Training, Tuning, and Metric Strategy If your business focuses on systematic trading, we’d like to share how you can most effectively re-train, adjust, and tune your deep learning models as you adapt to fluid market conditions.

In this post, we explore how Transformers enable developing high-performing NLP models under resource constraints.

Researchers Xavier Bouthillier of Inria and Gaël Varoquaux of Mila, surveyed the use of model experimentation methods across NeurIPS 2019 and ICLR 2020, two of the most prestigious international academic ML conferences.

SigOpt's Metric Constraints feature walkthrough: designing an accurate and efficient convolutional neural network.

Scalability is one of the main factors to consider when deciding whether an optimization solution will work for you and your organization.

Defining, selecting, and optimizing with the right set of metrics is critical to every modeling process, but these steps are often hard to execute well.

Defining just the right metrics to assess a model in the context of your business is no easy feat. Metric Constraints helps you establish guardrails for your optimization and experimentation loops.

Editor’s note: you can find Part 2 in this series here.

In this blog post, we discuss our recent collaboration trying to improve performance of a COVID-19 X-ray classification tool.

To boost any modeling projects that support these efforts, SigOpt is offering our solution for free to any researcher working on a COVID-19 related modeling project.

In Case You Missed It: Efficient Training and Tuning for Deep Learning Models Although much of the world is working from home (where and if possible) due to the COVID-19 pandemic, the markets are still online.

In Case You Missed It Modeling in Place A panel discussion with Alectio, NVIDIA, Pandora, and Yelp In today’s challenging sheltered environment, work from home is the new norm for many data scientists and engineers building and tuning models.

SigOpt works with algorithmic trading firms who represent over $600B in assets under management.

In Case You Missed It Recap for Tuning for Systematic Trading Talk 1: Intuition behind Bayesian optimization with and without multiple metrics Although much of the world is working from home (where and if possible) due to the COVID-19 pandemic, the markets are still—mostly—online.

At SigOpt, we are thrilled to collaborate with the outstanding community of experts from around the world.

The conference on Neural Information Processing Systems (NeurIPS) was held in December in Vancouver, Canada.

In this blog post, we discuss our recent progress applying multiobjective Bayesian optimization to minimizing reflectivity of glass.

Research on neural networks has become an international and fast-moving academic pursuit in the past several years.

SigOpt’s mission is to accelerate and amplify the impact of modelers everywhere.

Data science should never be a solo or individual sport.

This blog was originally published on MLconf.

SigOpt was proud to sponsor ICML for the 4th consecutive year, and we sent several team members to represent, both at the exhibitor booth and at the sessions.

Today, we are excited to announce the general availability of Metric Thresholds, a new feature that supercharges the performance of Multimetric optimization. Metric Thresholds helps you discover better models that meet your problem-specific needs by allowing you to define “thresholds” on your metrics.

At SigOpt, we provide a reliable, scalable service for enterprises and academics. In order to provide uptime that meets the demanding needs of our customers, we use AWS’ “Classic” Elastic Load Balancer (ELB) to distribute incoming requests among several AWS instances.

We collaborate with the University of Pittsburgh on fabricating nanostructured glass with ultrahigh transmittance, ultralow haze, and superomniphobicity.

SigOpt teams with the University of Pittsburgh for solar panel innovation.

The research team at SigOpt is fortunate to be part of an outstanding community of experts from around the world.

If you’re interested in more things computer vision, NeurIPS is coming up (we’ll be there- come say hi).

We are excited to release a new SigOpt feature into alpha testing: Training Monitor. This feature was designed to better empower our neural network developers.

t SigOpt is thrilled to congratulate our friend Roman Garnett for his recently being named an NSF CAREER award recipient.

Machine learning infrastructure tools can help bridge the gap between the modeler and the cluster by inserting an abstraction layer between the model builder and any infrastructure tools used to communicate with the cluster.

We are pleased to announce the addition of Projects, a feature that allows you and your team to organize experiments for easier access, viewing, and sharing.

Using SigOpt’s technology we have devised a method to generate high fidelity simulations of biological processes.

SigOpt, Inc. (“SigOpt”), a leading provider of solutions that maximize the performance of machine learning, deep learning and simulation models, announced a strategic partnership with Two Sigma, a leading systematic investment manager.

We are excited to announce improvements to our experiment list and search functionality in the Experiment Insights dashboard.

As part of our mission to make hyperparameter optimization accessible to modelers everywhere, we want to share our naming choices with other developers of HPO APIs.

In this latest Academic Interview, Brady Neal from the Mila (Quebec Artificial Intelligence Institute) discuss the paper, “A Modern Take on the Bias-Variance Tradeoff in Neural Networks” and its implications for deep learning practitioners.

The NeurIPS conference is held in Montreal this year, where leaders in the community will present innovative research over the course of tutorials, main conference proceedings and workshops.

Our research team at SigOpt has been very fortunate to be able to collaborate with outstanding researchers around the world, including through our internship program.

This post introduces the article A Nonstationary Designer Space-Time Kernel by Michael McCourt, Gregory Fasshauer, and David Kozak, appearing at the upcoming NeurIPS 2018 spatiotemporal modeling workshop.

This post introduces the article Automating Bayesian Optimization with Bayesian Optimization by Gustavo Malkomes and Roman Garnett, appearing in the upcoming NeurIPS 2018 proceedings.

This post introduces the article Efficient Nonmyopic Batch Active Search by Shali Jiang, Gustavo Malkomes, Matthew Abbott, Benjamin Moseley and Roman Garnett, appearing in the upcoming NeurIPS 2018 proceedings.

If you are an academic working on peer-reviewed research and need parameter tuning, we would love to help supercharge your research with SigOpt.

This post is an interview with the DeepDIVA team from the University of Fribourg to learn more about their framework and what it means for the broader community.

We are excited today to announce improvements in analyzing and viewing Multimetric experiments with the Experiment Insights dashboard.

This is the third of three blog posts during which we explore the concept of uncertainty - or noise - and its implications for Bayesian optimization.

This is the second of three blog posts during which we explore the concept of uncertainty - or noise - and its implications for Bayesian optimization.

This is the first of three blog posts during which we explore the concept of uncertainty - or noise - and its implications for Bayesian optimization.

Competitions like International Conference on Machine Learning help explore the complexities of developing effective ML pipelines under severe time restrictions and without expert intuition regarding the desired or expected output.

SigOpt is constantly working to extend the capabilities of our platform by supporting different types of optimization problems.

The research team at SigOpt works to provide the best Bayesian optimization platform for our customers. In our spare time, we also engage in research projects in a variety of other fields.

We’re happy to announce a strategic investment and technology development agreement with In-Q-Tel (IQT).

Happy New Year! 2017 was a great year for SigOpt.

AWS PrivateLink is a new solution that will enable SigOpt to connect directly with any AWS customer that has an Amazon Virtual Private Cloud (VPC).

This is the first edition of our new quarterly newsletter. In these updates, we will discuss newly released features, showcase content we have produced, published, or been cited in, and share interesting machine learning research that our research team has found. We hope you find these valuable and informative!

Today we announce the availability of SigOpt on AWS Marketplace. Now, with a single click, data scientists and researchers can access SigOpt’s powerful optimization-as-a-service platform designed to automatically fine-tune their machine learning (ML) and artificial intelligence (AI) workloads.

No one person is going to have all of the answers, so we're focused on making ourselves stronger as a team.

Congratulations to the other innovative startups that were honored this year!

Here at SigOpt, Gaussian processes and reproducing kernel Hilbert spaces (RKHS) are important components of our Bayesian optimization methodology.

Today, we’re honored to share that Gartner has listed us in its Cool Vendor 2017 report for AI Core Technologies.

In this blog post, we discuss another ingredient of Bayesian optimization: the sampling policy (or acquisition function).

We have revisited one of the traditional acquisition functions of Bayesian optimization, the upper confidence bound and looked to a slightly different interpretation to better inform optimization of it. In our article, we present a number of examples of this clustering-guided UCB method being applied to various optimization problems.

In this blog post, we are going to show solutions to some of the most common problems we’ve seen people run into when implementing hyperparameter optimization.

We are excited to announce SigOpt Organizations, the next step in the evolution of our web dashboard.

In this post, we will show you how Bayesian optimization was able to dramatically improve the performance of a reinforcement learning algorithm in an AI challenge.

In this post, we want to analyze a more complex situation in which the parameters of a given model produce a random output, and our multicriteria problem involves maximizing the mean while minimizing the variance of that random variable.

SigOpt’s API for hyperparameter optimization leaves us well-positioned to build exciting features for anyone who wants to perform Bayesian hyperparameter optimization in parallel.

In this post we will discuss the topic of multicriteria optimization, when you need to optimize a model for more than a single metric, and how to use SigOpt to solve these problems.

In this post we will show how to tune an MLlib collaborative filtering pipeline using Bayesian optimization via SigOpt. Code examples from this post can be found on our github repo.

Today is a big day at SigOpt. Since the seed round we secured last year, we’ve continued to build toward our mission to ‘optimize everything,’ and are now helping dozens of companies amplify their research and development and drive business results with our cloud Bayesian optimization platform.

In this post, we review the history of some of the tools implemented within SigOpt, and then we discuss the original solution to this black box optimization problem, known as full factorial experiments or grid search. Finally, we compare both naive and intelligent grid-based strategies to the latest advances in Bayesian optimization, delivered via the SigOpt platform.

In this post, we will detail how to use SigOpt with the Nervana Cloud and show results on how SigOpt and Nervana are able to reproduce, and beat, the state of the art performance in two papers.

SigOpt provides customers the opportunity to build better machine learning and financial models by providing users a path to efficiently maximizing key metrics which define their success. In this post we demonstrate the relevance of model tuning on a basic prediction strategy for investing in bond futures.

This blog post provides solutions for comparing different optimization strategies for any optimization problem, using hyperparameter tuning as the motivating example.

At SigOpt, our goal is to help our customers build better models, simulations, and processes by maximizing key metrics for them.

In this post on integrating SigOpt with machine learning frameworks, we will show you how to use SigOpt and TensorFlow to efficiently search for an optimal configuration of a convolutional neural network (CNN).

In this post on integrating SigOpt with machine learning frameworks, we will show you how to use SigOpt and XGBoost to efficiently optimize an unsupervised learning algorithm’s hyperparameters to increase performance on a classification task.

When building the new and improved v1 API, we stepped back and chose to more closely adhere to RESTful architectural principles.

Is it possible to use optimized machine learning models to beat Vegas? The short answer is yes; read on to find out how.

SigOpt allows experts to build the next great model and apply their domain expertise instead of searching in the dark for the best experiment to run next. With SigOpt you can conquer this tedious, but necessary, element of development and unleash your experts on designing better products with less trial and error.

The first post in our SigOpt in Depth series which deals with more technical aspects of topics that appear in the SigOpt Fundamentals series.

Using the best approximation gives our customers the fastest path to optimal behavior, which minimizes the costs of experimentation.

In this first post on integrating SigOpt with machine learning frameworks, we’ll show you how to use SigOpt and scikit-learn to train and tune a model for text sentiment classification in under 50 lines of Python.

SigOpt provides customers the opportunity to define uncertainty with their observations and using this knowledge we can balance observed results against their variance to make predictions and identify the true behavior behind the uncertainty.

Gaussian processes are powerful because they allow you to exploit previous observations about a system to make informed, and provably optimal, predictions about unobserved behavior. They do this by defining an expected relationship between all possible situations; this relationship is called the covariance and is the topic of this post.

Rescale gives users the dynamic computational resources to run their simulations and SigOpt provides tools to optimize them. By using Rescale’s platform and SigOpt’s tuning, efficient cloud simulation is easier than ever.

When we created SigOpt, our mission was simple: optimize everything. What do we mean by that?

At SigOpt, we can help you raise this metric automatically and optimally whether you are optimizing an A/B test, machine learning system, or physical experiment.

SigOpt provides a simple API and web interface for quickly and easily leveraging cutting-edge optimization research to solve this problem for you.

Research & Company Blog

Modeling Managed

Keeping track of it all: recording and organizing model training runs