In this discussion, Scott Clark, SigOpt Co-founder and General Manager, reviews the Intelligent Experiment framework at the Ai4 Webinar. During this talk, Scott explains that tools that enable traditional experimentation – such as experiment tracking – are often good at telling you what you’ve done and, sometimes, what is working. But they aren’t very good at telling you what to do next. Intelligent experimentation tooling is designed to add this component to your workflow.

Here are a few highlights from the talk and discussion:

• SigOpt enables developers to ask the right questions during the modeling process
• Intelligent Experiment is about bolting on top of your current flow and make better decisions with your experiments
• SigOpt works in any domain. It is a tried and trusted technology that is used by the most respected modelers in the world in both academic and applied settings

For a more more specific summary of the presentation, click through to view any segment you missed:

1:11 – Start of Scott’s Presentation and Intros

1:43 – Scott has worked on tuning models than a decade and explored. A key insight is the same issues he experienced as a grad student were the same problems at the companies he worked at.

5:30 – Scott reviews where SigOpt adds value in the larger machine learning journey. Producing a machine learning model is complex and multi-scale problem with several different factors that determine success. SigOpt adds value in the experimentation stage when developers are trying different topologies, models, data augmentation techniques, and tuning hyperparameters. This is an iterative process. Modeling is a scientific process and requires experimentation to get to a specific result.

8:34 – SigOpts value is all about making models more efficient, more productive, generate more insights, and deploy faster into production. This flywheel of experimentation helps you generate a higher ROI inside your organization.

8:51 – This how SigOpt fits in with the rest of the Intel AI software stack. Designed to give you better results on AI workloads and make it easier to deploy AI workloads 

10:13 – Key customers who are using SigOpt include OpenAI, Two Sigma, and the University of Fribourg

 10:37 – Example of the flow you use when using SigOpt to optimize your experiment flow. Intelligent Experiment is about bolting on top of your current flow and make better decisions with your experiments

12:15 – SigOpt works in any domain. It is a tried and trusted technology that is used by the most respected modelers in the world in both academic and applied settings.

13:20 – Intelligent Experimentation defined. It is the act of taking all the tasks included in experimentation and giving order. Intelligent Experimentation is a means to help augment the decision maker by allowing them design, explore, and optimize to make the most out of their experiments.

15:10 – Design, Explore, and Optimize is framework to help you run intelligent experiments. Design the art of asking the right questions. Exploring is seeing how all the metrics and methods interact allowing to understand your model more deeply. Optimizing is about obtaining the result as quickly and efficiently as possible

17:14 – Design. Scott provided an overview of various techniques that are critical in the workflow, such as defining and selecting a wide variety of metrics for a more complete perspective on model performance or selecting the right type of data for each particular step in the training process (such as training data versus validation data). But design is also about having the tools that enable you to phrase your questions in the form of insightful experiments. SigOpt enables you to set metric constraints, parameter constraints, prior knowledge, and metric strategy in a way that allows for more precise design of experiments so you can learn more, faster than is otherwise possible.

19:37 – With a few lines of code, all the key information can be tacked and stored so the individual design choices can used to highlight tradeoffs.

20:09 – Explore. To have confidence in a model requires understanding it broadly and deeply. You should explore many metrics and a variety of architectures in combination to get a better grasp on your modeling problem. SigOpt – and especially the SigOpt Dashboard – are unique in that they are designed to give you better insights the more training runs and hyperparameter optimization experiments you manage with the SigOpt API. Take metrics as an example. SigOpt enables you to track up to 50 metrics, phrase certain metrics as constraints and optimize multiple. In his talk, Scott discussed a variety of examples of metrics that you may need to include in your experimentation, including training metrics, validation metrics, guardrail metrics and production metrics.

25:51 – Optimize. Finally, optimizing your models ensure your modeling problem is efficient, and fast giving you confidence to put these models in production or publish them as part of your research. In terms of techniques, Scott explained that it is important to consider whether you optimize a single metric or balance multiple in the optimization process. He also explained that it is also important to consider which optimization method you select for your model. Bayesian optimization is a good choice for its sample efficiency. Grid search is a good choice for the certainty it brings to models with few parameters to optimize. Similarly, the parallel bandwidth you want to run on is also an important consideration. SigOpt enables you to easily manage this entire optimization process with a few lines of code. You can use any optimization method, including SigOpt’s proprietary optimizer, and automatically schedule your jobs across your compute resources. There are also more than a dozen advanced optimization features that enable you to make the optimization process fit your specific modeling needs. 

30:32 – Recap of the Design, Explore, Optimize framework and how SigOpt adds value in these three areas.

31:16 – Examples of how Numenta, PayPal, and Two Sigma have applied SigOpt using the Design, Explore, Optmize framework to achieve better results for their companies.

34:12 – Demo of how SigOpt can easily provide you value with a few lines of code

40:24 – Sign up for SigOpt for free at SigOpt.com/signup

41:02 – Come join SigOpt’s first summit to learn directly from customers and developers

42:00 – Questions from the audience