Our research team at SigOpt has been working with many customers to improve how they define and optimize their experiments. They have seen a wide variety of experiments spanning multiple industries. Each experiment and each industry may have a completely different set of goals. An experiment can mean completely different things to different people. Mike McCourt, Head of Engineering at SigOpt, and Harvey Cheng, Head of Research at SigOpt, share their insights into defining and designing an experiment.
Insights from Head of Engineering
SigOpt’s Intelligent Experimentation platform helps you design better experiments. But what is an experiment? An experiment can be a sequence of observations, but that depends who you’re asking. For the Industrial Design community, an experiment could mean designing a sequence of steps. Depending on who you ask, you’re gonna get different responses. Some people might want to minimize variance in their experiments or even optimize some loss function; however, other people might be less interested in optimizing a function, and more interested in learning more about their problem space. They are interested in “what kinds of things are possible in this space?” So something that needs to be kept in mind is the design goal. What is your design goal? Do you want to learn or optimize? For example, Novelis works with scrap metal experiments while the University of Pittsburgh works with nanostructured surface experiments. Two completely different types of experiments. Scrap metal experiments might take 1 millisecond to run, while nanostructured surface experiments might take an hour to run. The objective might be trivial — something as simple as optimizing your L2 loss, but it’s those black box constraints that make it hard. Enforcing the constraints is the difficult part of an experiment. It takes a potentially straightforward problem in Linear Programming to something more complex where you don’t know what is going on under the hood of the Black Box Optimization. This is where SigOpt’s Intelligent Experimentation platform shines. Other hyper parameter optimization tools might be comparable in other areas, but SigOpt’s proprietary optimizer is what gives customers the ability to tackle complex problems.
It’s not just about finding the best answers. It’s also about finding out what’s possible. With SigOpt’s Intelligent Experimentation platform, you can get much closer to answering both of these questions. But even in simulation, you might not always have the right answer. So do you want to run your experiments in simulation or in the real world? It depends on your problem. It might make sense to run experiments in the real world, but is it too expensive and time consuming to do real world experiments? Is it worth the investment to run real world experiments, or would you rather start with simulation experiments which are cheaper, but might not give you the level of fidelity that you’re looking for.
The variety in the different types of experiments that you can design gives you the opportunity to go after new problems and answer new questions. SigOpt can even help HPC users lead experimentation in the manufacturing processes. SigOpt helps you get your models from testing and simulation into production.
Insights from Head of Research
SigOpt is an Intelligent Experimentation platform designed to accelerate and amplify the impact of modelers everywhere. Modelers can vary significantly based on their role, the problem that they’re solving, and the industry that they’re in. Are the modelers scientists? If so, they might have a completely different notion of what an experiment means to them. Scientists might be dealing with Material Science problems. They might want to know how to make silicon hardware smaller and smaller. Are the modelers physicists? They might be more interested in high fidelity simulations of the real world. Are the modelers data scientists? If so, their definition of an experiment might look like an A/B test. They might want to know which types of images or ads resonate best with their customers. On the other hand, a machine learning engineer might be more interested in detecting pedestrians for autonomous vehicles. And of course, quantitative traders might be more interested in knowing the best times to buy or sell stock.
However, the one thing that all of these different industries have in common is the alignment between model optimization and their core business goals. You might be using the intelligent experimentation platform for running experiments and tuning variables in order to optimize your model, but that is just one piece of the puzzle. The model you’re experimenting with might need to work with other models, or it might need to coordinate with other processes. It is part of a larger whole, and by merely optimizing the model is not enough to satisfy those core business goals. The model might enable you to make the quickest and smartest decisions, but are you reaching your core revenue or safety goals? The optimized model might be giving you the best results you’ve ever seen, but is there a disconnect between your model outcome and your core goals? What good is an optimized model that doesn’t deliver results in the real world? The Intelligent Experimentation platform can help you discover the root cause of this disconnect. Is the model overfitting the data? This could be a likely scenario where the model is fully optimized, but does not have a good sense of the rest of the world. Or is there something else going on? These are the factors that every modeler across any industry or role needs to keep in mind when designing their experiments. SigOpt’s Intelligent Experimentation platform helps you ask the right questions for your core goals and design experiments to get to the answers you need and successfully reach your goals.
To learn more about how experts approach experiment design, I encourage you to watch a panel discussion from the SigOpt Summit. If you’d like to start your own Intelligent Experimentation for your project, sign up to use SigOpt for free.