SigOpt partnered with MLconf on a webinar that focused on practical best practices for metrics, training, and hyperparameter optimization. During this discussion, we asked three questions of the audience, and the responses were particularly insightful for the question posed above: What makes modeling so hard? 

Different Problems During Different Phases of Development

It would be easy if there was a single problem that everyone pointed to as the most challenging. But this is not the case in model development. Rather than there being a consistent problem or two that is hardest for all modelers at all times, there are instead a variety of problems that become more or less intense depending on the modeling phase. In our poll, for example, 34% claimed that understanding model behavior was their biggest challenge, and another 29% identified tuning or selecting hyperparameters. The remaining 37% was split across defining and selecting metrics (18%) and tracking their work in the process (16%). There are a variety of challenges in model development that can crop up at different times and with different levels of intensity depending on circumstances (e.g., access to tooling) and phases (e.g., whether you are at the beginning or nearing the end of your project). 

Most Modelers Hack it Together

There has been an explosion of tools, software, and services offerings in machine learning to address these barriers to model development. But modelers are often still hacking it together themselves. In a second poll, 61% said they still hack together even basic tracking of training runs and tuning jobs. This seems to be the case even for modelers who are actively using one of those tools today. More than 90% who reported that they do use a tool today also reported that they still find themselves hacking things together. This suggests that tools that do exist aren’t getting the job done. 

Teams Tend to Be Centralized When Smaller, Distributed When Bigger

When modeling teams are smaller, they tend to be centralized so they can rapidly collaborate to apply machine learning to the product development process. And as modeling teams grow, they are increasingly distributed or embedded in departments to make a deeper business-line impact. In our poll, 100% of teams with fewer than 10 modelers were centralized, 50% of teams with 10 – 100 modelers were centralized, and 0% of teams with over 100 modelers were centralized. This suggests that the challenges modelers face may change as their teams grow. A centralized team likely has different barriers to modeling than a distributed one. 

If you’re interested in seeing the broader webinar context in which we gathered and discussed these results, watch the recording. If you want to try out the product, join our beta program for free access, execute a run to track your training, and launch an experiment to automate hyperparameter optimization.

Use SigOpt free. Sign up today.