There's a bit in the Metaflow docs that talks about choosing resources, like RAM: "as a good measure, don't request more resources than what your workflow actually needs. On the other hand, never optimize resources prematurely."
The problem is that for memory, too little means out-of-memory crashes, so the tendency I've seen is to over-provision memory, which ends up getting very expensive at scale.
This choice between "my process crashes" and "I am incentivized to make my process organizationally expensive" isn't ideal. Do you have any ways you deal with this at Netflix, or have you seen ways other Metaflow users deal with it?
I have some ideas on how this could be made better (some combination of being able to catch OOM situations deterministically, memory profiling, and sizing RAM by input size for repeating batch jobs), based in part on some tooling I've been working on for memory profiling: https://pythonspeed.com/fil, so would love to talk about it if you're interested.
While it is true that auto-sizing resources is hard and the easiest approach is to oversize @resources, the situation isn't as bad as it sounds:
1) In Metaflow, @resource requests are specific to a function/step, so you end up using resources only for a short while typically. It would be expensive to keep big boxes idling 24/7 but that's not necessary.
2) You can use spot instances to lower costs, sometimes dramatically.
3) It is pretty easy to see the actual resource consumption on any monitoring system, e.g. CloudWatch, so you can adjust manually if needed.
4) A core value proposition of Metaflow is to make both prototyping and production easy. While optimizing resource consumption may be important for large-scale production workloads, it is rarely the first concern when prototyping.
In practice at Netflix, we start with overprovisioning and then focus on optimizing only the workflows that mature to serious production and end up being too expensive if left unoptimized. It turns out that this is a small % of all workflows.
One way I deal with this problem in an alternative workflow manager (Nextflow), is by calculating the memory requirement for the ~95th percentile of a job, and submitting with a rule "If this crashes from going OOM, re-submit with memory*N" (up to some max number of retries/RAM). This lets most jobs sail through with a relatively low amount of RAM, and the bigger jobs end up taking a bit more time and resources.
The better your estimator function, of course, the tighter constraints you can use.
In many cases the memory usage is linear with input data, so you can come up with a function that predicts memory usage, add some padding, and then you don't need retries. E.g. this example here: https://pythonspeed.com/articles/estimating-memory-usage/
The Spark community (LinkedIn) developed Dr Elephant to profile jobs and provide suggestions for reducing memory/cpu consumption. Metaflow would need something similar:
I've seen that, yeah. I've already implemented a memory profiler for Python batch jobs (https://pythonspeed.com/fil), but starting to think about how to integrate it into specific pipeline frameworks.
It would be great if the infra layer can provide some help on automated resource scaling, especially for RAM. The ML solver/tooling layer has also been making progress on this front, for example Dask for limited-RAM pandas, h2o.ai has limited RAM solvers, xgboost has an external memory version, pytorch/tensorflow models are mostly trained on SGD and only needs to load data batch by batch. It's nice that Metaflow can integrate with any python code and thus benefit from all of the efforts made on the solver/tooling layer.
What is Metaflow's explicit support for Transfer Learning tasks? In other words, how do I know what models to use or not use? I am surmising from the techblog post that there is a stable set of content-intrinsic features, and that can be separated from perhaps more dynamic features-sets that characterize audiences, presentation treatment, and viewing (as conditioned on all the other stuff). But it sounds like there is a stable set of features for prediction tasks, as well, which is to say that for a task like predicting an audience for movie X in region Y, you'll need some set of features, and that we have some set of trained models (and recommend analytic components) available that match some or all of those features for this task. Is that a "thing", or is the workflow support simpler than that, and should that be a "thing"?
Good question! What you are asking is pretty much the core question for a certain set of ML tasks at Netflix.
Metaflow is rather unopinionated about those types of questions, since they are subject to active research and experimentation. Metaflow aims to make it easy to conduct the research and experiments but it is up to the data scientist to choose the right modeling approach, features etc.
In some cases, individual teams have built a thin layer of tooling on top of Metaflow to support specific problems they care about. I could imagine such a layer for specific instances of transfer learning, for instance.
In general, we are actively thinking if/how Metaflow could support feature sharing in general. It is a tough nut to crack.
Hi! How do you handle floating point determinism? Can some ML be reproduced in any architecture? Can you build the code with another compiler version? Can you use newer SIMD instructions?
Or you're forever tied to the initial hardware+compiler version?
Reproducibility is a spectrum. A good starting point is to snapshot the exact version of the code that produced a model. Even better, you should snapshot the full dependency graph, including transitive dependencies, of all (compiled) libraries, which Metaflow does with @conda. Together with data snapshots, this gives a pretty good foundation for reproducibility.
Depending on the libraries you use, the exact results may or may not be reproducible on other architectures. If cross-platform reproducibility is important to you, you should choose your libraries accordingly. Metaflow provides the tools for choosing the level of reproducibility that your application requires.
Hi Ville, thanks for coming on here to answer questions. I see that Metaflow has been made compatible with R now. Are there any plans to do the same with Julia?
Also I'm happy to answer any questions (I lead the Metaflow team at Netflix).