Parallel SGD scales training out by exploding the batch size and synchronising every worker at every step and specialised networks only postpone the problem. KungFu, built with the Large Scale Data & Systems Group at Imperial College London and published at OSDI ‘20, lets the user declare how workers synchronise, monitor gradient and network statistics cheaply and change synchronisation strategy or parallelism at runtime. It started as my Master’s thesis, recognised as a Distinguished Project and was presented at the SOSP ‘19 AI Systems Workshop.
KungFu: Adaptive Distributed Deep Learning
Nov 20192 posts
-
Master's Thesis: A Distributed ML Framework with Flexible Synchronisation
The thesis behind KungFu. Why Parallel SGD forces large batches on large clusters and a system that lets the user declare, monitor and adapt how workers synchronise.
Read -
KungFu: Adaptive Deep Learning at Symposium of Operating Systems Principles 2019 & My Learning Experience at SOSP'19
Monitor training statistics, change worker synchronisation on the fly or even dynamically scale training cluster poster at AI Systems Workshop, Symposium of Operating Systems Principles, Deerhurst Resort, Huntsville, Ontario, Canada
Read