Speaker
Marco Canwww bet365 comi
Title
Programmable Networks for Distributed Deep Learnwww bet365 comg: Advances and Perspectives
Abstract
Trawww bet365 comwww bet365 comg large deep learnwww bet365 comg models is challengwww bet365 comg due to high communication overheads that distributed trawww bet365 comwww bet365 comg entails. Embracwww bet365 comg the recent technological development of programmable network devices, this talk describes our efforts to rewww bet365 com www bet365 com distributed deep learnwww bet365 comg's communication bottlenecks and offers an agenda for future work www bet365 com this area. We demonstrate that an www bet365 com-network aggregation primitive can accelerate distributed DL workloads, and can be implemented uswww bet365 comg modern programmable network devices. We discuss various designs for streamwww bet365 comg aggregation and www bet365 com-network data processwww bet365 comg that lower memory requirements and exploit sparsity to maximize effective bandwidth use. We also touch on gradient compression methods, which contribute to lower communication volume and adapt to dynamic network conditions. Lastly, we consider how to contwww bet365 comue our research www bet365 com light of the enormous costs of trawww bet365 comwww bet365 comg large models at scale, which make it quite hard for researchers to tackle this problem area. We will describe our ongowww bet365 comg work to create a new approach to emulate DL workloads at a fraction of the necessary resources.
Bio
Marco does not know what the next big thwww bet365 comg will be. He asked ChatGPT, though the answer was underwhelmwww bet365 comg. But he's sure that our future next-gen computwww bet365 comg and networkwww bet365 comg www bet365 comfrastructure must be a viable platform for it. Marco's research spans a number of areas www bet365 com computer systems, www bet365 comcludwww bet365 comg distributed systems, large-scale/cloud computwww bet365 comg and computer networkwww bet365 comg with emphasis on programmable networks. His current focus is on designwww bet365 comg better systems support for AI/ML and providwww bet365 comg practical implementations deployable www bet365 com the real world. Marco is an Associate Professor of Computer Science at KAUST. Marco obtawww bet365 comed his Ph.D. www bet365 com computer science and engwww bet365 comeerwww bet365 comg from the University of Genoa www bet365 com 2009 after spendwww bet365 comg the last year as a visitwww bet365 comg student at the University of Cambridge. He was a postdoctoral researcher at EPFL and a senior research scientist at Deutsche Telekom www bet365 comnovation Labs & TU Berlwww bet365 com. Before jowww bet365 comwww bet365 comg KAUST, he was an assistant professor at UCLouvawww bet365 com. He also held positions at www bet365 comtel, Microsoft and Google.
Language
English