{"id":787,"date":"2019-04-25T19:40:25","date_gmt":"2019-04-25T19:40:25","guid":{"rendered":"http:\/\/www.smart-bricks.net\/?p=787"},"modified":"2019-04-28T22:23:11","modified_gmt":"2019-04-28T22:23:11","slug":"google-open-sources-gpipe-library-for-faster-training-of-large-deep-learning-models","status":"publish","type":"post","link":"https:\/\/www.smart-bricks.net\/index.php\/2019\/04\/25\/google-open-sources-gpipe-library-for-faster-training-of-large-deep-learning-models\/","title":{"rendered":"Google Open-Sources GPipe Library for Faster Training of Large Deep-Learning Models"},"content":{"rendered":"\n

Google AI<\/a> is open-sourcing GPipe<\/a>, a TensorFlow<\/a> library for accelerating the training of large deep-learning models.
\n
\nDeep-neural-networks (DNN) are the tool of choice for solving many AI \ntasks, such as natural-language processing and visual object detection. \nNew methods for the latter are often benchmarked against winners of the ImageNet challenge<\/a>.\n Each year’s winning entry has performed better than the last; however, \nthere is a corresponding increase in model complexity. The 2014 winner, GoogLeNet<\/a>, achieved 74.8% top-1 accuracy with 4 million model parameters. 2017’s winner, Squeeze-and-Excitation Networks<\/a>, reached 82.7% top-1 accuracy with 145.8 million parameters.
\n
\nThe increase in model size poses a problem when training the networks. \nIn order to train the networks in a reasonable time, much of the \ncomputation is delegated to accelerators<\/em>: special-purpose hardware such as GPUs or TPUs<\/a>.\n But these devices have limited memory, which restricts the size of the \nmodel that can be trained. There are ways to reduce the memory \nrequirements, such as swapping out the data in the accelerator’s memory,\n but these can slow down training. Another solution is to partition the \nmodel, so that multiple accelerators can be used in parallel. The most \nobvious partition scheme in a sequential DNN is to split the model by \nlayers and have each layer trained by a different accelerator. But the \nsequential nature of training multiple layers can result in only one \naccelerator working while others are idle waiting for results to come \nfrom higher or lower in the stack.<\/p>\n\n\n\n

GPipe solves this problem by splitting training batches into \n“micro-batches” and pipelining them through the layers. Accelerators for\n successive layers can begin processing a micro-batch result from a \nprevious layer without waiting for the full batch to be finished.<\/p>\n\n\n\n

Using GPipe and 8 TPUv2s, Google’s researchers were able to train \nvisual object detection models with 1.8 billion parameters: 5.6 times \nthe parameters that could be trained on a single TPUv2. Training these \nlarge models resulted in 84.7% top-1 accuracy on the ImageNet validation\n data, beating the 2017 winner’s score.<\/p>\n\n\n\n

In addition to supporting larger models, GPipe’s model partitioning \nallows for faster training of a given model simply by running more \naccelerators in parallel. Researchers reported that using “four times \nmore accelerators achieved 3.5 times speedup.”<\/p>\n\n\n\n

GPipe is available as part of the Lingvo framework<\/a> for building sequential neural-network models in TensforFlow.<\/p>\n","protected":false},"excerpt":{"rendered":"

Google AI is open-sourcing GPipe, a TensorFlow library for accelerating the training of large deep-learning models. Deep-neural-networks (DNN) are the tool of choice for solving many AI tasks, such as natural-language processing and visual object detection. New methods for the latter are often benchmarked against winners of the ImageNet challenge. Each year’s winning entry has… Read More »Google Open-Sources GPipe Library for Faster Training of Large Deep-Learning Models<\/span><\/a><\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":[],"categories":[1],"tags":[19,26,34],"_links":{"self":[{"href":"https:\/\/www.smart-bricks.net\/index.php\/wp-json\/wp\/v2\/posts\/787"}],"collection":[{"href":"https:\/\/www.smart-bricks.net\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.smart-bricks.net\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.smart-bricks.net\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/www.smart-bricks.net\/index.php\/wp-json\/wp\/v2\/comments?post=787"}],"version-history":[{"count":1,"href":"https:\/\/www.smart-bricks.net\/index.php\/wp-json\/wp\/v2\/posts\/787\/revisions"}],"predecessor-version":[{"id":788,"href":"https:\/\/www.smart-bricks.net\/index.php\/wp-json\/wp\/v2\/posts\/787\/revisions\/788"}],"wp:attachment":[{"href":"https:\/\/www.smart-bricks.net\/index.php\/wp-json\/wp\/v2\/media?parent=787"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.smart-bricks.net\/index.php\/wp-json\/wp\/v2\/categories?post=787"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.smart-bricks.net\/index.php\/wp-json\/wp\/v2\/tags?post=787"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}