! S 11: Distributed Deep Learning

Question 1

Deep learning frameworks

Answer 1

• software libraries that provide tools for development, training, and deployment of deep learning models
• e.g. Tensorflow, Py Troch
• focus on neural networks (basis for deep learning)

Question 2

Deep Learning Frameworks - Steps

Answer 2

• Build Computational Graph from network definition
• Input training data & compute loss function
• Update parameters

Question 3

Deep Learning Frameworks - Model Training

Answer 3

• Define-and-run = complete 1. (build graph) before others e.g. TensoFlow, Caffe
• Define-by-run = combine 1. & 2. (input training data & compute loss function) e.g. PyTorch

Question 4

ONNX

Answer 4

• Open NN eXchange
• Opn-source shared model representation for framework interoperability & shared optimization

Question 5

Distributed ML

Answer 5

• using multiple computer ressources that are distributed across a network / machines
• leverage power of parallel computing
• better training, handle larger data sets, more complex deep learning tasks

Question 6

Ways of distributed ML

Answer 6

1. Data Parallelization
2. Model Parallelization

Question 7

Data Parallelization

Answer 7

• each machine receives copy of model & operates on batch of data
• Each worker: training on non-overlapping batch -> update of paramaters
• Requirment: parameter syncrhonization

Question 8

Model Parallelization

Answer 8

• each machine processes differen portion of model
• Forward pass: layer computs output signal -> workers that hold next layer

Question 9

Tensorflow

Answer 9

• offers tools, libraries & open-source platform for ML created by Google
• tensor = multidimensional array (like numpy with GPU support)
• Main feature: Express numeric computation as graph
• graph can be exported to train & run at different playes