TensorFlow

Internal

Hyunghun Cho
([email protected])

1
Overview

Dataflow-like model
Runs on a wide variety of different H/W platform

Source: github.com/zer0n/deepframeworks

Source: tensorflow.org 2
Basic concepts

Tensor
definition: an array with more than two axes
arbitrary dimensionality array
Directed graph describes T/F computation
node: instantiation of an Operation
0* 0*
Operation i node out
n
an abstract computation Source: T/F white paper
have attribute(s)
Kernel
particular implementation of an Operation
run on a type of device (e.g. CPU, GPU)
Variable
special Operation to persistent mutable Tensor
Session
Created to interact with T/F system

3
Programming Model

Example T/F code and corresponding computation graph

Single machine and distributed system architecture

Source: T/F white paper 4

Previous work

DistBelief
Downpour SGD
Sandblaster L-BFGS

Related to
Project Adam
MSR
Parameter Source: Large Scale Distributed Deep Networks

Server project

Source: parameter server architecture github wiki

Source: Project Adam paper

5
Feature Comparison

For detail, refer to Wikipedia

Tensor
Feature Theano Torch Caffe Chainer CNTK
Flow

Run on
O O O O O O
Single Machine

Run on
Distributed O X X X X O
Machines

Symbolic
O O X X O X
differentiation

Implemented
O X X O X X
by C++

Source: T/F white paper

6
Execution Mode

Single Device
Multi Device
Node placement
Cross-Device Communication
Distributed
Fault Tolerance
Error handling between Send-Receive node pair
Periodic health check to worker process

7
Programming Idioms

Programming Idioms
Data Parallel Training
sequential SGD
Model Parallel Training
Recurrent deep LSTM
Concurrent Steps

8
Code Metrics

Source
https://github.com/tensorflow/tensorflow

Code Summary
Total 114MB
3373 files including C/C++, python, HTML,

Top 5 languages for implementation

C++ and Python are the major languages

language files blank comment code

C++ 1092 46473 43399 276160

C/C++ Header 779 23457 44727 86274

Python 641 27622 46660 97570

Protocol Buffers 179 2217 7294 8724

Java 167 8296 17325 49374

C# 116 4285 8653 34347

Protocol Buffers: provide mechanism for serializing structured data

9
How it works

Python-C++ connection with SWIG wrapper

v v

[tensorflow.i] [py_func.i]

[py_func.h] [py_func.cc]

10
Code Structure

C++ implementation under /core folder

C/C++ Protocol
Folder C++
Header Buffers
./tensorflow/core/client/ 511 511

./tensorflow/core/common_runtime/ 1384 8526 9910

./tensorflow/core/common_runtime/g
644 3674 4318
pu/

./tensorflow/core/distributed_runtime/ 581 2579 3160

./tensorflow/core/distributed_runtime/
434 2759 3193
rpc/
./tensorflow/core/example/ 116 209 45 370
./tensorflow/core/framework/ 3539 14022 451 18012
./tensorflow/core/graph/ 952 5586 6538
./tensorflow/core/kernels/ 9180 42188 11 51379
./tensorflow/core/lib/core/ 573 1240 25 1838
./tensorflow/core/lib/gtl/ 1452 1943 3395

./tensorflow/core/lib/hash/ 36 400 436

./tensorflow/core/lib/histogram/ 60 324 384

./tensorflow/core/lib/io/ 340 2134 2474

./tensorflow/core/lib/jpeg/ 78 767 845

./tensorflow/core/lib/png/ 37 311 348

11
./tensorflow/core/lib/random/ 690 856 1546
C++ framework

Key classes

12
C++ kernels

Inherit from OpKernel

Kernel is implemented per CPU / GPU [How to]
GPU version uses CUDA library

[constant_op.h]
[constant_op.cc]

[constant_op_gpu.cu.cc] 13
Code Structure

Python implementation under /python folder

C/C++ Protocol Pytho

Folder C++
Header Buffers n
./tensorflow/python/ 168 168

./tensorflow/python/client/ 33 475 2031 2539

./tensorflow/python/framework/ 13 686 7097 7796

./tensorflow/python/kernel_tests/ 25391 25391

./tensorflow/python/lib/core/ 26 316 342

./tensorflow/python/lib/io/ 52 75 31 158
1499
./tensorflow/python/ops/ 14995
5
./tensorflow/python/platform/ 888 888
./tensorflow/python/platform/def
389 389
ault/
./tensorflow/python/summary/ 1168 1168
./tensorflow/python/summary/im
693 693
pl/
./tensorflow/python/tools/ 280 280

./tensorflow/python/training/ 6 7732 7738

14
./tensorflow/python/user_ops/ 7 7
Python Implementation

Operations

Trainings

15
Code Summary

The Python part

Various operations and trainings
API:
the most complete and the easiest to use
The C++ part
Framework and kernel functions
API:
offer some performance advantages
supports deployment to small devices such as Android

16
Meta Framework

Keras
TensorFlow Slim
a lightweight library for defining, training and evaluating models
Skflow
provide Scikit Learn style API
PrettyTensor
support a chainable object syntax to quickly define neural networks
TFLearn
a modular and transparent deep learning library

17