Unit 5

UNIT-5: TENSORFLOW
1. FEATURES OF TENSORFLOW

TensorFlow is an open-source machine learning framework developed by Google, widely used

for building and training machine learning models, especially deep learning models. Its key features
include:

• Flexibility: TensorFlow supports a broad range of tasks, from simple linear regression to
complex deep neural networks, making it suitable for both research and production.
• Scalability: It can execute computations on various platforms, including multiple CPUs,
GPUs, mobile devices, and embedded systems, enabling efficient large-scale processing.
• Ecosystem: TensorFlow offers a rich set of tools and libraries:
o TensorBoard: Visualization tool for model debugging and performance analysis.
o TensorFlow Lite: Lightweight version for mobile and embedded devices.
o TensorFlow.js: Enables machine learning in web browsers.
• Community Support: As an open-source project, it benefits from a large, active community
that contributes to its development, documentation, and troubleshooting.
• Integration: Seamlessly integrates with popular Python libraries like NumPy, Pandas, and
Keras (now a core part of TensorFlow), enhancing its usability in data science workflows.

Why is it called TensorFlow?

• A tensor is just a fancy word for data in any shape — a number, a list, a table, an image
(everything can be seen as a tensor).
• Flow means moving that data through different steps (mathematical operations) to get a result.

2. TENSOR DATA STRUCTURE

In the world of deep learning and artificial intelligence, tensors are the fundamental
units of data representation.
Whether you are dealing with images, sounds, videos, or text, in the background,
everything is represented and manipulated as tensors.
Thus, understanding how to create, modify, transform, and operate on tensors is
an essential skill for any aspiring machine learning practitioner.

1. WHAT EXACTLY IS A TENSOR?

A tensor is a mathematical object that can be thought of as a generalized form of a

matrix.
While a matrix is a two-dimensional array of numbers (with rows and columns), a
tensor can extend into more than two dimensions — even into three, four, or
higher.

1 By kuber kr jha
 Unit 5
In simple words, a tensor is just a multi-dimensional container that stores
elements of the same data type (for example, integers or floating-point numbers).

Depending on the number of dimensions, tensors are categorized as follows:

• A single number is called a scalar (0D tensor).

• A list of numbers is called a vector (1D tensor).
• A table of numbers is called a matrix (2D tensor).
• A cube of numbers (like stacked matrices) is a 3D tensor, and so on.

Thus, tensors are a generalization of scalars, vectors, and matrices.

2. IMPORTANT PROPERTIES OF TENSORS

Every tensor has certain important properties that define its structure and behavior:

(A) RANK

The rank of a tensor refers to the number of dimensions it possesses.

For instance:

• A scalar has rank 0.

• A vector has rank 1.
• A matrix has rank 2.
• A cube-like structure has rank 3.

Thus, the higher the rank, the more complex the tensor structure becomes.

(B) SHAPE

The shape of a tensor describes the size of the tensor along each of its
dimensions.
For example, a tensor with shape (2, 3) has two rows and three columns.

2. By kuber kr jha
 Unit 5
Shape is important because many operations (like addition or multiplication) require
tensors to have compatible shapes.

(C) SIZE

The size of a tensor is the total number of elements stored inside it.
It is calculated by multiplying the numbers in the shape.
For example, a tensor of shape (3, 4) contains 3 × 4 = 12 elements.

(D) DATA TYPE (DTYPE)

Each tensor has a data type, called dtype, which specifies the kind of data it holds.
Common data types include integers (int32), floating-point numbers (float32), booleans,
or even strings.
Handling dtype correctly is critical because mismatched data types can cause errors
during operations

3. CREATING TENSORS

Tensors can be created in several ways, depending on the nature of your data and your
application requirements.

You can create tensors by:

• Converting Python lists or NumPy arrays into tensors using libraries like
TensorFlow.
• Using built-in functions to create tensors filled with zeros, ones, random
values, or sequences.

3. By kuber kr jha
 Unit 5
Example in TensorFlow:

import tensorflow as tf
# From list
tensor1 = tf.constant([1, 2, 3])
# From nested list (matrix)
tensor2 = tf.constant([[1, 2, 3], [4, 5, 6]])
# Tensor of all zeros
tensor_zeros = tf.zeros([2, 3])
# Tensor of all ones
tensor_ones = tf.ones([3, 3])
# Random tensor
tensor_random = tf.random.uniform([2, 2], minval=0, maxval=1)

These methods make it extremely easy to generate data structures required for
machine learning models.

4. TENSOR MANIPULATION TECHNIQUES

Handling tensors effectively means being able to modify their shape, access their
elements, and combine them in different ways.
Let's explore the main manipulation techniques in detail:

(A) RESHAPING TENSORS

Reshaping allows you to change the organization of elements without changing

the actual data inside the tensor.
For example, you can convert a flat 1D tensor into a matrix.

tensor = tf.constant([1, 2, 3, 4, 5, 6])

reshaped = tf.reshape(tensor, [2, 3]) # Shape becomes (2,3)

Reshaping is critical when preparing data batches for training deep learning models.

4. By kuber kr jha
 Unit 5
(B) SLICING AND INDEXING TENSORS

Just like you slice lists in Python, you can slice tensors to access specific elements,
rows, columns, or blocks.

Example:

matrix = tf.constant([[1, 2, 3], [4, 5, 6]])

# Access the first row
first_row = matrix[0]
# Access element at row 1, column 2
element = matrix[1, 2]

Slicing is crucial for processing parts of data during training.

(C) EXPANDING AND REDUCING DIMENSIONS

Sometimes, you need to add or remove dimensions to make tensor shapes compatible
for operations.

• Expand dimensions adds a new axis of size 1:

expanded = tf.expand_dims(tensor, axis=0) # Add a new first dimension

• Squeeze dimensions removes axes with size 1:

squeezed = tf.squeeze(expanded) # Remove unnecessary dimension

These operations are especially important in deep learning models, where input shapes
must match exactly.

(D) TRANSPOSING TENSORS

Transposing refers to swapping the axes of a tensor.

For 2D tensors, it is similar to flipping rows into columns and vice-versa.

transposed = tf.transpose(matrix)

Transposing is essential in operations like matrix multiplication.

(E) CONCATENATING AND SPLITTING TENSORS

You can combine tensors or split one tensor into multiple smaller ones.

5. By kuber kr jha
 Unit 5
• Concatenation merges tensors along a specified axis:
combined = tf.concat([tensor_a, tensor_b], axis=0)

• Splitting divides a tensor into parts:

splitted = tf.split(matrix, num_or_size_splits=2, axis=0)

These operations are vital for handling batches and parallel computations.

5. MATHEMATICAL OPERATIONS WITH TENSORS

Tensors can undergo all standard mathematical operations, performed element-wise

unless stated otherwise.

Examples include:

# Element-wise addition
result = tf.add(tensor1, tensor1)

# Element-wise multiplication
result = tf.multiply(tensor1, tensor1)

# Matrix multiplication
matmul_result = tf.matmul(tensor2, tf.transpose(tensor2))

Learning these operations is crucial because deep learning heavily relies on matrix and
tensor mathematics.

6. BROADCASTING IN TENSOR OPERATIONS

Broadcasting is a powerful feature that automatically expands smaller tensors to

match the dimensions of larger tensors during operations.

For instance, adding a tensor of shape (3,) to a tensor of shape (2, 3) is possible because
the smaller tensor is broadcasted (copied across rows) automatically.

small = tf.constant([1, 2, 3])

large = tf.constant([[4, 5, 6], [7, 8, 9]])
broadcasted_add = large + small

Understanding broadcasting avoids the need for manual dimension adjustments.

6. By kuber kr jha
 Unit 5
7. DATA TYPE CASTING

At times, you might need to change the data type of a tensor, such as converting an
integer tensor to a floating-point tensor.

float_tensor = tf.cast(tensor1, dtype=tf.float32)

This is useful when different operations or neural networks expect tensors of specific
types.

8. DEVICE PLACEMENT AND TENSOR STORAGE (ADVANCED)

In TensorFlow, tensors can automatically be placed on available hardware — either a

CPU or a GPU — to optimize performance.

Manual placement is possible:

with tf.device('/GPU:0'):
tensor_on_gpu = tf.constant([1.0, 2.0, 3.0])

Efficient device management becomes important for large models and faster
computations.

4. TENSORBOARD VISUALIZATION
TensorBoard is a powerful tool that provides visualizations and insights into your
machine learning models.
It helps you monitor metrics like loss and accuracy, visualize the model graph, examine
histograms, analyze images, and much more.

Key features include:

• Scalars: Plot graphs for loss, accuracy, learning rate over time.
• Histograms: Track how weights and biases change during training.
• Graphs: Visualize the entire structure of the computation graph.
• Images, Audio, Text: Visualize model inputs and outputs.
• Projector: Visualize high-dimensional data like embeddings.

Typical workflow:

1. During training, log data using tf.summary.

7. By kuber kr jha
 Unit 5
2. Run a TensorBoard server pointing to the log directory.
3. Open the TensorBoard dashboard in a browser.

Example:

import tensorflow as tf
from tensorflow import summary

log_dir = "logs/"
writer = tf.summary.create_file_writer(log_dir)

@tf.function
def train_step(x, y):
with writer.as_default():
tf.summary.scalar('loss', 0.5, step=1)

Launch TensorBoard via:

tensorboard --logdir=logs/

8. By kuber kr jha
 Unit 5
5. TENSORS, VARIABLES, AND AUTOMATIC
DIFFERENTIATION
(A) WHAT ARE TENSORS?

At the heart of TensorFlow (and many other deep learning libraries) lies the concept of
a Tensor.

A Tensor is simply a multi-dimensional array or list of numbers.

It is the basic unit of data in TensorFlow.

You can think of tensors in a hierarchy of dimensions:

Tensor Type Shape Example

Scalar (0D Tensor) No dimensions 5

Vector (1D Tensor) 1 dimension [1, 2, 3]

Matrix (2D Tensor) 2 dimensions [[1, 2], [3, 4]]

3D Tensor 3 dimensions [[[1], [2]], [[3], [4]]]

In everyday deep learning tasks:

• Input data (like images) are stored as tensors.

• Weights of a neural network are tensors.
• Predictions made by a model are tensors.

A simple code example:

import tensorflow as tf

# Scalar Tensor
scalar = tf.constant(42)

# Vector Tensor
vector = tf.constant([1, 2, 3])

9. By kuber kr jha
 Unit 5
# Matrix Tensor
matrix = tf.constant([[1, 2], [3, 4]])

# 3D Tensor
tensor_3d = tf.constant([[[1], [2]], [[3], [4]]])

Each tensor has:

• Shape: Size across dimensions (rows, columns, depth, etc.)

• Rank: Number of dimensions (0 for scalar, 1 for vector, etc.)
• Dtype: Data type (e.g., float32, int32)

(B) WHAT ARE VARIABLES?

While Tensors are usually immutable (cannot change once created),

Variables in TensorFlow are special mutable tensors whose values can change
during computation.

This property makes Variables extremely important for machine learning

models, because:

• During training, the model's parameters (weights and biases) are constantly
updated to minimize error.
• These parameters must be stored in a way that supports updating — hence,
Variables.

Creating a Variable:

# Create a variable
my_var = tf.Variable([[1, 2], [3, 4]])

# Assign a new value

my_var.assign([[5, 6], [7, 8]])

Key features of Variables:

• Mutable: Can be updated using .assign(), .assign_add(), .assign_sub().

• Trainable: They are automatically tracked by TensorFlow’s optimization tools
(like gradient descent).

In contrast, constants are fixed and cannot be reassigned after creation.

10. By kuber kr
jha
 Unit 5
(C) AUTOMATIC DIFFERENTIATION
WHY DO WE NEED DIFFERENTIATION IN MACHINE LEARNING?

When we train a model, we aim to minimize a loss function (a measure of how

wrong the model is).
To do this, we need to know:

• "In which direction should we change our weights?"

• "By how much should we change them?"

This is where derivatives (gradients) come in:

• A derivative tells us the rate of change of a function.

• In machine learning, it tells us how much a small change in a weight will change
the loss.

Automatic differentiation is TensorFlow’s built-in tool that:

• Automatically computes these derivatives for you.

• Accurately and efficiently calculates gradients across complex chains of
mathematical operations.

🌟 HOW DOES TENSORFLOW DO IT?

TensorFlow uses an engine called Gradient Tape (tf.GradientTape).

You "record" computations under a tape,

and TensorFlow automatically tracks all operations applied to Variables.

Then you can ask it:

"Tell me the gradient of the final output with respect to each input!"

Example:

# Create a variable
x = tf.Variable(3.0)

# Start recording
with tf.GradientTape() as tape:
y = x ** 2 + 5 * x + 3 # Some computation

# Calculate gradient of y with respect to x

11. By kuber kr
jha
 Unit 5
dy_dx = tape.gradient(y, x)
print(dy_dx.numpy()) # Output: 2*x + 5 = 2*3 + 5 = 11

Explanation:

• We created a function y = x² + 5x + 3.
• TensorFlow automatically differentiates it and gives the slope (11) at x=3.

Without automatic differentiation, computing gradients for complex neural networks

would be extremely tedious and error-prone.

6. GRAPHS AND TF.FUNCTION

(A) WHAT ARE GRAPHS IN TENSORFLOW?

To truly understand TensorFlow, you must understand Computation Graphs.

A computation graph is a way of visualizing and organizing mathematical

computations.
Instead of executing operations immediately (as happens in regular Python),
TensorFlow builds a graph of the entire computation first, and then executes it.

✏ IMAGINE THIS:

Suppose you write:

z = (x + y) * (x - y)

TensorFlow doesn't immediately calculate the result. Instead, it creates a graph like
this:

x ----+
|--> + -->
y ----+ \
* --> z
x ----+ /
|--> - -->
y ----+

Each node represents an operation (e.g., +, -, *)

Each edge represents a tensor (data) flowing between operations.

12. By kuber kr
jha
 Unit 5
Key ideas:

• Nodes = Operations (Add, Multiply, etc.)

• Edges = Tensors (data flowing between ops)

Thus, the whole model is built as a graph structure, not just line-by-line code
execution.

WHY USE GRAPHS?

Advantages of computation graphs:

Benefit Explanation
Optimization TensorFlow can rearrange the graph to improve performance (e.g.,
parallelize operations).
Portability Graphs can run on different hardware (CPU, GPU, TPU) without
changing code.
Serialization You can save the graph and reuse it later (deploy it to production
easily).
Efficiency Graph execution can be faster because TensorFlow understands the
"whole picture" and can optimize memory, compute usage.

13. By kuber kr
jha
 Unit 5
In short, graphs allow TensorFlow to be smart and efficient behind the scenes.

(B) WHAT IS tf.function?

Now comes the most powerful tool: tf.function.

In default mode, TensorFlow uses Eager Execution — operations are executed

immediately.

Example:

x = tf.constant(2)
y = tf.constant(3)
print(x + y) # Output is immediately calculated as 5

But eager execution is slower because TensorFlow cannot optimize the sequence of
operations.

✨ TF.FUNCTION: MAKING CODE FASTER

tf.function converts a regular Python function into a highly optimized TensorFlow

graph.

• You write normal Python code.

• TensorFlow records the operations inside a graph.
• Then it compiles and optimizes them.
• Execution becomes much faster (almost like C++ performance).

Usage:

@tf.function
def my_function(x, y):
return x * x + y

When you call my_function(3, 2), TensorFlow:

• Records that it needs to square x and add y.

• Optimizes the entire computation.
• Executes it very quickly.

14. By kuber kr
jha
 Unit 5
(C) UNDER THE HOOD OF TF.FUNCTION

When you use @tf.function, several things happen:

1. TensorFlow traces the Python function, meaning it watches what operations you
are using.
2. It builds a computation graph from those operations.
3. It optimizes the graph (for speed, memory usage).
4. It executes the graph in highly efficient TensorFlow runtime (not slow Python).

Thus, tf.function acts like a bridge:

Taking easy-to-write Python code → Building a superfast

computation graph → Running it optimally.

(D) WHEN AND WHY USE TF.FUNCTION?

You should use tf.function when:

• You have functions that are called many times, especially inside training
loops.
• You want to improve speed significantly.
• You are building models that will be deployed in production.

Especially during model training, using tf.function can boost performance

drastically compared to Eager Execution.

(E) EXAMPLE: WITHOUT AND WITH TF.FUNCTION

Without tf.function (Eager mode)

def simple_add(x, y):

return x + y

print(simple_add(tf.constant(2), tf.constant(3)))

This is slow because every operation runs immediately.

With tf.function (Graph mode)

@tf.function
def simple_add(x, y):
return x + y

15. By kuber kr
jha
 Unit 5
print(simple_add(tf.constant(2), tf.constant(3)))

Now TensorFlow creates and optimizes a graph internally, leading to faster execution.

7. MODULES, LAYERS, AND MODELS IN

TENSORFLOW
(A) MODULES IN TENSORFLOW

A module in TensorFlow refers to a reusable component that encapsulates trainable

variables (like weights) and the computation performed on them.

WHY MODULES ARE IMPORTANT

In deep learning, a model is typically composed of several layers, each responsible for
performing certain operations like activation, transformation, or regularization. Each
of these layers needs to manage its own weights and computations, which is where
modules come into play.

KEY FEATURES OF MODULES:

• Encapsulate Variables: A module stores variables such as weights or biases

in a model.
• Reusability: A module can be reused multiple times in different parts of a
model or across different models.
• Composability: A module can be composed of other modules, allowing for
complex hierarchical structures.
CREATING A SIMPLE MODULE IN TENSORFLOW

In TensorFlow, the base class tf.Module can be used to define custom modules:

import tensorflow as tf

class SimpleModule(tf.Module):
def __init__(self, name=None):
super().__init__(name=name)
self.weight = tf.Variable([[1.0, 2.0], [3.0, 4.0]]) # Trainable variable

def __call__(self, x):

return tf.matmul(x, self.weight) # Simple computation

16. By kuber kr
jha
 Unit 5
• The SimpleModule has a trainable variable weight, and the computation is defined
in the __call__ method.
• You can call this module just like a function, passing inputs to compute the result.
(B) LAYERS IN TENSORFLOW

In deep learning, layers are fundamental components that perform specific

computations and transformations on data. Layers are organized in a neural network to
progressively learn more abstract features from raw data (like an image, text, or audio).

WHY LAYERS MATTER

A layer in a neural network typically has the following characteristics:

• Weights and Biases: Layers hold trainable parameters (weights and

biases) that adjust during training.
• Activation Function: Most layers use an activation function (e.g., ReLU,
Sigmoid) to introduce non-linearity, which helps the network learn complex
patterns.
• Forward Pass: Layers define how inputs are transformed into outputs.

TYPES OF LAYERS

Some commonly used layers in TensorFlow:

1. Dense Layer (Fully Connected): Every input is connected to every output

unit.
o Example: tf.keras.layers.Dense(units=64, activation='relu')
2. Convolutional Layer (Conv2D): Used for image data to extract features like
edges or textures.
o Example: tf.keras.layers.Conv2D(filters=32, kernel_size=(3,3), activation='relu')
3. Recurrent Layer (LSTM/GRU): Used for sequential data like time series or
text.
o Example: tf.keras.layers.LSTM(128)
4. Dropout Layer: Prevents overfitting by randomly setting a fraction of input
units to zero during training.
o Example: tf.keras.layers.Dropout(0.5)
5. BatchNormalization Layer: Normalizes the input to a layer, helping the
model train faster.
o Example: tf.keras.layers.BatchNormalization()

17. By kuber kr
jha
 Unit 5
HOW LAYERS WORK TOGETHER IN A MODEL

A neural network is typically built by stacking these layers. Each layer transforms its
input and passes it to the next layer.

Example of a simple feedforward neural network:

model = tf.keras.Sequential([
tf.keras.layers.Dense(128, activation='relu', input_shape=(784,)),
tf.keras.layers.Dense(10, activation='softmax')
])

• The first layer is a Dense layer with 128 units and ReLU activation.
• The second layer is another Dense layer with 10 units, typically used for
classification (10 classes, hence softmax activation).

Each layer has its trainable weights and will adjust them during training via
backpropagation.

(C) MODELS IN TENSORFLOW

A model in TensorFlow refers to the overall architecture that defines how input data
flows through the network and how it produces output. A model is composed of layers
and handles the training process.

WHY MODELS ARE CRUCIAL

In TensorFlow, models are responsible for:

1. Forward Propagation: Passing inputs through the layers of the network to

produce predictions.
2. Training: Updating weights using a chosen optimizer to minimize a loss
function.
3. Evaluation: Assessing the model’s performance on unseen data.

MODEL TYPES IN TENSORFLOW

TensorFlow provides several ways to build models, but the most common approach is
using the Keras API, which simplifies the process.

1. Sequential Model: A linear stack of layers. Each layer has one input and one
output.
18. By kuber kr
jha
 Unit 5
Example: tf.keras.Sequential([...])
o
2. Functional API: More flexible and allows for multiple inputs and outputs,
shared layers, and non-linear architectures.
o Example:

inputs = tf.keras.Input(shape=(784,))
x = tf.keras.layers.Dense(64, activation='relu')(inputs)
outputs = tf.keras.layers.Dense(10, activation='softmax')(x)
model = tf.keras.Model(inputs=inputs, outputs=outputs)

This approach allows for more complex architectures such as multi-input

models, multi-output models, or models with shared layers.

BUILDING A MODEL IN TENSORFLOW

You can build models using either the Sequential API or Functional API, but the
Keras Model class is the base class for all models.

Sequential Model Example:

model = tf.keras.Sequential([
tf.keras.layers.Dense(64, activation='relu', input_shape=(784,)),
tf.keras.layers.Dense(10, activation='softmax')
])

Here, the model consists of:

• A Dense layer with 64 units and ReLU activation.

• A Dense output layer with 10 units (e.g., for a classification task with 10
classes).

(D) TRAINING A MODEL

Once you've built a model, it’s time to train it. Training a model involves the following
steps:

1. Forward Pass: Feed the data into the model to compute predictions.
2. Loss Calculation: Calculate how far off the predictions are from the true labels.
3. Backpropagation: Compute gradients using automatic differentiation.
4. Weight Update: Apply the gradients to update the model’s weights.

19. By kuber kr
jha
 Unit 5
TRAINING LOOP

TensorFlow's model.fit() handles the training loop for you:

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

model.fit(train_data, train_labels, epochs=10)

Here:

• optimizer='adam': Uses the Adam optimizer to minimize the loss.

• loss='categorical_crossentropy' : Suitable for multi-class classification problems.
• metrics=['accuracy']: Tracks accuracy during training.

(E) EVALUATION AND PREDICTION

Once the model is trained, it can be evaluated using evaluation functions or used for
making predictions.

EVALUATING THE MODEL

test_loss, test_accuracy = model.evaluate(test_data, test_labels)
print(f"Test accuracy: {test_accuracy}")

• The evaluate() function computes the test loss and accuracy on unseen data.
MAKING PREDICTIONS

predictions = model.predict(test_data)

• provides the predicted outputs (e.g., class probabilities or values) for the
predict()
given input data.
(F) SAVING AND LOADING MODELS

TensorFlow makes it easy to save your models and load them later.

SAVING THE MODEL:

model.save('my_model.h5') # Save as HDF5 file

LOADING THE MODEL:

loaded_model = tf.keras.models.load_model('my_model.h5')

• The model, including its architecture, weights, and training configuration, can be
saved and reused.

20. By kuber kr
jha
 Unit 5
8. TRAINING LOOPS IN TENSORFLOW
(A) WHAT IS A TRAINING LOOP?

A training loop is the sequence of operations that you repeat during the training
process of a machine learning model. It typically consists of:

1. Forward Propagation: Pass the inputs through the model to get the
predictions.
2. Loss Calculation: Compare the predictions to the true labels to compute the
loss (how "wrong" the model is).
3. Backpropagation: Compute the gradients using automatic differentiation
to understand how to update the weights.
4. Weight Update: Use the optimizer to adjust the weights in a direction that
reduces the loss.
5. Iteration: Repeat the process for a certain number of iterations or epochs.

THE TRAINING LOOP IS REPEATED MULTIPLE TIMES OVER ALL THE DATA IN THE
DATASET AND ACROSS SEVERAL EPOCHS TO ENSURE THE MODEL LEARNS WELL.
(B) STEPS IN A TRAINING LOOP
1. INPUT DATA: FEEDING DATA TO THE MODEL

At the start of each training iteration, data (inputs) is fed into the model. This data is
typically batched to ensure more efficient computation.

# Example: Batch input

for batch_inputs, batch_labels in train_dataset:
model(batch_inputs)

2. FORWARD PROPAGATION: PASSING THE DATA THROUGH THE MODEL

The input data is passed through the layers of the neural network (from input to
output). Each layer performs its specific computation, and the final output (prediction)
is produced.

# Forward pass
predictions = model(batch_inputs)

The model's output is compared to the ground truth labels (true labels), and the
loss is computed. The loss function measures how well the predictions match the
actual labels.

21. By kuber kr
jha
 Unit 5
3. LOSS CALCULATION: MEASURING THE ERROR

The loss quantifies how far off the model’s predictions are from the actual outputs. For
example, in a classification task, this could be cross-entropy loss; for regression, it
could be mean squared error.

# Loss computation
loss = loss_function(batch_labels, predictions)

Common loss functions in TensorFlow:

• Mean Squared Error (tf.keras.losses.MeanSquaredError) — for regression.

• Categorical Cross-Entropy (tf.keras.losses.CategoricalCrossentropy) — for classification.
• Binary Cross-Entropy (tf.keras.losses.BinaryCrossentropy) — for binary classification.

4. BACKPROPAGATION: COMPUTING GRADIENTS

Backpropagation involves computing the gradients of the loss function with respect to
the model’s trainable parameters (weights). TensorFlow’s Gradient Tape is used
for automatic differentiation to compute these gradients.

# Backpropagation (computing gradients)

with tf.GradientTape() as tape:
predictions = model(batch_inputs)
loss = loss_function(batch_labels, predictions)

gradients = tape.gradient(loss, model.trainable_variables)

5. WEIGHT UPDATE: OPTIMIZING THE WEIGHTS

After computing the gradients, an optimizer (like Adam, SGD, or RMSprop) is

used to update the model’s weights in the direction that minimizes the loss.

# Optimizer (e.g., Adam) updates weights

optimizer.apply_gradients(zip(gradients, model.trainable_variables))

6. REPEAT: THE LOOP CONTINUES UNTIL THE MODEL IS TRAINED

Once the weights are updated, the loop continues to the next iteration or epoch,
repeating the process for more batches or epochs. Each pass through the data helps the
model learn better representations and makes the predictions more accurate.

22. By kuber kr
jha
 Unit 5
(C) HOW TO IMPLEMENT A TRAINING LOOP
USING TENSORFLOW’S MODEL.FIT()

In TensorFlow, you can use the built-in fit() method to handle the training loop for
you. This abstracts away much of the complexity of the manual training loop. Here's
how you can use it:

# Assuming a model is already defined

model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

# Train the model on the training data

model.fit(train_dataset, epochs=10)

This function:

• Iterates over the entire dataset for the specified number of epochs.
• Performs forward propagation, loss calculation, backpropagation, and weight
update automatically.

Advantages:

• Much more concise and readable.

• Efficient and handles most of the manual steps internally.

CUSTOM TRAINING LOOP

If you need more control over the training process (for example, custom gradient
updates, adding callbacks, or managing different types of metrics), you can write a
custom training loop:

# Example of custom training loop

for epoch in range(epochs):
for batch_inputs, batch_labels in train_dataset:

with tf.GradientTape() as tape:

# Forward pass
predictions = model(batch_inputs, training=True)
loss = loss_function(batch_labels, predictions)

# Backpropagation
gradients = tape.gradient(loss, model.trainable_variables)

# Optimizer update
optimizer.apply_gradients(zip(gradients, model.trainable_variables))

print(f"Epoch {epoch+1}, Loss: {loss.numpy()}")

23. By kuber kr
jha
 Unit 5
BENEFITS OF CUSTOM TRAINING LOOPS:

• Flexibility: You can easily change how you compute losses, apply custom
learning rates, or modify the optimizer.
• Control: You can add features like logging, custom callbacks, or experimenting
with different training techniques.

(D) KEY CONCEPTS IN TRAINING LOOPS

1. BATCHING: DATA IS USUALLY DIVIDED INTO BATCHES. EACH BATCH IS PASSED
THROUGH THE MODEL SEPARATELY, AND THE GRADIENTS ARE COMPUTED AND APPLIED
FOR EACH BATCH.

• Batch size: The number of samples per batch.

• Stochastic Gradient Descent (SGD): A training method where a single
training example is used for each update.
• Mini-batch Gradient Descent: Combines the benefits of both full-batch and
stochastic gradient descent.
2. EPOCHS: ONE FULL PASS THROUGH THE ENTIRE DATASET.

• Epoch = Full training set passed through the model once.

• The model is trained for multiple epochs to improve its performance.
3. LEARNING RATE: THE SIZE OF THE STEPS TAKEN TO MINIMIZE THE LOSS FUNCTION.
TOO SMALL A LEARNING RATE CAN MAKE THE TRAINING SLOW, WHILE TOO LARGE CAN
CAUSE THE MODEL TO MISS OPTIMAL SOLUTIONS.

• Learning Rate Schedulers: Adjust the learning rate during training (e.g.,
reduce learning rate on plateau).

4. OPTIMIZER: THE ALGORITHM USED TO ADJUST THE MODEL'S WEIGHTS BASED

ON THE COMPUTED GRADIENTS.

• Common optimizers: Adam, SGD, RMSprop.

• Optimizers vary in their approach to computing the gradient step size.

24. By kuber kr
jha
 Unit 5
5. GRADIENT CLIPPING: SOMETIMES GRADIENTS CAN BECOME VERY LARGE, CAUSING
THE MODEL TO DIVERGE. GRADIENT CLIPPING IS A TECHNIQUE USED TO CLIP
GRADIENTS DURING BACKPROPAGATION TO PREVENT THEM FROM EXCEEDING A
CERTAIN THRESHOLD.
6. REGULARIZATION: TECHNIQUES LIKE L2 REGULARIZATION OR DROPOUT ARE
OFTEN APPLIED DURING TRAINING TO PREVENT THE MODEL FROM OVERFITTING.
(E) MONITORING TRAINING PROGRESS
1. METRICS:

• During training, it’s important to track the performance metrics such as

accuracy, loss, etc.
• These can be added when compiling the model, e.g., metrics=['accuracy'].
model.compile(optimizer='adam', loss='categorical_crossentropy', metrics=['accuracy'])

2. CALLBACKS:

Callbacks are functions or methods that are called during certain points in the training
loop. Common callbacks include:

• EarlyStopping: Stops training when the model's performance stops improving.

• ModelCheckpoint: Saves the model at specified intervals.
• LearningRateScheduler: Adjusts the learning rate dynamically.

Example with EarlyStopping:

early_stop = tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=3)

model.fit(train_data, train_labels, epochs=100, validation_data=(val_data, val_labels), callbacks=[early_stop])

3. VISUALIZATION:

You can visualize the training progress using TensorBoard or by plotting the loss and
accuracy curves over time.

9. FEATURES OF TENSORFLOW PLAYGROUND

WHAT IS TENSORFLOW PLAYGROUND?

TensorFlow Playground is an interactive web tool that allows you to visually explore
and experiment with the basic concepts of neural networks. It provides an easy-to-use
interface where you can adjust different parameters, train a simple neural network, and

25. By kuber kr
jha
 Unit 5
observe how it behaves as you modify the architecture and hyperparameters. It is an
excellent tool for understanding the mechanics of neural networks and

experimenting with them without needing to write any code.

KEY FEATURES OF TENSORFLOW PLAYGROUND

1. DATA

Data refers to the input that will be fed into the neural network for training and
testing. TensorFlow Playground allows you to choose different datasets, including
simple 2D datasets and more complex ones with features and labels.

TYPES OF DATA:

1. Linear Data: Data points that can be separated by a straight line.

2. Spiral Data: Data points organized in a spiral pattern, often used for testing
more advanced neural network architectures.
3. XOR Data: A more complex dataset where data points cannot be linearly
separated, showcasing the need for non-linear transformations.

26. By kuber kr
jha
 Unit 5
4. Gaussian Data: Randomly generated data points, often used for testing the
robustness of the model.

CUSTOMIZING DATA:

• You can modify the number of features (input dimensions).

• You can also adjust the labels (target outputs) to see how the network performs
on different types of problems.

2. THE RATIO OF TRAINING AND TEST DATA

In TensorFlow Playground, you can control how the data is split into training and
test datasets. This is crucial for evaluating the generalization ability of the neural
network.

IMPORTANCE OF TRAINING VS. TESTING DATA:

• Training Data: Used to train the model (e.g., learning weights).

• Testing Data: Used to evaluate the performance of the model after it has been
trained to ensure it generalizes well to unseen data.

Typically, you would split the data in a ratio, such as:

• 80% training data, 20% testing data

• 70% training, 30% testing
• This split can be adjusted in TensorFlow Playground to observe how different
splits affect model performance.

OVERFITTING AND UNDERFITTING:

• If too much data is used for testing, the model may not learn enough from the
training data, leading to underfitting.
• Conversely, if too little data is used for testing, the model might not be able to
generalize to unseen examples, leading to overfitting.

3. FEATURES

In the context of neural networks, features refer to the inputs to the model that are
used to make predictions.

27. By kuber kr
jha
 Unit 5
HOW FEATURES IMPACT MODEL PERFORMANCE:

• Linear Features: Features that can be easily separated by a straight line or

simple decision boundary.
• Non-linear Features: Features that require more complex models (like
multiple layers or non-linear activation functions) to separate them.

In TensorFlow Playground, you can experiment with the number of features, their
interactions, and how they influence the model's ability to make accurate predictions.

4. HIDDEN LAYERS

The hidden layers are intermediate layers between the input and output layers in a
neural network. These layers play a key role in learning complex patterns from the
data.

HOW HIDDEN LAYERS WORK:

• Each hidden layer is composed of neurons (also called units).

• Each neuron in the hidden layer applies an activation function to its input to
produce an output that is passed to the next layer.
• The number of hidden layers and the number of neurons per layer can
be adjusted in TensorFlow Playground.

IMPACT OF HIDDEN LAYERS:

• Single Hidden Layer: For simpler problems, a single hidden layer might be
sufficient.
• Multiple Hidden Layers: For more complex problems (like spiral or XOR
problems), more layers are required to capture the non-linear relationships
between input features.
KEY FACTORS TO CONSIDER:

• Too Few Layers: The network might not be able to capture the complexity of
the data (underfitting).
• Too Many Layers: The network might overfit the training data and perform
poorly on unseen data (overfitting).

28. By kuber kr
jha
 Unit 5
5. EPOCHS

An epoch refers to one complete pass of the entire dataset through the model during
training. In TensorFlow Playground, you can adjust the number of epochs to see how
the model's performance improves over time.

WHY EPOCHS MATTER:

• The number of epochs determines how long the model will train. With too few
epochs, the model might not learn enough from the data (underfitting).
• With too many epochs, the model may start memorizing the training data
(overfitting) and fail to generalize to new, unseen data.

In TensorFlow Playground, you can observe how the model's error decreases over each
epoch and how it stabilizes or increases as training continues.

6. LEARNING RATE

The learning rate controls how much the model's weights are updated with respect to
the computed gradient during training. It is a hyperparameter that significantly affects
the performance of the training process.

IMPACT OF LEARNING RATE:

• High Learning Rate: The model may converge too quickly and overshoot the
optimal solution, resulting in poor performance.
• Low Learning Rate: The model may converge very slowly and might get stuck
in local minima, making the training process inefficient.

In TensorFlow Playground, you can adjust the learning rate to see how it influences the
model's ability to converge during training.

7. ACTIVATION FUNCTION

The activation function introduces non-linearity into the model, allowing it to learn
more complex patterns from the data. TensorFlow Playground allows you to choose
from several activation functions.

29. By kuber kr
jha
 Unit 5
COMMON ACTIVATION FUNCTIONS:

1. Sigmoid: Outputs values between 0 and 1, making it useful for binary

classification tasks.
2. ReLU (Rectified Linear Unit): Outputs the input if it's positive and zero
otherwise. ReLU is commonly used in hidden layers due to its simplicity and
ability to speed up training.
3. Tanh (Hyperbolic Tangent): Outputs values between -1 and 1, providing a
smooth curve and handling negative values.
4. Softmax: Used in the output layer for multi-class classification problems.

HOW ACTIVATION FUNCTIONS AFFECT LEARNING:

• The activation function in the hidden layers determines the network's ability
to learn complex relationships between input features.
• Choosing the right activation function is important for model performance, as
each has different properties, such as how quickly they learn and whether they
suffer from problems like vanishing gradients.

8. REGULARIZATION

Regularization is a technique used to prevent the model from overfitting by

penalizing large weights or making the model simpler. In TensorFlow Playground, you
can enable L2 Regularization to control the magnitude of the weights during
training.

TYPES OF REGULARIZATION:

1. L2 Regularization (Ridge): Penalizes large weights by adding a term to the

loss function that is proportional to the sum of the squared weights. This
encourages the model to use smaller weights, which can help prevent overfitting.
2. Dropout: Temporarily "drops out" random units (neurons) during training to
prevent the model from becoming too reliant on any one feature or neuron.
9. PROBLEM TYPE

TensorFlow Playground allows you to experiment with different types of problems,

including:

1. Binary Classification: Two classes (e.g., 0 or 1).

30. By kuber kr
jha
 Unit 5
2. Multi-class Classification: More than two classes (e.g., classifying images into
categories).
3. Regression: Predicting continuous values (e.g., predicting house prices).
HOW PROBLEM TYPE AFFECTS NETWORK DESIGN:

• For binary classification, a sigmoid activation function is commonly used

in the output layer.
• For multi-class classification, the softmax activation function is used.
• For regression, a linear activation in the output layer is typically used to
predict continuous values.

31. By kuber kr
jha