Campaign Budget Optimization Using Deep Q-Learning

Technical Documentation

Table of Contents

1. Introduction
2. Project Structure
3. Core Components
4. Module Documentation
5. System Requirements
6. Data Requirements
7. Model Workflow
8. Step-by-Step Guide
9. Code Execution Flow
10. Output Interpretation
11. Conclusion

1. Introduction

This project implements an intelligent campaign budget optimization system using Deep
Q-Learning, combining Q-Learning with Deep Neural Networks to analyze historical campaign
data and recommend optimal budget allocations.

In the world of digital marketing, efficient budget allocation across ad campaigns is essential for
maximizing return on investment (ROI). With the growing complexity of digital channels and the
vast amounts of data generated by campaigns, making optimal budget decisions manually has
become increasingly difficult. This project leverages Deep Q-Learning (DQL) — a blend of
Q-Learning, a reinforcement learning technique, and deep neural networks — to automate and
optimize budget allocation across various advertising campaigns.

The Campaign Budget Optimization System developed in this project is designed to

intelligently analyze historical ad performance data, identify patterns in campaign metrics, and
recommend budget adjustments that maximize effectiveness and efficiency. Deep Q-Learning
enables this system to learn and adapt based on previous data without explicitly programmed
rules, allowing it to improve with each iteration. By framing budget allocation as a
decision-making process within an environment, the model can learn the action-value function
Q(s,a)Q(s,a)Q(s,a), which estimates future rewards for a given action aaa in state sss, capturing
insights into how each ad's performance will respond to different budget levels.
This system includes several key components:

● Deep Q-Network (DQN): The core model architecture that evaluates different budget
decisions and learns from feedback on each decision’s success.
● Prioritized Experience Replay: A technique that allows the model to prioritize certain
experiences, enhancing learning efficiency by focusing on impactful budget adjustments.
● Budget Reallocation Algorithm: A post-training process that reallocates budgets
across campaigns based on the trained model’s predictions.
● Visualization Tools: Dashboards that provide insights into budget reallocation impacts,
model performance, and feature importance for ad performance.

Reinforcement Learning Background

Q-Learning is a model-free reinforcement learning algorithm that learns to make optimal

decisions by experiencing outcomes of actions in an environment. The core principle is learning
an action-value function Q(s,a) that estimates the expected future rewards for taking action a in
state s.

Q-learning Update Rule:

Q(s,a) ← Q(s,a) + α[r + γ max Q(s',a') - Q(s,a)]

Where:

● α: Learning rate
● γ: Discount factor
● r: Immediate reward
● s: Current state
● a: Current action
● s': Next state
● max Q(s',a'): Maximum expected future reward

In Deep Q-Learning, the Q-function is approximated using a neural network, allowing us to

handle continuous state spaces.

2. Project Structure

project/
├── main.py
├── data_processing.py
├── model.py
├── training.py
├── budget_allocation.py
├── visualization.py
├── recommendations.py
├── feature_importance.py
└── utils.py

3. Core Components

3.1 Deep Q-Network (DQN)

The project uses a DQN with the following architecture:

● Input Layer: Campaign feature vector

● Hidden Layers: 3 dense layers (32 → 64 → 32 neurons)
● Output Layer: 5 actions (budget adjustment decisions)

3.2 Prioritized Experience Replay

Implements importance sampling for more efficient learning:

Priority: p_i = |δ_i| + ε

Sampling probability: P(i) = p_i^α / Σp_j^α
Importance sampling weight: w_i = (N·P(i))^(-β)

Where:

● δ_i: TD error
● ε: Small constant
● α: Priority exponent
● β: Importance sampling exponent
● N: Buffer size

4. Module Documentation

4.1 main.py
Purpose: Entry point for the application orchestrating the entire optimization process.

● Key Functions:
○ main(): Orchestrates data loading, feature preprocessing, model training,
budget reallocation, and visualization.

4.2 data_processing.py

Purpose: Manages data loading, cleaning, and preprocessing.

● Key Functions:
○ load_data(file_path): Loads campaign data from an Excel file.
○ clean_data(df): Performs data cleaning operations.
○ preprocess_data(df, user_specified_columns): Executes feature
engineering and normalization.

4.3 model.py

Purpose: Implements the DQN and Prioritized Experience Replay buffer.

● Classes:
○ PrioritizedReplayBuffer
○ DQNAgent

4.4 training.py

Purpose: Manages the DQN agent's training process.

● Key Functions:
○ train_model(df, feature_columns, agent, episodes,
batch_size, max_steps_per_episode): Trains the DQN agent on
historical data.

4.5 budget_allocation.py

Purpose: Implements budget reallocation logic.

● Key Functions:
○ reallocate_budget(agent, df, feature_columns): Generates new
budget allocations.

5. System Requirements
Required Libraries:

6. Data Requirements

The model expects an Excel file with columns:

● Ad set name
● Amount spent (INR)
● Results
● Impressions
● Reach
● Link clicks
● CTR (link click-through rate)
● CPM (cost per 1,000 impressions) (INR)
● Landing page views
● Platform (optional)
● Impression device (optional)

7. Model Workflow

7.1 Data Processing Pipeline

Raw Data → Cleaning → Feature Engineering → State Representation

7.2 Model Training Flow

Initialize DQN → Experience Collection → Prioritized Replay → Model

Update

7.3 Budget Allocation Process

State Evaluation → Action Selection → Budget Adjustment → Impact
Calculation

8. Step-by-Step Guide

Step 1: Data Preparation

1. Prepare your campaign data in Excel format.

2. Ensure all required columns are present.
3. Clean any missing or incorrect values.

Step 2: Running the Model

# Execute main.py
python main.py

# Follow the prompts:

Enter the path to your Excel file: /path/to/your/data.xlsx
Enter the columns to use: Platform,Impression device,Ad set name
Enter the number of training episodes (recommended: 100-1000): 500
Enter the batch size for training (recommended: 32-128): 64

Step 3: Monitor Training

The model displays:

● Training progress bar

● Episode rewards
● Epsilon value changes
● Total reward per episode

9. Code Execution Flow

9.1 Data Loading and Processing

# In main.py
df = load_data(file_path)
validate_data(df)
df = clean_data(df)
df, feature_columns = preprocess_data(df, user_specified_columns)

9.2 Model Initialization

state_size = len(feature_columns)
action_size = 5 # 5 possible budget adjustments
agent = DQNAgent(state_size, action_size)

9.3 Training Process

trained_agent, training_history = train_model(

df,
feature_columns,
agent,
episodes=episodes,
batch_size=batch_size
)

9.4 Budget Reallocation

df_reallocation = reallocate_budget(trained_agent, df,

feature_columns)

9.5 Results Generation

recommendations = generate_recommendations(df, df_reallocation)

print_recommendations(recommendations)
create_dashboard(df, df_reallocation, feature_importance,
training_history)

10. Output Interpretation

10.1 Performance Metrics

# ROI Improvement
roi = ((new_results - old_results) / old_results) * 100

# Cost Efficiency
efficiency = (new_cpa - old_cpa) / old_cpa * 100

# Overall Performance
improvement = (total_new_results - total_current_results) /
total_current_results * 100

10.2 Recommendations Format

{
'Combination': 'Platform_Device_Campaign',
'Action': 'Increase/Decrease/Maintain',
'Original Budget': 1000,
'Recommended Budget': 1200,
'Expected Improvement': 15.5,
'Reason': 'Higher budget leads to better results'
}

10.3 Visualization Outputs

The dashboard includes:

● Budget allocation comparisons

● Performance metrics trends
● Feature importance rankings
● ROI analysis
● Platform-specific insights

11. Conclusion
The Campaign Budget Optimization System demonstrates a promising application of Deep
Q-Learning in the field of digital marketing. By integrating reinforcement learning techniques
with real-world campaign data, the system provides a robust approach for optimizing advertising
budgets dynamically. Through the use of Deep Q-Networks and prioritized experience replay,
the system adapts to complex, continuous data spaces and captures nuanced relationships
between budget decisions and campaign outcomes.
The implementation of this system shows that machine learning can automate the traditionally
manual process of budget allocation, transforming it into an efficient, data-driven task. The
developed model’s ability to allocate budgets based on anticipated engagement, reach, and ROI
allows marketers to strategically maximize campaign effectiveness while minimizing
overspending. Additionally, the visualization dashboard offers actionable insights into campaign
performance, enabling marketing teams to easily interpret and act on the model’s
recommendations.

In conclusion, this project showcases the potential of using reinforcement learning for intelligent
budget optimization in advertising. Future work can expand the model’s scope to include
multi-objective optimization, additional campaign variables, and real-time learning. Such
enhancements would further increase the adaptability and precision of the system, making it an
even more valuable asset in the fast-evolving landscape of digital marketing.