CT80A0000 – DATA-INTENSIVE SYSTEMS
DISTRIBUTED QUERY PROCESSING
Lecture
Jiri Musto, D.Sc.
QUERY PROCESSING
High level user query
Query
Processor
Low-level data manipulation
commands for DBMS
3
QUERY PROCESSING COMPONENTS
Query language
E.g. SQL
Query execution
Steps of a given query
Query optimization
Manual or automatic (DBMS)
We assume a homogeneous distributed DBMS
4
OPTIMIZATION: SELECTING ALTERNATIVES
Each query can have multiple methods to reach the result
2+2, 1+3, 2*2, 2^2
To select the best option, there needs to be some metrics
Minimize the cost function (I/O, CPU, Communication)
Fastest, cheapest, transfer cost, total time, response time, etc.
Either done automatically by the system or manually by the user
5
TYPES OF OPTIMIZERS
Exhaustive search
Cost-based
Optimal
Heuristics
Not optimal (near optimal)
Optimize individual operations
Find a solution with reasonable cost
6
ONE OR MULTIPLE QUERIES
Single query at a time
Cannot use common intermediate results
Easier to optimize
Multiple queries at a time
Efficient if many similar queries
Decision space is much larger
7
OPTIMIZATION STRATEGY
Static
Optimize prior to the execution
Difficult to estimate the size of the intermediate results & error propagation
Dynamic
Run time optimization
Exact information on the intermediate relation sizes
Have to reoptimize for multiple executions
Hybrid
Compile using a static algorithm
If the error in estimate sizes is larger than threshold, reoptimize at run time
8
OPTIMIZATION DECISION SITES
Centralized
One site decides “best” schedule
Simple
Need knowledge about the entire distributed database
Distributed
Each site cooperates to determine schedule
Need only local information
Cost of cooperation
Hybrid
One site determines the global schedule
Each site optimizes the local schedules
9
QUERY PROCESSING METHODOLOGY
10
STEP 1 – QUERY DECOMPOSITION
Normalize and analyze the query
Bad queries are rejected
Simplify the query as much as possible
Remove irrelevant parameters
Restructure the query to get a more efficient option
11
STEP 2 – DATA LOCALIZATION
What fragments / partitions are involved in the query
Localize the query plan
Global parameters are changed to local ones
Optimize the localized plans
12
STEP 3 – GLOBAL QUERY OPTIMIZATION
Find the best (not necessarily optimal) global schedule
Minimize a cost function
Join processing
Bushy vs. linear trees
Which relation to ship where?
Ship-whole vs ship-as-needed
How data is joined together
Semijoins or not, what join methods are used
13
QUERY OPTIMIZATION PROCESS
QEP = Query Execution Plan Search/Solution space
Set of possible solutions (query trees)
Cost model
E.g. I/O cost + CPU cost + communication cost
Search strategy
How do we move inside the search space?
Exhaustive search, heuristic algorithms
Deterministic, randomized
14
NORMAL QUERY PROCESSING ISSUES
The optimizer needs sufficient knowledge about runtime
Runtime conditions should remain stable during query execution
Good for systems with few data sources and a controlled environment
What about changing environments?
Or large numbers of data sources?
Unpredictable runtime conditions?
15
EXAMPLE: QEP WITH BLOCKED OPERATOR
Join
• Student data cannot be
accessed
Join
• Whole process is blocked Grade
until regaining access
Join
• What if this would be
Course
reorganized?
Student Project
16
ADAPTIVE QUERY PROCESSING
Receive information from the execution environment
Modify process accordingly
Communication between optimizer and runtime environment and other components
Additional components
Monitoring (statistics, data, network, cost), assessment, reaction
Embedded in control operators of QEP
Tradeoff between reactiveness and overhead of adaptation
Change schedule, replace operators, modify behaviour
17
CONCLUSION ON QUERY PROCESSING
There are multiple ways to organize query processing
Centralized, distributed, hybrid
Queries need to be optimized automatically or manually
There are multiple methods of searching for optimal solution
The organization of query processing has an impact
Most often the best option is not the “most optimal” solution
18
