Practical Distributed Processing
Using MySQL Built-In Functionality
Bob Burgess radian6 Technologies MySQL User Conference 2010
Who Am I
Database Analyst, radian6 Several Years Experience:
Sybase, Oracle, MySQL AIX, Linux programming
Scope
MySQL concepts used in this system How I solved our particular problem Practical set-up of our distributed system
Scope NOT
Complete course on distributed processing Complete coverage of related MySQL functionality
The Problem
Influencer calculation: complex and getting more complex More and more data More and more customers and profiles Not scalable to run the stored procedure on just the main database
Design Goals
Move calculation off of main database Keep existing stored procedure No single controller Horizontally scalable!
Influencer calculation
Whos most influential? (For this topic.)
subset of database do stuff influencer data
(two MyISAM tables)
sphinx index
MySQL Functionality Used
MySQL Functionality Used
Replication Blackhole Storage Engine Federated Storage Engine Public-Key Authentication
Replication
Master Database Replica Database
binlogs
INSERT... UPDATE... INSERT... DELETE...
relay logs
INSERT... UPDATE... INSERT... DELETE...
Replication
Master Database
server_id = 1 log_bin = /store/log/mysql-bin grant replication slave on *.* to 'repl'@'%' identified by 'secret'; /store/log/mysql-bin.000001
binlogs
INSERT... UPDATE... INSERT... DELETE...
Replication
server_id = 2 relaylog = /store/relaylog/relaylog
change master to master_host="masterserver", master_port=3306, master_user="repl", master_password="secret", master_log_file="mysql-bin.000001", master_log_pos=0; start slave;
Replica Database
relay logs
INSERT... UPDATE... INSERT... DELETE...
/store/relaylog/relaylog.000001
Black Hole Tables
> create table t (a int) engine=blackhole; Query OK, 0 rows affected (0.02 sec)
> insert into t values (1); Query OK, 1 row affected (0.00 sec) > select * from t; Empty set (0.00 sec)
Black Hole Tables
Before-Insert trigger
create trigger t_bi before insert on t for each row insert into real_table values (new.a);
real table
Federated Tables
Points to a table in another database
server1
table `t`
create table t (a int)
federated table `tm` server2 create table tm (a int) engine=federated connection="mysql://user:pass@server1/schemaname/t"
Not enabled by default in 5.1!
Federated Tables
Points to a table in another database
server1
table `t`
create table t (a int)
federated table `tm` server2
federated table `tm` server3
federated table `tm` server4
create table tm (a int) engine=federated connection="mysql://user:pass@server1/schemaname/t"
Public-Key Authentication
Secure shell (ssh) or secure copy (scp) without a password
Public-Key Authentication
# scp file mysql@main:/...
worker
main db
# cd /root/.ssh # ssh-keygen # cat id_rsa.pub
# vi /home/mysql/.ssh/authorized_keys
Distributed System Why It Works For Us
All involved database data fits on the worker database Calculation produces a self-contained data set MyISAM can be used for the resulting data
Distributed System General Architecture
main db server main db
replication
federated
MYD & MYI
scp file copy
worker db
worker 1
MYD & MYI
Distributed System General Architecture
main db server main db
MYD & MYI MYD & MYI
MYD & MYI
worker db worker 1
MYD & MYI
worker db worker 3 worker db worker 2
MYD & MYI
MYD & MYI
The Non-Queue... Queue
--- Topics --No. Name 101 Apples 102 Oranges 103 Grapes 104 Bananas 105 Pears
--- To-do List --101 104 105
The Non-Queue... Queue
------ Topics--Topics --101 Apples 102 Oranges No. Name 103 Grapes 101 Apples 104 Bananas 102 Oranges 105 Pears
Recalc -Reqd Full No 103 Grapes No 104 Bananas Partial 105 Pears Full
Recalc -Date Priority Running 10:00am high 101 8:00am 0 7:30am 0 10:10am high 302 10:05am low 0
not running highest priority earliest recalc date
Processing Cycle
Get work... Get next Topic to do
is there any? yes
no
sleep
Processing Cycle
Is data current? Let replication catch up!
replication stopped?
yes
sleep 10 min.
replication behind?
yes
sleep 1 min.
Processing Cycle
Replace stale Topic Profile someone else likely has it...
waited for replication?
yes
get a new Topic to do
Processing Cycle
Playing Nice with Others
mark Topic with my ID
sleep short random time
Topic still has my ID?
no
sleep 1 min.
begin again
Processing Cycle
run the stored proc
success ? yes
no
log the error
sleep 1 min.
begin again
Processing Cycle
flush the data table for this Profile (locally)
copy to main db server
flush the data table for this Profile (on main db)
update Profile table on main: not running
Processing Cycle
back to the top
Code main loop
trap 'STOP=1' TERM STOP=0 while [ $STOP -eq 0 ] ; do
done
Code who am I?
NODEID_QUERY="select @@server_id-1000;" INSTANCE=$2 NODEID=`$MYSQL -e"$NODEID_QUERY"` UNIQUE_INSTANCEID=$((NODEID*100+INSTANCE))
Code whos next?
NEEDY=`$MYSQL_MASTER -e"$NEEDY_TP_QUERY" \ 2>/tmp/mysql.err` set -- $NEEDY TPID=$1; RECALC_SCOPE=$2; RECALC_DATE=$3
Code whos next, the SQL
NEEDY_TP_QUERY=" select topicFilterId,influencerRecalcReqd, unix_timestamp(influencerRecalcDate) RecalcDate from TopicFilter where influencerRecalcReqd>0 and influencerRunning=0 and influencerRecalcDate<=now() and active in (1,2) and influencerPriority>=$MINPRIORITY order by influencerPriority desc, influencerRecalcDate asc, topicFilterId limit 1;"
Code inner loop
while [ ! -z "${TPID}" -a $STOP -eq 0 ]; do
done
Code how far behind?
REPLICATION_DELAY= `$MYSQL -e'show slave status\G' | grep "Seconds_Behind_Master" | cut -f2 -d':'`
Code call the proc
$MYSQL -e"call proc(${TPID},${RECALC_SCOPE})" >${INFLUENCER_LOG}${TPID}.${INSTANCE}.log 2>&1 RUN_SUCCESS=`tail -1 ${INFLUENCER_LOG}${TPID}.${INSTANCE}.log | grep -c SUMMARY` if [ $RUN_SUCCESS -eq 1 ] ; then copy tables else report error fi
Code send the result back
$MYSQL -e"use influencer; flush tables Dynamics_${TPID};" if scp -p -B ${TABLE_SOURCE}/Dynamics_${TPID}.* ${TABLE_MASTER_PUSH_LOC} 2>&1 && \ scp -p -B ${TABLE_SOURCE}/Post_${TPID}.* ${TABLE_MASTER_PUSH_LOC} 2>&1 then copy files to backup remove tables from worker else report error fi
Code was I killed?
if [ $STOP -eq 1 ]; then echo "---- Script stopped by Kill (TERM signal) ----" fi
A Blackhole Table Trigger
create table BlogPost (...) engine=blackhole; create table PostAuthor ( blogPostId bigint not null primary key, author varchar(189)) default character set=utf8; create trigger BlogPost_bir before insert on BlogPost for each row begin if new.mediaProviderId>1 then insert ignore into PostAuthor (blogPostId,author) values (new.blogPostId, new.author); end if; end;
Launch Script
Shell script to launch workers On
Starts several workers
Off
sends a signal to each worker stops after this stored proc call
Kill
abort and clean up database
Monitor Scripts
Replication Disk space Errors from the worker script
Procedure fails scp fails gzip fails replication is behind
Design Decisions
MyISAM vs. InnoDB
Size Tables easily copied Table locking / Replication: Smaller Queries Required
SSD Storage
Development Directions
Migrate stored proc to Java More workers Problems in Code not atomic:
flush table / copy table copy result tables to main db
Take Away...
Replication and Federated tables aren't hard. Thousands of MyISAM tables isn't crazy.
Distributed
processing isn't rocket surgery.
Thank you!
Bob Burgess [email protected]
Slides on the conference site. Email me for sample scripts.
