Adding 1.21 Gigawatts to Applications with RabbitMQ (DPC 2015)
0 likes685 views
The document provides a comprehensive overview of RabbitMQ, including its installation, basic message queueing concepts, and different types of exchanges such as fanout, direct, and topic. It also addresses real-world use cases, message acknowledgment, and methods for clustering RabbitMQ to enhance reliability and processing capabilities. Additionally, the document discusses advanced features like priority queues and delayed message scheduling.
![fanout/consumer.php
1 $q = $channel->queue_declare(
2 '', false, false, false, true);
3 $queue_name = $q[0];
4
5 $channel->exchange_declare(
6 'test_exchange', 'fanout', false, false, false);
7 $channel->queue_bind($queue_name, 'test_exchange');](https://image.slidesharecdn.com/150627-adding1-150628083927-lva1-app6892/85/Adding-1-21-Gigawatts-to-Applications-with-RabbitMQ-DPC-2015-32-320.jpg)
![direct/consumer.php
1 $q = $channel->queue_declare('', false, false, false, true);
2 $queue_name = $q[0];
3 $channel->exchange_declare(
4 'test_direct', 'direct', false, false, false);
5
6 // Bind for each routing key we want (BINDING KEY)
7 $channel->queue_bind($queue_name, 'test_direct', 'apple');
8 $channel->queue_bind($queue_name, 'test_direct', 'orange');
9 $channel->queue_bind($queue_name, 'test_direct', 'banana');](https://image.slidesharecdn.com/150627-adding1-150628083927-lva1-app6892/85/Adding-1-21-Gigawatts-to-Applications-with-RabbitMQ-DPC-2015-41-320.jpg)
![1 $channel->basic_consume(
2 'test_queue', '', false, false, false, false,
3 function(AMQPMessage $message) {
4 echo $message->body . "n";
5 throw new Exception('oh noes!!!');
6
7 $channel = $message->delivery_info['channel'];
8 $tag = $message->delivery_info['delivery_tag'];
9 $channel->basic_ack($tag);
10 });
Message Acknowledgement](https://image.slidesharecdn.com/150627-adding1-150628083927-lva1-app6892/85/Adding-1-21-Gigawatts-to-Applications-with-RabbitMQ-DPC-2015-55-320.jpg)
![/etc/rabbitmq/rabbitmq.config
[
{rabbit, [
{loopback_users, []},
{vm_memory_high_watermark, 0.8}
]}
].](https://image.slidesharecdn.com/150627-adding1-150628083927-lva1-app6892/85/Adding-1-21-Gigawatts-to-Applications-with-RabbitMQ-DPC-2015-74-320.jpg)
![/etc/rabbitmq/rabbitmq.config
[{{{[{{[{{}}{][[[{[{{}[[}{[[{}[][}{}}}{}}{{,},]{
[[{rabbit, [{{}[[}{,,{}[][}{[][][{}{{{{}}}}[[}{{
{{}}{loopback_users, []},[][][]{}{}{}<}{[}[][][}
[{{[{vm_memory_high_watermark, 0.8}]]{}{[[[]]{}]
{{]}[{[{{}[[}{]]{}[][,{}[][}{[][][{}.[]}{]][][]}
]...{}[][,]{.}[][}{}[[[{}{][]}{}{}[}{}{}{]{}{}}[](https://image.slidesharecdn.com/150627-adding1-150628083927-lva1-app6892/85/Adding-1-21-Gigawatts-to-Applications-with-RabbitMQ-DPC-2015-75-320.jpg)
![node1$ rabbitmqctl cluster_status
Cluster status of node rabbit@node1 ...
[{nodes,[{disc,[rabbit@node1]}]},{running_nodes,[rabbit@node1]}]
...done.
Creating a cluster](https://image.slidesharecdn.com/150627-adding1-150628083927-lva1-app6892/85/Adding-1-21-Gigawatts-to-Applications-with-RabbitMQ-DPC-2015-76-320.jpg)
![node2$ rabbitmqctl join_cluster --ram rabbit@node1
node3$ rabbitmqctl join_cluster rabbit@node2
node3$ rabbitmqctl cluster_status
Cluster status of node rabbit@node3 ...
[{nodes,[{disc,[rabbit@node3,rabbit@node1]},{ram,[rabbit@node2]}]},
{running_nodes,[rabbit@node2,rabbit@node1,rabbit@node3]}]
...done.
Creating a cluster](https://image.slidesharecdn.com/150627-adding1-150628083927-lva1-app6892/85/Adding-1-21-Gigawatts-to-Applications-with-RabbitMQ-DPC-2015-77-320.jpg)
![node1$ rabbitmqctl stop_app
node2$ rabbitmqctl forget_cluster_node rabbit@node1
node1$ rabbitmqctl reset
node1$ rabbitmqctl start_app
node2$ rabbitmqctl cluster_status
Cluster status of node rabbit@node2 ...
[{nodes,[{disc,[rabbit@node3]},{ram,[rabbit@node2]}]},
{running_nodes,[rabbit@node2,rabbit@node3]}]
...done.
Removing Nodes](https://image.slidesharecdn.com/150627-adding1-150628083927-lva1-app6892/85/Adding-1-21-Gigawatts-to-Applications-with-RabbitMQ-DPC-2015-79-320.jpg)
Adding 1.21 Gigawatts to Applications with RabbitMQ (DPC 2015)
- 1. Adding 1.21 Gigawatts to Applications with RabbitMQ James Titcumb Dutch PHP Conference 2015
- 2. James Titcumb www.jamestitcumb.com www.roave.com www.phphants.co.uk www.phpsouthcoast.co.uk @asgrim Who is this guy?
- 3. What is a message?
- 6. Scaling with Rabbit RabbitMQApplication Background processing
- 7. Scaling with Rabbit RabbitMQApplication Background processing Background processing
- 8. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing
- 9. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing Background processing
- 10. Scaling with Rabbit RabbitMQApplication Background processing Background processing Background processing Background processing Background processing
- 11. Real world uses?
- 12. SOA
- 13. Why RabbitMQ?
- 15. (on trusty64 Vagrant, other OSs may vary) Installing RabbitMQ
- 16. ● add apt repo ○ deb http://www.rabbitmq.com/debian/ testing main ● add signing key ○ http://www.rabbitmq.com/rabbitmq-signing-key-public.asc ● apt-get update ● apt-get install rabbitmq-server ● rabbitmq-plugins enable rabbitmq_management ● sudo service rabbitmq-server restart Using Apt
- 23. Objective: Basic Queuing Producer Consumer test_queue 1 2 3 4 5
- 25. Please wait, connecting... 1 use PhpAmqpLibConnectionAMQPConnection; 2 3 $connection = new AMQPConnection( 4 'localhost', 5672, 'guest', 'guest', '/'); 5 6 $channel = $connection->channel();
- 26. basic/producer.php 1 use PhpAmqpLibMessageAMQPMessage; 2 3 $channel->queue_declare( 4 'test_queue', false, true, false, false); 5 6 $message = new AMQPMessage('my test message'); 7 $channel->basic_publish($message, '', 'test_queue');
- 27. basic/consumer.php 1 $channel->basic_consume( 2 'test_queue', '', false, true, false, false, 3 function(AMQPMessage $message) { 4 echo $message->body . "n"; 5 }); 6 7 while (count($channel->callbacks)) 8 $channel->wait();
- 31. 1 use PhpAmqpLibMessageAMQPMessage; 2 3 $channel->exchange_declare( 4 'test_exchange', 'fanout', false, false, false); 5 6 $message = new AMQPMessage('test msg #' . $id); 7 $channel->basic_publish($message, 'test_exchange'); fanout/producer.php
- 32. fanout/consumer.php 1 $q = $channel->queue_declare( 2 '', false, false, false, true); 3 $queue_name = $q[0]; 4 5 $channel->exchange_declare( 6 'test_exchange', 'fanout', false, false, false); 7 $channel->queue_bind($queue_name, 'test_exchange');
- 34. A word on Temporary Queues test_exchangeProducer Messages go nowhere
- 36. Objective: Direct Exchange test_direct BK = apple BK = banana, apple Consumer Consumer Producer BK = orange, banana, apple Consumer
- 37. Objective: Direct Exchange test_direct BK = apple BK = banana, apple Consumer Consumer Producer MESSAGE ROUTING KEY = ORANGE BK = orange, banana, apple Consumer
- 38. Objective: Direct Exchange test_direct BK = apple BK = banana, apple Consumer Consumer Producer MESSAGE ROUTING KEY = BANANA BK = orange, banana, apple Consumer
- 39. Objective: Direct Exchange test_direct BK = apple BK = banana, apple Consumer Consumer Producer MESSAGE ROUTING KEY = APPLE BK = orange, banana, apple Consumer
- 40. direct/producer.php 1 $channel->exchange_declare( 2 'test_direct', 'fanout', false, false, false); 3 4 $messageContent = 'test msg, key=' . $routingKey; 5 $message = new AMQPMessage($messageContent); 6 $channel->basic_publish( 7 $message, 'test_direct', $routingKey);
- 41. direct/consumer.php 1 $q = $channel->queue_declare('', false, false, false, true); 2 $queue_name = $q[0]; 3 $channel->exchange_declare( 4 'test_direct', 'direct', false, false, false); 5 6 // Bind for each routing key we want (BINDING KEY) 7 $channel->queue_bind($queue_name, 'test_direct', 'apple'); 8 $channel->queue_bind($queue_name, 'test_direct', 'orange'); 9 $channel->queue_bind($queue_name, 'test_direct', 'banana');
- 43. Exchanges: Topic
- 44. Objective: Topic Exchange test_topic BK = *.vegetable BK = # Consumer Consumer Producer BK = green.# Consumer BK = *.grass.* / *.*.long Consumer
- 45. Objective: Topic Exchange test_topic BK = *.vegetable BK = # Consumer Consumer Producer RED.VEGETABLE BK = green.# Consumer BK = *.grass.* / *.*.long Consumer
- 46. Objective: Topic Exchange test_topic BK = *.vegetable BK = # Consumer Consumer Producer GREEN.VEGETABLE BK = green.# Consumer BK = *.grass.* / *.*.long Consumer
- 47. Objective: Topic Exchange test_topic BK = *.vegetable BK = # Consumer Consumer Producer GREEN.GRASS.LONG BK = green.# Consumer BK = *.grass.* / *.*.long Consumer
- 50. Fetch message Logging Sequence ApplicationBrowser Log Server HTTP request JSON via AMQP Error! HTTP response RabbitMQ
- 51. Flexibility!
- 55. 1 $channel->basic_consume( 2 'test_queue', '', false, false, false, false, 3 function(AMQPMessage $message) { 4 echo $message->body . "n"; 5 throw new Exception('oh noes!!!'); 6 7 $channel = $message->delivery_info['channel']; 8 $tag = $message->delivery_info['delivery_tag']; 9 $channel->basic_ack($tag); 10 }); Message Acknowledgement
- 56. RPC
- 57. RPC example Producer Consumer test_queue 1 reply_to: amq.gen-Xa2 1’
- 58. TTL
- 59. TTL - per queue message Producer Consumer test_queue 1 2 3 4 5 } 10s
- 60. TTL - per message Producer Consumer test_queue 1 2 3 4 5 5s 3s 7s 1s 9s
- 61. TTL - queue Producer test_queue } 5s (if no consumers)
- 62. DLX
- 64. Scheduling / Delayed messages
- 65. Priority
- 67. Infrastructure
- 68. Problem: SPOF
- 70. Clustering RabbitMQ Node 1 RabbitMQ Node 3 RabbitMQ Node 2 RabbitMQ Node 4 RabbitMQ Node 5 RabbitMQ Node 6 Load Balance / Floating IP / Low TTL DNS etc.
- 72. Disk / RAM
- 73. Configuration...
- 74. /etc/rabbitmq/rabbitmq.config [ {rabbit, [ {loopback_users, []}, {vm_memory_high_watermark, 0.8} ]} ].
- 75. /etc/rabbitmq/rabbitmq.config [{{{[{{[{{}}{][[[{[{{}[[}{[[{}[][}{}}}{}}{{,},]{ [[{rabbit, [{{}[[}{,,{}[][}{[][][{}{{{{}}}}[[}{{ {{}}{loopback_users, []},[][][]{}{}{}<}{[}[][][} [{{[{vm_memory_high_watermark, 0.8}]]{}{[[[]]{}] {{]}[{[{{}[[}{]]{}[][,{}[][}{[][][{}.[]}{]][][]} ]...{}[][,]{.}[][}{}[[[{}{][]}{}{}[}{}{}{]{}{}}[
- 76. node1$ rabbitmqctl cluster_status Cluster status of node rabbit@node1 ... [{nodes,[{disc,[rabbit@node1]}]},{running_nodes,[rabbit@node1]}] ...done. Creating a cluster
- 77. node2$ rabbitmqctl join_cluster --ram rabbit@node1 node3$ rabbitmqctl join_cluster rabbit@node2 node3$ rabbitmqctl cluster_status Cluster status of node rabbit@node3 ... [{nodes,[{disc,[rabbit@node3,rabbit@node1]},{ram,[rabbit@node2]}]}, {running_nodes,[rabbit@node2,rabbit@node1,rabbit@node3]}] ...done. Creating a cluster
- 79. node1$ rabbitmqctl stop_app node2$ rabbitmqctl forget_cluster_node rabbit@node1 node1$ rabbitmqctl reset node1$ rabbitmqctl start_app node2$ rabbitmqctl cluster_status Cluster status of node rabbit@node2 ... [{nodes,[{disc,[rabbit@node3]},{ram,[rabbit@node2]}]}, {running_nodes,[rabbit@node2,rabbit@node3]}] ...done. Removing Nodes
- 80. Solution 2: HA
- 81. HA + Queue Mirroring RabbitMQ Node 1 RabbitMQ Node 2 Load Balance / Floating IP / Low TTL DNS etc.
- 83. Any questions? :) https://joind.in/14253 James Titcumb @asgrim