How to connect to multiple Elasticsearch client nodes without a load balancer

Originally published on Nov 10, 2017, at ObjectRocket.com/blog.

Since all Rackspace ObjectRocket for Elasticsearch® clusters come with multiple client nodes, a common question we get from our customers is why we don’t provide a load balancer to connect to all of them.

Depending on your preferences, you might see a single connection to a single node as a point of cluster failure. A load balancer can prove effective to manage a pool of connections. However, Elasticsearch is designed to work without a load balancer. In this blog post, we go over how to use all of the clients we provide in Elasticsearch.

The Python setup

Let’s look at the Python® setup:

from elasticsearch import Elasticsearch
Import certifi

es = Elasticsearch(['dc-port-0.es.objectrocket.com', 'dc-port-1.es.objectrocket.com', 'dc-port-2.es.objectrocket.com', 'dc-port-3.es.objectrocket.com'],
    http_auth=('YOUR_USERNAME', 'YOUR_PASSWORD'),
    port=12345,
    use_ssl=True,
    verify_certs=True,
    ca_certs=certifi.where(),
)

The first argument shows you that Elasticsearch accepts a list of hosts. That means the client can continue with the setup on their own. The setup is similar to other tools, like Beats. The setup script would be similar to the following example:

output:
elasticsearch:
hosts: ["https://dc-port-0.es.objectrocket.com:port", "https://dfw-port-1.es.objectrocket.com:port", "https://dfw-port-2.es.objectrocket.com:port", "https://dfw-port-3.es.objectrocket.com:port"]

# HTTP basic auth
username: "YOUR_USERNAME"
password: "YOUR_PASSWORD"

Most tools and clients available support multiple hosts. To make it easy for you, we include connection snippets in the ObjectRocket UI for Python®, Ruby, C#, Java®, Javascript®, Go, Logstash, and Beats. Just pick the technology you’re using and then copy the snippet with your hostnames pre-populated.

Alternatives for applications that won’t take multiple hosts

A few applications don’t make a list of hosts. The most notable is Kibana®, which accepts a single host. To work around this, you can use a couple of alternatives.

Point each part of your app at a different client

For non-mission-critical uses and applications, you can point at a single client node. Unless the application has a super-high request rate, a single client should be able to manage the load. If you have a number of these types of applications, just point each one at a different client to balance the load.

Load balance locally

In the few cases where the client or application doesn’t support it, and you need redundancy in client connections, you can set up local load-balancing. You can do this with nginx®, HAProxy®, and others, or just set up a local hostname in Domain Name System (DNS) that uses round-robin between the Elasticsearch clients. Once again, we see very few cases where this is necessary, but there are solutions available when it does come up.

Conclusion

Almost every scenario you come across allows you to supply a list of hosts that handle the balancing for you, but, in a pinch, there are examples out there to help you load balance locally.

Learn more about Rackspace DBA Services.

Use the Feedback tab to make any comments or ask questions. You can also click Sales Chat to chat now and start the conversation.