Troubles with ElasticSearch

Over the last 12 months, I have played with ElasticSearch and its companions a handful of times – as a result, there's been a rant brewing that most of my colleagues have already been exposed to. ElasticSearch, LogStash and Kibana were a fairly simple solution a few short years ago. The latest iterations (v7+) of the software have doubled-down on proprietary features, feeling a lot more like the enterprise software that they've quickly become.

What I've been after is a small (3 node), stable deployment of ElasticSearch with Kibana. It needs to ingest logs from a couple of sources and we need to be able to monitor the data through dashboards and alert when things don't look right. To avoid adding another thing to maintain, I started with the 1st-party hosted option.

Elastic Cloud

The company behind Elastic appears to offer a simple cloud solution to bypass the complexity that they have injected into their software. It's only a few days into using their cloud solution that you realise that it's almost entirely useless unless you're willing to pay through the nose. A simple, 3 node deployment would work for the use-case I had specced out – if we could actually use the nodes we paid for. An arbitrary CPU credit system imposed on the VMs results in an underperforming cluster that can't even process light loads. The recommendation from Elastic is to just increase the size of the nodes, which does slightly increase the amount of credits available, but it's only the largest nodes which don't have the limited credit system. A single node of that size would cost vastly more than the entire cluster as originally planned – it doesn't seem like this is a mistake by Elastic.

Self-hosted

If you do choose to run your own cluster, then get ready for a world of pain. Lists of roles for nodes, certificate management between various components of the system (Kibana → ElasticSearch, ElasticSearch cluster communciation, Beat → Kibana, Beat → ElasticSearch) - the amount of configuration and understanding of the system effectively means that an expert on all things ElasticSearch needs to be in an organisation just to setup reliable logging. And that doesn't even begin to address the issues of ElasticSearch ingest vs Logstash (which is a topic I've been trying to get my head around).

"Elastic Cloud for Kubenetes"

Ultimately, the most sensible way I've found to deploy ElasticSearch is through their experimental (07/06/2021) Helm chart, to Kubernetes. They've called this mode of deployment "Elastic Cloud for Kubenetes" or ECK, a terrible name as this is a mode of deployment for the self-hosted variant. I've become a bit of a fan of Kubernetes, and whilst I can agree that there are other valid ways to write infrastructure as code and to manage and configure applications, this seems to be the industry standard for running complex software. Deploying ElasticSearch with Kubernetes removes all of the Firewall configuration, the cluster configuration, change deployment pains, and massively simplified the setup which I had originally envisaged. In fact, despite not being able to move to Kubernetes immediately in production, the process of setting up ElasticSearch in Kubernetes resulted in some understanding around performance improvments for our existing production cluster.

I'm not trying to suggest that this will be the same for everyone, there are disadvantages to using Kubenetes (some RAM is required for container orchestration software - and it represents a significant percent on smaller nodes) – but if you want to maintain any sanity while deploying this software then the only viable, cost-effective way to deploy is Kubernetes.

Licensing issues

Alerting & X-Pack

On a separate thread from the deployment is the observability featureset in Kibana. Instead of just being a search engine for your logs, Kibana is supposed to be a fully-featured dashboard product. Whilst the dashboarding experience itself leaves a lot to be desired, it does at least work without any paywalls. The recent "Kibana Alerts" feature is supposedly free - but only when sending alerts back into ElasticSearch or Kibana. This means a 3rd-party alerting solution is required to sit on-top of either ElasticSearch as a whole, or on top of the ElasticSearch index to which Kibana Alerts are written.

There are already solutions that sit on top of ElasticSearch for alerting - ElastAlert springs to mind as one which can be configured through YAML. From what I can tell, it has limited support for ElasticSearch 7, and isn't written with automatically closing alerts in mind. Another alternative would be the popular OpenDistro for ElasticSearch - which is fully featured and offers UI-based configuration within Kibana. OpenDistro is presented as an alternative to the proprietary X-Pack distribution, and as such does not integrate well with the rest of X-Pack. For my usage, I wrote a quick-and-dirty node application to tail the ElasticSearch alerts index and create & resolve alerts in Slack and in our pager system - really simple but effective, the only worry is that we might not know if it stops working.

ElasticSearch & OpenSearch

One final issue (or nail in the coffin, depending on whether you're already running a cluster before reading this), the license under which ElasticSearch was published has been changed recently. Instead of the original Apache 2.0 license, it has been changed to a non-free license. It is now a source-available license which limits the usage of the software, specifically when being provided as a service. I would strongly suggest that any readers consider whether Amazon's recent OpenSearch fork would serve as a suitable replacement.