Amazon EC2 servers m5.large for master and m5.xlarge for agents each with 200 GB SSD for indexes.Keep in mind that the Rancher Master server and Rancher Agent 3 are NEVER displayed (but they’re there for sure!). This is what the entire stack looks like. Applications are nurtured by Beats containers and lead their life cycles on VMWare servers with Rancher.Elasticsearch, Kibana, Logstash, Traefik, Curator and Cerebro all reside on Amazon VPC servers with Rancher.And also a homebrew log file cleaner to keep the logs themselves under control. Then I added a Beats stack with filebeat and metricbeat on all application servers to collect logs or other data. The Traefik containers are in turn connected to Amazon ALB’s for Kibana and Cerebro. Traefik containers handle both the secure SSL traffic on TLS 1.2 and good Ciphers only (I dislike POODLE) and the load balancing to the applications in the back. įor ElasticSearch management I picked Cerebro (Kopf does not work with Secure Elastic Stack) and last, Curator to clean up old indexes. I used the ELK Stack from Rancher 1.6 (see here ) and replaced all the containers with docker Search-Guard container for Elasticsearch, Logstash and Kibana from. When building this ELK platform, I looked at it in much the same way. The fun thing with LEGO is that you can combine all types of bricks and make something completely new. Setting up the entire stackĪs a child, I almost exclusively played with Lego (Nintendo and internet didn’t really exist or were still pretty unheard of). Simply looking at the cost, we choose to use the open source version of Search Guard, the ‘Community Edition’ at ACA Group.
Securing filebeats license#
An important difference is that X-Pack needs a paid license and Search Guard is open source for a lot of features and you can get a paid license when you need to integrate LDAP or AD and/or you need to control access to specific data.
Usually, when you look at an Elk Stack in a company, you can find it at an http address (so not encrypted), you do not have to log in (even guests of the network that stumble upon the server address can have a look at your log errors) and anyone can just manipulate the data inside Elasticsearch (ideal if you want to cover up failed login attempts)!īoth the open source Search Guard security plugin as well as X-Pack from (recently renamed to the less-catchy Elastic Stack Features ) help make the Elastic Stack secure. However, little to no attention is paid to safety and security. And it actually works as well, no funny business there. If you are using Rancher (for Docker container orchestration), you can actually have a running Elastic Stack within just a few minutes.
Securing filebeats install#
Then, everything is visualized in Kibana with nice graphs and filters so you can easily find what you need.Īn Elastic Stack is fairly easy to install with Docker. Introduction to B+ELK a.k.a Elastic StackĪn Elastic Stack uses different ‘Beats’ to collect and ship the data to Logstash, where it is transformed and then pushed and stored in Elasticsearch. A system like that gets the logs from all your servers, transforms and indexes them and finally stores them, ready for analysis and reporting. The solution is to use a log management system. It gets even harder when the servers and their logs are part of a highly available environment, where there are lots of servers and lots of logs and it is even more difficult (read: impossible) to get logs from servers that no longer exist due to an autoscaling ‘scale in’ event. Giving all these people access to these logs will probably result in living hell, if it’s at all possible. And then there are your colleagues: developers, auditors and regular users that need to read these logs or get reports. Let’s say you are a system engineer/person involved in DevOps and have all these logs sitting on lots of permanent or temporary (cloud) servers.
0 kommentar(er)
