Introduction

To achieve its purpose, correlating user information with network performance data, WiFiMon needs RADIUS and/or DHCP logs to be streamed in an Elasticsearch structure. For that purpose, an ELK cluster was built on VMs. A total of five VMs were used, with three of them defined as Elasticsearch master-eligible and data nodes, one VM configured as coordinating node where Kibana was installed too, and another one dedicated to Logstash.

The sources generating log files are a freeRadius and dhcp a DHCP server where Filebeat was installed as an agent. Thus the data flow starts with Filebeat collecting log events and forwarding them to Logstash. At Logstash, the logs are filtered/enriched according to the needs of WiFiMon, before sending them toward Elasticsearch nodes in the cluster.

VMs Specifications

This setup consists of five VMs each of them having the specifications shown in the following table:

Cluster Setup

Setting up an ELK cluster means installing the software packages implementing its components. Some configuration must also be done as a preparation, before starting with the configuration of the cluster itself.

Anchor
DNS and Roles DNS and Roles

DNS and Roles

The following table shows the DNS configuration and the role each machine plays in the cluster.

...

Cluster node is considered to be the one that joins the cluster. In this setup, cluster nodes are the master-eligible/data nodes and the coordinating node. The pipeline node is not, it doesn’t join the cluster.

Package Installation

A cluster is a collection of nodes.

Being a cluster of Elasticsearch nodes, Java (at least version 8) is required, so the java-1.8.0-openjdk package was installed on in each node.

Having the Java dependency satisfied, the next step was to install the elasticsearch package on in each cluster node, that is not in the pipeline node. For more information see Install Elasticsearch with RPM.

On In the coordinating node, along with elasticsearch, the kibana package was installed, too. For more information see Install Kibana with RPM.

On pipeline node The logstash package was installed in the logstash packagepipeline node. For more information see Installing Logstash.

The filebeat package was installed in a the dhcp server and in the freeRadius server which implements the Eduroam Service Provider. For more information see Repositories for APT and YUM.

All the packages implementing the cluster's components (elasticsearch, logstash, kibana, filebeat) must be of the same version. This setup is about version 7.8.0.

System Configuration

Each node’s hostname is set to its FQDN, according to the values shown in the VMs DNS table. This value is referenced in the configuration file of Elasticsearch, and is also used in certificates for hostname validation.

It is recommended to disable system swapping, which can result in parts of JVM Heap or even its executable pages being swapped out to disk.

...

Port 9200/tcp is used to query the cluster using the Elasticsearch REST API. Port 9300/tcp is used for internal communication between cluster nodes. Port 5044/tcp is where Logstash listens for beats of log events sent from Filebeat. Port 5601/tcp is used to access Kibana platform from the browser.

SSL/TLS Certificates

...

Port 9200/tcp is used to query the cluster.

This setup uses firewalld for the implementation of firewall. On each node a "wifimon" custom zone is created to hold the specific configuration. On wifimon-kibana.example.org node

...

, furthermore, some configuration goes into public zone to allow access for the kibana platform and the cluster components.

On wifimon-kibana.example.org:

firewall-cmd --zone=public --list-ports 
5601/tcp

firewall-cmd --zone=public --list-rich-rules 
rule family="ipv4" source ipset="wifimon-components" port port="9200" protocol="tcp" accept

firewall-cmd --ipset=wifimon-components --get-entries

Create the instances.yml file with the following contents:

instances:
- name: node1
  dns: wifimon-node1.example.org
  ip: 10.0.0.1
- name: node2
  dns: wifimon-node2.example.org
  ip: 10.0.0.2
- name: node3
  dns: wifimon-node3.example.org
  ip: 1
10.0.0.2
10.0.0.3
10.0.0.3
- name: kibana
  dns: wifimon-kibana.example.org
  ip:.5
10.10.10.111
192.168.1.15

firewall-cmd --zone=wifimon --list-ports 
9300/tcp

firewall-cmd --zone=wifimon --list-sources
10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4
- name: logstash
  dns: 

Generate the CA certificate and key:

/usr/share/elasticsearch/bin/elasticsearch-certutil ca --ca-dn CN='WiFiMon CA' --days 3650 --keysize 4096 --out $(pwd)/wifimon-ca.zip --pass --pem

The above will create the wifimon-ca.zip file in the current directory. Unzip it.

Generate components certificates and keys:

On wifimon-node{1,2,3}.example.org:

firewall-cmd --zone=wifimon --list-ports 
9300/tcp

firewall-cmd --zone=wifimon --list-sources
10.0.0.1 10.0.0.2 10.0.0.3 10.0.0.4

On wifimon-logstash.example.org:

firewall-cmd --zone=wifimon --list-ports 
5044/tcp

firewall-cmd --zone=wifimon --list-sources
10.10.10.111 192.168.1.15

SSL/TLS Certificates

The cluster communication is secured by configuring SSL/TLS encryption. The elasticsearch-certutil was used to generate a CA certificate, utilized for signing the certificates of the cluster components. This utility comes with the elasticsearch installation, and in this case was used the one installed in the wifimon-kibana.example.org node.

Create the instances.yml file with the following contents:

instances:
- name: node1
  dns: wifimon-node1.example.org
- name: node2
  dns: wifimon-node2.example.org
- name: node3
  dns: wifimon-node3.example.org
- name: kibana
  dns: wifimon-kibana.example.org
- name: logstash
  dns: wifimon-logstash.example.org
- name: filebeat

Generate the CA certificate and key:

/usr/share/elasticsearch/bin/elasticsearch-certutil ca --ca-dn CN='WiFiMon CA' --days 3650 --keysize 4096 --out $(pwd)/wifimon-ca.zip --pass --pem

The above will create the wifimon-ca.zip file in the current directory. Unzip it.

Generate components certificates and keys:

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca-cert $(pwd)/ca/ca.crt --ca-key $

/usr/share/elasticsearch/bin/elasticsearch-certutil cert --ca-cert $(pwd)/ca/ca.crt --ca-key $(pwd)/ca/ca.key --days 1000 --in $(pwd)/instances.yml --keysize 4096 --out $(pwd)/wifimon-certs.zip --pass --pem

...

For more information on elasticsearch-certutil see its documentation page.

Cluster Configuration

Configuring a cluster means configuring the nodes it consists of, which in turn means defining cluster-general and node-specific settings. Elasticsearch defines these settings in configuration files located under the /etc/elasticsearch directory.

Anchor
jvm_options jvm_options

JVM Options

JVM options are defined in the /etc/elasticsearch/jvm.options file. By default Elasticsearch tells JVM to use a heap of minimum and maximum of 1 GB size. The more heap available, the more memory it can use for caching, however it is recommended to use no more than 50% of the total memory.

...

Master-Eligible / Data Nodes

In a heavy data traffic cluster of many nodes, it is recommended to have the master-eligible and data nodes separated and dedicated to their own role. In this setup, however, there are three nodes configured as having both functionalities.

By default a node is a master-eligible, data, and ingest node, which means (a) it can be elected as master node to control the cluster, (b) it can hold data and perform operations on them, and (c) it is able to filter and enrich a data document before being indexed. Having a dedicated pipeline node with filtering/enriching capabilities there’s no need for the ingest feature, it has been however enabled because it is used for monitoring purposes.

...

cluster.name: wifimon
node.name: ${HOSTNAME}
node.master: true
node.voting_only: false
node.data: true
node.ingest: true
node.ml: false
cluster.remote.connect: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: wifimon-node1.example.org
discovery.seed_hosts: [
    "wifimon-node1.example.org",
    "wifimon-node2.example.org",
    "wifimon-node3.example.org"
]
#cluster.initial_master_nodes: [
#    "wifimon-node1.example.org",
#    "wifimon-node2.example.org",
#    "wifimon-node3.example.org"
#]
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.http.ssl.key: /etc/elasticsearch/certs/node1.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/node1.crt
xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/certs/ca.crt
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/node1.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/node1.crt
xpack.security.transport.ssl.certificate_authorities:
 /etc/elasticsearch/certs/ca.crt
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true

...

cluster.name: wifimon
node.name: ${HOSTNAME}
node.master: true
node.voting_only: false
node.data: true
node.ingest: true
node.ml: false
cluster.remote.connect: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: wifimon-node2.example.org
discovery.seed_hosts: [
    "wifimon-node1.example.org",
    "wifimon-node2.example.org",
    "wifimon-node3.example.org"
]
#cluster.initial_master_nodes: [
#    "wifimon-node1.example.org",
#    "wifimon-node2.example.org",
#    "wifimon-node3.example.org"
#]
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.http.ssl.key: /etc/elasticsearch/certs/node2.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/node2.crt
xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/certs/ca.crt
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/node2.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/node2.crt
xpack.security.transport.ssl.certificate_authorities:
 /etc/elasticsearch/certs/ca.crt
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true

...

cluster.name: wifimon
node.name: ${HOSTNAME}
node.master: true
node.voting_only: false
node.data: true
node.ingest: true
node.ml: false
cluster.remote.connect: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: wifimon-node3.example.org
discovery.seed_hosts: [
    "wifimon-node1.example.org",
    "wifimon-node2.example.org",
    "wifimon-node3.example.org"
]
#cluster.initial_master_nodes: [
#    "wifimon-node1.example.org",
#    "wifimon-node2.example.org",
#    "wifimon-node3.example.org"
#]
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.http.ssl.key: /etc/elasticsearch/certs/node3.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/node3.crt
xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/certs/ca.crt
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/node3.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/node3.crt
xpack.security.transport.ssl.certificate_authorities:
 /etc/elasticsearch/certs/ca.crt
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true

...

The node.name is set to the value of ${HOSTNAME}, that is the value of the node’s FQDN. This The value for this setting can also be configured given explicitly to some value.

The node.master makes this node eligible to be elected as a master node which controls the cluster. Every master-eligible node, which is not a voting_only node, can be the master node of the cluster.

...

The cluster.remote.connect setting makes this node to function as a cross-cluster client able to connect to remote clusters. This is not the case of this setup so it is set to false.

...

For more information about the aforementioned settings see Node, Network Settings, Important discovery and cluster formation settings, and Secure a cluster.

Coordinating Node

A coordinating node is a node that has node.master, node.data, and node.ingest settings set to false, which means you are left with a node actually behaving as a load-balancer, routing the requests on the appropriate nodes in the cluster.

A coordinating node is an Elasticsearch node which joins the cluster as every cluster node. In this setup, the coordinating node is named wifimon-kibana.example.org because the Kibana visualization platform has been installed and configured on in it.

Below is the configuration of wifimon-kibana.example.org as an Elasticsearch coordinating node. It follows the same pattern as the master-eligible/data nodes, but with their functionalities set to false.

...

cluster.name: wifimon
node.name: ${HOSTNAME}
node.master: false
node.voting_only: false
node.data: false
node.ingest: false
node.ml: false
cluster.remote.connect: false
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: wifimon-kibana.example.org
discovery.seed_hosts: [
    "wifimon-node1.example.org",
    "wifimon-node2.example.org",
    "wifimon-node3.example.org"
]
xpack.security.enabled: true
xpack.security.http.ssl.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.http.ssl.key: /etc/elasticsearch/certs/kibana.key
xpack.security.http.ssl.certificate: /etc/elasticsearch/certs/kibana.crt
xpack.security.http.ssl.certificate_authorities: /etc/elasticsearch/certs/ca.crt
xpack.security.transport.ssl.key: /etc/elasticsearch/certs/kibana.key
xpack.security.transport.ssl.certificate: /etc/elasticsearch/certs/kibana.crt
xpack.security.transport.ssl.certificate_authorities:
 /etc/elasticsearch/certs/ca.crt
xpack.monitoring.enabled: true
xpack.monitoring.collection.enabled: true

Setup Passwords

Elasticsearch comes with built-in users configured, each of them having a set of privileges but with their passwords not set, and consequently unable to be used for authentication.

...

For more information on Built-in users follow the link.

Kibana Platform

Kibana is a browser-based interface that allows for searching, viewing, and interacting with the data stored in the cluster. It’s a visualization platform for creating charts, tables, and maps to represent the data. Kibana should be configured in an Elasticsearch node. The configuration of Kibana is done by editing the /etc/kibana/kibana.yml file.

...

For more information on Kibana configuration settings, see Configuring Kibana.

Anchor
cluster_exploration cluster_exploration

Cluster Exploration

Even though it is possible to explore the cluster by using the Kibana platform, this section is about querying the cluster by using the REST API provided by Elasticsearch. The querying commands are executed in wifimon-kibana.example.org node and the user elastic is used for authentication.

...

curl -XGET --cacert /etc/elasticsearch/certs/ca.crt --user elastic 'https://wifimon-kibana.example.org:9200/_cat/health?v'

...

/wifimon-kibana.example.org:9200/_cat/health?v'

Our cluster is of green status, but this will change to yellow after stopping the elasticsearch instance in the master node, which was intentionally chosen in order to see the election of the new master.

...

Querying the cluster again from wifimon-kibana.example.org node shows that the wifimon-node1.example.org has gone, and the wifimon-node3.example.org has been elected as the new master. The cluster status is now yellow.

Start the elasticsearch instance on wifimon-node1.example.org node and query the cluster again. The wifimon-node1.example.org will join the cluster and the status of the cluster will become green, while wifimon-node3.example.org continues to be the master node.

Filebeat Configuration

Filebeat monitors log files for new content, collect collects log events, and forwards them to Elasticsearch, either directly or via Logstash. In Filebeat terms one speaks about a) the input which looks in the configured log data locations, b) the harvester which reads a single log for new content and sends new log data to libbeat, and c) the output which aggregates and sends data to the configured output. For more information see Filebeat overview.

...

Below are the sample log files to be used in tests. It's about a log event when of a user interact interacting with the eduroam Eduroam Service Provider and another one while interacting with the dhcp DHCP server.

Sun Mar 10 08:16:05 2019
    Service-Type = Framed-User
    NAS-Port-Id = "wlan2"
    NAS-Port-Type = Wireless-802.11
    User-Name = "username@example.org"
    Acct-Session-Id = "82c000cd"
    Acct-Multi-Session-Id = "CC-2D-E0-9A-EB-A3-88-75-98-6C-31-AA-82-C0-00-00-00-00-00-CD"
    Calling-Station-Id = "88-75-98-6C-31-AA"
    Called-Station-Id = "CC-2D-E0-9A-EB-A3:eduroam"
    Acct-Authentic = RADIUS
    Acct-Status-Type = Start
    NAS-Identifier = "Eduroam"
    Acct-Delay-Time = 0
    NAS-IP-Address = 192.168.192.111
    Event-Timestamp = "Mar 8 2019 08:16:05 CET"
    Tmp-String-9 = "ai:"
    Acct-Unique-Session-Id = "e5450a4e16d951436a7c241eaf788f9b"
    Realm = "example.org"
    Timestamp = 1552029365

...

Jun 18 19:15:20 centos dhcpd[11223]: DHCPREQUEST for 192.168.1.200 from a4:c4:94:cd:35:70 (galliumos) via wlp6s0
Jun 18 19:15:20 centos dhcpd[11223]: DHCPACK on 192.168.1.200 to a4:c4:94:cd:35:70 (galliumos) via wlp6s0

File Output

As mentioned above, Filebeat will be firstly configured to dump the output in a file. Below is shown the configuration file of Filebeat for each agent. It configures an input of type log, which is needed to read lines from log files. There's also the output which configures the path and the filename to dump the data in, and finally the section of processors to drop some fields Filebeat adds by default, and to add the logtype field used in the Logstash beats-pipeline output.

RADIUS Server

The following is the Filebeat configuration on the radius RADIUS server, which dumps the data in the /tmp/sample_logs_output.json file.

...

The important settings here are the multiline.* ones which manage multiline formatted logs. The .pattern matches lines starting with white-space. The .negate and .match work together, and combined as false and after make consecutive lines that match the pattern to be appended to the previous line that doesn't match it. This makes all the lines starting with whiteswhite-pace space to be appended to the line that hold the date, actually the first line in the radius_sample_logs. For more information see Manage multiline messages.

...

{"@timestamp":"2020-06-28T13:07:37.183Z","@metadata":{"beat":"filebeat","type":"_doc","version":"7.8.0"},"logtype":"radius","message":"Sun Mar 10 08:16:05 2019\n\tService-Type = Framed-User\n\tNAS-Port-Id = \"wlan2\"\n\tNAS-Port-Type = Wireless-802.11\n\tUser-Name = \"username@example.org\"\n\tAcct-SessionId = \"82c000cd\"\n\tAcct-Multi-Session-Id = \"CC-2D-E0-9A-EB-A3-88-75-98-6C-31-AA82-C0-00-00-00-00-00-CD\"\n\tCalling-Station-Id = \"88-75-98-6C-31-AA\"\n\tCalledStation-Id = \"CC-2D-E0-9A-EB-A3:eduroam\"\n\tAcct-Authentic = RADIUS\n\tAcctStatus-Type = Start\n\tNAS-Identifier = \"Eduroam\"\n\tAcct-Delay-Time = 0\n\tNASIPtNAS-IP-Address = 192.168.0192.22111\n\tEvent-Timestamp = \"Mar 8 2019 08:16:05 CET\"\n\tTmpString-9 = \"ai:\"\n\tAcct-Unique-Session-Id = \"e5450a4e16d951436a7c241eaf788f9b\"\n\tRealm = \"example.org\"\n\tTimestamp = 1552029365"}

The logs are located in the message field. The logtype field holds the radius value, thus differentiating these events from the dhcp ones when arriving at Logstash pipeline.

DHCP Server

The following is the Filebeat configuration on the dhcp DHCP server, which dumps the data in the /tmp/sample_logs_output.json file.

...

The lines to include from dhcp DHCP logs are the ones containing DHCPACK string, which represent the final phase of dhcp DHCP operations. These lines are filtered with the include_lines setting.

...

The logtype field contains the dhcp value, thus differentiating these events from the radius ones, when arriving at Logstash pipeline.

Filtering Log Events

Apart from adding or dropping named fields, processors can also be used to filter log events when certain criteria are met. For example, to send out only the log events containing the value Eduroam in the NAS-Identifyer field, the following configuration could be applied.

...

For more information on configuring processors see Filter and enhance the exported data.

Logstash Output

This section shows how to configure Filebeat’s logstash output to feed the pipeline node.

...

The hosts setting specifies node and port where Logstash service listens for incoming log events. The ${key_passphrase} references the passphrase of filebeat.key stored in Filebeat keystore -- it's about mutual SSL/TLS authentication, the client (filebeatFilebeat) is forced to provide a certificate to the server (logstashLogstash) for or the connection to won't be established.

For this configuration to work, the Elasticsearch index template must be manually loaded. Template autoloading is only supported for the elasticsearch output. Replace elastic-password-goes-here with the proper password and run:

...

The above command loads the template from wifimon-kibana.example.org node where elasticsearch is installed. Detailed information is written in the Filebeat log file.

Monitoring

The Kibana platform allows for monitoring the health of Filebeat service. For this to happen, the following configuration must be added in the /etc/filebeat/filebeat.yml file.

...

The ${beats_system_password} references the password of the beats_system built-in user which is stored in Filebeat keystore.

Logstash Configuration

Logstash is a data collection engine with real-time pipelining capabilities. A Logstash pipeline consists of three elements, the input, filter, and output. The input plugins consume data coming from a source, the filter plugins modify the data as specified, and the output plugins send data to a defined destination. In this setup data comes from Filebeat agents, with their logstash output configured to feed the Logstash instance on port 5044/tcp.

JVM Options

The JVM Options for Logstash are defined in the /etc/logstash/jvm.options file. The configuration is the same as the one configuring the JVM Options of Elasticsearch.

Logstash Settings

Logstash settings are defined in the /etc/logstash/logstash.yml file, which contains the following:

path.data: /var/lib/logstash
path.logs: /var/log/logstash
queue.type: persisted
xpack.monitoring.enabled: true: persisted
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.username: "logstash_system"
xpack.monitoring.elasticsearch.password: "${logstash_system_password}"
xpack.monitoring.elasticsearch.hosts: "https://wifimon-kibana.example.org:9200"
xpack.monitoring.elasticsearch.ssl.usernamecertificate_authority: "logstash_system/etc/logstash/certs/ca.crt"
xpack.monitoring.elasticsearch.ssl.passwordverification_mode: "${logstash_system_password}"certificate
xpack.monitoring.elasticsearch.hosts: "https://wifimon-kibana.example.org:9200"
xpack.monitoring.elasticsearch.ssl.certificate_authority: "/etc/logstash/certs/ca.crt"
xpack.monitoring.elasticsearch.ssl.verification_mode: certificate
xpack.monitoring.elasticsearch.sniffing: true

The path.data and path.logs defines the directories logstash will use to write persistent data and log messages, respectively.

The queue.type set the queue to persisted to provide protection against data loss by using an on-disk queue. For more information see Persistent Queues.

...

.sniffing: true

The path.data and path.logs defines the directories logstash will use to write persistent data and log messages, respectively.

The queue.type set the queue to persisted, to provide protection against data loss by using an on-disk queue. For more information see Persistent Queues.

The other settings configure Logstash to send monitoring data over SSL/TLS encryption.

Logstash Pipelines

Logstash pipelines are defined in the /etc/logstash/pipelines.yml file, which contains:

...

For each pipeline, an id and the configuration file is defined. The beats-pipeline functions as a gate receiving logs from both (radius and dhcp) streams, and then forwarding these logs to the proper pipeline.

Anchor
beats_pipeline beats_pipeline

Beats Pipeline

As already mentioned above, the beats-pipeline acts as receiver / forwarder of log-events coming from radius RADIUS and dhcp DHCP streams. It doesn’t configure any filter element, but the input and output ones.

...

The beats plugin configures Logstash to listen on port 5044. It also provides settings for SSL/TLS encryption and forces the peer (filebeatFilebeat) to provide a certificate for identification. The output defines which pipeline to forward the data to, based on the value of logtype field sent from filebeat Filebeat agent.

RADIUS Pipeline

The radius-pipeline is configured in the /etc/logstash/conf.d/radius-pipeline.conf file. It receives radius RADIUS log-events sent from the beats-pipeline.

...

The output defines the stdout plugin which dumps the filtered data in the standard output, allowing for testing a data flow of Filebeat → Logstash → Logstash_STDOUT.

DHCP Pipeline

The dhcp-pipeline is configured in the /etc/logstash/conf.d/dhcp-pipeline.conf file. It receives dhcp DHCP log-events sent from the beats-pipeline.

...

The output defines the stdout plugin which dumps the filtered data in the standard output, allowing for testing a data flow of Filebeat → Logstash → Logstash_STDOUT.

Streaming to STDOUT

Having Filebeat agents configured to feed Logstash, whose pipelines are configured to dump data to STDOUT, makes it possible to test a data flowing through Filebeat → Logstash → Logstash_STDOUT.

...

Set the journal to follow recently appended entries for logstash service:

journalctl --follow --unit logstash.service

On radius RADIUS server run the test_filebeat.sh script as root user.

...

{
"Called-Station-Id" => "CC-2D-E0-9A-EB-A3:eduroam",
"Acct-Status-Type" => "Start",
"NAS-IP-Address" => "162.13.218.132",
"@timestamp" => 2019-12-10T17:35:38.054Z,
"Calling-Station-Id" => "389c0235f65590b4c80c8b6be576abb6a66c89e5331b43c65cdfa4f66f9463cc374d46eeb88041624ccdd258814d80a9aeb16494e4c60b16ae82896aab703ec8",
"Timestamp" => "1552029365",
"geoip" => {
"country_code3" => "GB",
"ip" => "162.13.218.132",
"timezone" => "Europe/London",
"country_code2" => "GB",
"continent_code" => "EU",
"latitude" => 51.4964,
"country_name" => "United Kingdom",
"location" => {
"lat" => 51.4964,
"lon" => -0.1224
},
"longitude" => -0.1224
},
"tags" => []
}

On dhcp DHCP server run the test_filebeat.sh script as root user.

...

You may have noticed in the output of radius-pipeline that the value of NAS-IP-Address have been changed from private IP to 162.13.218.132 (www.geant.org). This was done intentionally in order to see the results of geoip filter, which gives nothing for private IPs.

Streaming Logs Into Cluster

Until now the streaming of data has been triggered manually by using the sample data. This allowed for testing the configuration of Filebeat and Logstash, and also having a first view of results.

This section is about configuring the components pointing to real data files and implement a streaming through the path Filebeat → Logstash → Elasticsearch.

Filebeat Inputs

In the /etc/filebeat/filebeat.yml file under the filebeat.inputs, the paths should now point to the full path in the filesystem where the RADIUS or the DHCP logs are located.

...

Multiple files can be given to paths setting as a list or as a glob-based pattern.

Create User and Role

In order to send log events to the cluster, the user logstash_writer with the role logstash_writer_role must be created. The role assigns the cluster permissions of monitor and manage_index_templates, and privileges of write and create_index for radiuslogs and dhcplogs indices. Granted with these permissions, the logstash_writer user is able to write data into the index.

...

set +o history
curl -X POST --cacert /etc/elasticsearch/certs/ca.crt --user elastic 'https://wifimon-kibana.example.org:9200/_security/user/logstash_writer?pretty' -H 'Content-Type: application/json' -d'
{
  "username": "logstash_writer",
  "roles": ["logstash_writer_role"],
  "full_name": null,
  "email": null,
  "password": "some-password-goes-here",
  "enabled": true
}
'
set -o history

Logstash Output

On radius-pipeline and dhcp-pipeline configuration files, the output should be configured to send data to Elasticsearch cluster. This is done by configuring the Logstash output elasticsearch plugin.

...

curl -XGET --cacert /etc/elasticsearch/certs/ca.crt --user elastic 'https://wifimon-kibana.example.org:9200/_cat/indices/radiuslogs?v'
curl -XGET --cacert /etc/elasticsearch/certs/ca.crt --user elastic 'https://wifimon-kibana.example.org:9200/_cat/indices/dhcplogs?v'

ILM Configuration

The intention of WiFiMon is not to keep the logs forever, they are only needed for a limited period of time. New log events keep coming so, after that time period has passed, the old logs should be deleted.

Logs are stored in the radiuslogs and dhcplogs indices. The index lifecycle management is achieved by creating and applying ILM policies, which can trigger actions upon indexes based on certain criteria. More information about ILM can be found at ILM Overview page.

Create Policy

This setup is about deleting the index when it’s one day old. Run the following command in the wifimon-kibana.example.org node to create the wifimon_policy policy.

...

curl -XGET --cacert /etc/elasticsearch/certs/ca.crt --user elastic "https://wifimon-kibana.example.org:9200/_ilm/policy/wifimon_policy?pretty"

Apply Policy

The policy must be associated with the indexes upon which it will trigger the configured actions. For this to happen the policy must be configured in the index template used to create the index.

...

curl -XGET --cacert /etc/elasticsearch/certs/ca.crt --user elastic "https://wifimon-kibana.example.org:9200/_template/wifimon_template?pretty"

Logstash Output

The Logstash elasticsearch output plugin provides settings to control the Index Lifecycle Management. Include the ILM settings on radius-pipeline and dhcp-pipeline configuration files, so that the elasticsearch output plugin becomes:

...

Restart the logstash service to apply the new settings.

Keystores

In order not to have sensitive information hardcoded in the configuration files and just protecting that information with filesystem permissions, it is recommended to make use of keystores provided by the Elasticsearch components.

Anchor
elasticsearch_keystore elasticsearch_keystore

Elasticsearch

To configure Elasticsearch keystore run the following commands on each cluster node.

...

/usr/share/elasticsearch/bin/elasticsearch-keystore list
keystore.seed
xpack.security.http.ssl.secure_key_passphrase
xpack.security.transport.ssl.secure_key_passphrase

Anchor
kibana_keystore kibana_keystore

Kibana

To configure Kibana keystore run the following commands on wifimon-kibana.example.org node.

...

sudo -u kibana /usr/share/kibana/bin/kibana-keystore list
server.ssl.keyPassphrase
elasticsearch.username
elasticsearch.password

Anchor
logstash_keystore logstash_keystore

Logstash

To configure Logstash keystore run the following commands on wifimon-logstash.example.org node.

...

/usr/share/logstash/bin/logstash-keystore --path.settings /etc/logstash/ list
fingerprint_key
logstash_system_password
logstash_writer_password
pkcs8_key_passphrase

Anchor
filebeat_keystore filebeat_keystore

Filebeat

To configure Filebeat keystore run the following commands on the freeRadius server servers where Filebeat is installed.

...

filebeat keystore list
beats_system_password
key_passphrase

References

The following links were very useful while writing this material and performing the tests mentioned in it.

...