...

For more information on Kibana configuration settings, see Configuring Kibana.

Cluster Exploration

...

curl -XGET --cacert /etc/elasticsearch/certs/ca.crt --user elastic 'https://wifimon-kibana.example.org:9200/_cat/health?v'

...

Our cluster is of green status, but this will change to yellow after stopping the elasticsearch instance in the master node, which was intentionally chosen in order to see the election of the new master.

...

Querying the cluster again from wifimon-kibana.example.org node shows that the wifimon-node1.example.org has gone, and the wifimon-node3.example.org has been elected as the new master. The cluster status is now yellow.

...

Filebeat monitors log files for new content, collect collects log events, and forwards them to Elasticsearch, either directly or via Logstash. In Filebeat terms one speaks about a) the input which looks in the configured log data locations, b) the harvester which reads a single log for new content and sends new log data to libbeat, and c) the output which aggregates and sends data to the configured output. For more information see Filebeat overview.

...

Below are the sample log files to be used in tests. It's about a log event when of a user interact interacting with the eduroam Eduroam Service Provider and another one while interacting with the dhcp DHCP server.

Sun Mar 10 08:16:05 2019
    Service-Type = Framed-User
    NAS-Port-Id = "wlan2"
    NAS-Port-Type = Wireless-802.11
    User-Name = "username@example.org"
    Acct-Session-Id = "82c000cd"
    Acct-Multi-Session-Id = "CC-2D-E0-9A-EB-A3-88-75-98-6C-31-AA-82-C0-00-00-00-00-00-CD"
    Calling-Station-Id = "88-75-98-6C-31-AA"
    Called-Station-Id = "CC-2D-E0-9A-EB-A3:eduroam"
    Acct-Authentic = RADIUS
    Acct-Status-Type = Start
    NAS-Identifier = "Eduroam"
    Acct-Delay-Time = 0
    NAS-IP-Address = 192.168.192.111
    Event-Timestamp = "Mar 8 2019 08:16:05 CET"
    Tmp-String-9 = "ai:"
    Acct-Unique-Session-Id = "e5450a4e16d951436a7c241eaf788f9b"
    Realm = "example.org"
    Timestamp = 1552029365

...

The following is the Filebeat configuration on the radius RADIUS server, which dumps the data in the /tmp/sample_logs_output.json file.

...

The following is the Filebeat configuration on the dhcp DHCP server, which dumps the data in the /tmp/sample_logs_output.json file.

...

The lines to include from dhcp DHCP logs are the ones containing DHCPACK string, which represent the final phase of dhcp DHCP operations. These lines are filtered with the include_lines setting.

...

The hosts setting specifies node and port where Logstash service listens for incoming log events. The ${key_passphrase} references the passphrase of filebeat.key stored in Filebeat keystore -- it's about mutual SSL/TLS authentication, the client (filebeatFilebeat) is forced to provide a certificate to the server (logstashLogstash) for or the connection to won't be established.

For this configuration to work, the Elasticsearch index template must be manually loaded. Template autoloading is only supported for the elasticsearch output. Replace elastic-password-goes-here with the proper password and run:

...

Logstash is a data collection engine with real-time pipelining capabilities. A Logstash pipeline consists of three elements, the input, filter, and output. The input plugins consume data coming from a source, the filter plugins modify the data as specified, and the output plugins send data to a defined destination. In this setup data comes from Filebeat agents, with their logstash output configured to feed the Logstash instance on port 5044/tcp.

...

The queue.type set the queue to persisted, to provide protection against data loss by using an on-disk queue. For more information see Persistent Queues.

The other settings configures configure Logstash to send monitoring data over SSL/TLS encryption.

...

Anchor
beats_pipeline beats_pipeline

Beats Pipeline

As already mentioned above, the beats-pipeline acts as receiver / forwarder of log-events coming from radius RADIUS and dhcp DHCP streams. It doesn’t configure any filter element, but the input and output ones.

...

The beats plugin configures Logstash to listen on port 5044. It also provides settings for SSL/TLS encryption and forces the peer (filebeatFilebeat) to provide a certificate for identification. The output defines which pipeline to forward the data to, based on the value of logtype field sent from filebeat Filebeat agent.

RADIUS Pipeline

The radius-pipeline is configured in the /etc/logstash/conf.d/radius-pipeline.conf file. It receives radius RADIUS log-events sent from the beats-pipeline.

...

The dhcp-pipeline is configured in the /etc/logstash/conf.d/dhcp-pipeline.conf file. It receives dhcp DHCP log-events sent from the beats-pipeline.

...

Set the journal to follow recently appended entries for logstash service:

journalctl --follow --unit logstash.service

On radius RADIUS server run the test_filebeat.sh script as root user.

...

{
"Called-Station-Id" => "CC-2D-E0-9A-EB-A3:eduroam",
"Acct-Status-Type" => "Start",
"NAS-IP-Address" => "162.13.218.132",
"@timestamp" => 2019-12-10T17:35:38.054Z,
"Calling-Station-Id" => "389c0235f65590b4c80c8b6be576abb6a66c89e5331b43c65cdfa4f66f9463cc374d46eeb88041624ccdd258814d80a9aeb16494e4c60b16ae82896aab703ec8",
"Timestamp" => "1552029365",
"geoip" => {
"country_code3" => "GB",
"ip" => "162.13.218.132",
"timezone" => "Europe/London",
"country_code2" => "GB",
"continent_code" => "EU",
"latitude" => 51.4964,
"country_name" => "United Kingdom",
"location" => {
"lat" => 51.4964,
"lon" => -0.1224
},
"longitude" => -0.1224
},
"tags" => []
}

On dhcp DHCP server run the test_filebeat.sh script as root user.

...

In order to send log events to the cluster, the user logstash_writer with the role logstash_writer_role must be created. The role assigns the cluster permissions of monitor and manage_index_templates, and privileges of write and create_index for radiuslogs and dhcplogs indices. Granted with these permissions, the logstash_writer user is able to write data into the index.

...

To configure Filebeat keystore run the following commands on the freeRadius server servers where Filebeat is installed.

...