Installation and Configuration

The generic installation and configuration instructions are publicly available at https://github.com/GEANT/CAT/blob/master/tutorials/Configuration.md

After following all these, some amount of fine-tuning in the config files is needed. Most items are self-explanatory; specific documentation to be added here for neuralgic spots.

The development team will provide the initial and production-ready product configuration. The crucial parts that differentiate a normal CAT from Managed IdP deployment are:

• config-master.php → FUNCTIONALITY_LOCATIONS → CONFASSISTANT_RADIUS = "NONE"
• config-master.php → FUNCTIONALITY_LOCATIONS → CONFASSISTANT_SILVERBULLET = "LOCAL"
• generate and install all the per-NRO server cert CAs in config/SilverbulletServerCerts/* (GitHub scripts available, will be executed by dev team)
• generate and install the client cert issuing CA and corresponding (unprotected) private key in config/SilverbulletClientCerts/real.key and real.pem (see next section)

eduroam Managed IdP Client Certificate Root CA

It is crucial to have a trust anchor for all issued client certificates which is stable on the long-term. To that end, an offline hardware-backed CA is provisioned and kept in a physically safe position in GEANT property. The CA itself is created with the CA generation script publicly available on GitHub.

CA operations are performed on the project-procured Raspberry Pi 3. The Pi needs the following preparatory actions:

• install Raspian Stretch (or higher); required for having openssl 1.1+
• install the package rng-tools (provides access to the built-in hardware random number generator under /dev/hwrng)
• set the date and time (Raspberry Pi does not have a built-in clock)
• after installing all needed packages, remove the Pi from the network and never connect it again.

The script

CA.bootstrapNewRootCA

will generate TWO CAs, one with RSA/4096 bit keys, one with ECDSA/NIST P-521 keys. The latter one is future-proofing.

Afterwards, edit again settings/openssl-rsa.cnf  and settings/openssl-ecdsa.cnf settings with new URLs for the intermediate (Issuing) CA.

Subsequently, issue the command

CA.generateNewIntermediateCA

Immediately after creation, create a new CRL (to assert that there are no revoked certificates at this point in time) and a new OCSP statement for the newly created intermediates:

CA.newCRL

CA.newOCSPStatementForSerial_RSA 0x<serial number in hex of the new RSA intermediate certificate>

CA.newOCSPStatementForSerial_ECDSA 0x<serial number in hex of the new ECDSA intermediate certificate>

The result of this set of commands are the files needed for CA operation:

All of these files, but no others, are copied out of the CA environment for further use in operations (e.g. onto a USB stick).

If you ever need to revoke an intermediate, the corresponding scripts can be used (one variant for RSA, one for ECDSA, both to be called with the correspodning serial number of the certificate).

eduroam Managed IdP Server Certificate and CA set

eduroam installers will need to be configured with a server certificate trust (i.e. a root CA and a server name). To support the partitionability of the RADIUS service, each eduroam NRO gets its own self-signed root. This means approx. 200 self-signed CA certificates and server certificates need to be provisioned, all served by the RADIUS servers. The code to generate both the CA hierarchy and the FreeRADIUS configuration snippets to activate all those distinct personalities is available on GitHub.

The script will be executed by the dev team during initial installation, directly on one of the RADIUS servers (auth-1.hosted.eduroam.org) so that the server certificate private keys are immediately on the right host and need no copying.

The instructions to follow are in the Git repo.

The entire process will take a LONG while to complete.

Copy the CA certificates (without private key) to the web interface.

Copy the server certificates, the private keys and the FreeRADIUS config snippets to the other RADIUS servers in the cluster (auth-2.hosted.eduroam.org).

Store the CA certificate private key set offline in a safe place.

Important: the CA certificates need to contain a valid URL for their CRL Distribution Point. The CRLDP is set by the addnro.py script as: CRLDP.0=http://ocsp.hosted.eduroam.org/rsa/server/<NRO>/crl/root.crl (where <NRO> is the ccTLD of the NRO in question, in capitalised letters - e.g. "PL")

Service Operation

Web Service (hosted.eduroam.org)

Main Services:

• Apache2
• MySQL / MariaDB
• CAT PHP application

Logs:

• /var/log/CAT/*
• /var/log/apache2/*

RADIUS Servers (auth-1/2.hosted.eduroam.org)

Main Services:

• FreeRADIUS 3

Logs:

• /opt/tls/var/log/radius/*

OCSP Responder (ocsp.hosted.eduroam.org)

Main Services:

• Apache2
• PHP script for OCSP responses (contained in CAT distribution, utils/ocsp_web/*)

Logs:

• /var/log/apache2/*

Statistics for KPIs

The service statistics are collected with simple SQL queries from several databases.

Number of NROs and IdPs in the system

To be executed on the database "managed_idp" on hosted.eduroam.org. The data is cumulative since start of technical setup of the hosts.

SELECT COUNT(distinct p.inst_id) AS active_inst, i.country AS federation
   FROM silverbullet_user su LEFT JOIN profile p ON su.profile_id = p.profile_id LEFT JOIN institution i ON p.inst_id = i.inst_id 
   WHERE su.deactivation_status = "ACTIVE" 
   GROUP BY federation 
   ORDER BY active_inst DESC;

Example output at service launch day (20 Mar 2019):

+-------------+------------+
| active_inst | federation |
+-------------+------------+
|           6 | LU         |
|           5 | PL         |
|           1 | AM         |
|           1 | CA         |
|           1 | ES         |
|           1 | JP         |
+-------------+------------+

Certificate count (issued/revoked/expired)

To be executed on the database "managed_idp" on hosted.eduroam.org. The data is cumulative since start of technical setup of the hosts.

Total issued certificates

SELECT count(*) AS certcount, ucase(substr(substr(cn,locate('.',cn)+1),1, 2)) AS userfed 
   FROM silverbullet_certificate 
   GROUP BY userfed 
   ORDER BY certcount DESC;

Example output at service launch day:

+-----------+---------+
| certcount | userfed |
+-----------+---------+
|       125 | PL      |
|        60 | LU      |
|        20 | NO      |
|        13 | AM      |
|         4 | JP      |
|         2 | CA      |
|         1 | UA      |
+-----------+---------+

Total revoked certificates

SELECT count(*) AS certcount, ucase(substr(substr(cn,locate('.',cn)+1),1, 2)) AS userfed 
   FROM silverbullet_certificate 
   WHERE revocation_status = "REVOKED" 
   GROUP BY userfed 
   ORDER BY certcount DESC;

Example output at service launch day:

+-----------+---------+
| certcount | userfed |
+-----------+---------+
|        11 | AM      |
|         5 | NO      |
|         5 | PL      |
|         3 | LU      |
|         1 | JP      |
+-----------+---------+

Total expired certificates (certificates which were revoked before they expired are always counted under 'revoked', even after expiry)

SELECT count(*) AS certcount, ucase(substr(substr(cn,locate('.',cn)+1),1, 2)) AS userfed 
   FROM silverbullet_certificate 
   WHERE expiry < NOW() AND revocation_status = "NOT_REVOKED" 
   GROUP BY userfed 
   ORDER BY certcount DESC;

Example output at service launch day:

+-----------+---------+
| certcount | userfed |
+-----------+---------+
|        11 | PL      |
|         7 | LU      |
+-----------+---------+

Authentication count

To be executed on the database "eduroam" on auth-1.hosted.eduroam.org and auth-2.hosted.eduroam.org (each server maintains its own counters). These statistics can't be accumulative because data is deleted after 6 months. So, instead, the queries below return the counters for the last 24h. They should be run on exactly the same time of day every day to maintain full statistics coverage.

Successful authentications

SELECT COUNT(*) AS authcount, SUBSTR(username,LOCATE('@',username)+1) AS
realm FROM eduroamauth WHERE reply = "Access-Accept" AND authdate >=
TIMESTAMPADD(DAY, -1, NOW()) GROUP BY realm ORDER BY authcount DESC;

Example output at day after service launch:

+-----------+-----------------------------+
| authcount | realm                       |
+-----------+-----------------------------+
|       288 | 9-9.lu.hosted.eduroam.org   |
|        54 | 16-15.no.hosted.eduroam.org |
|        40 | 10-10.lu.hosted.eduroam.org |
|         6 | 11-19.lu.hosted.eduroam.org |
|         2 | 23-21.ua.hosted.eduroam.org |
|         1 | 20-20.jp.hosted.eduroam.org |
+-----------+-----------------------------+

Failed authentications

SELECT COUNT(*) AS authcount, SUBSTR(username,LOCATE('@',username)+1) AS
realm FROM eduroamauth WHERE reply = "Access-Reject" AND authdate >=
TIMESTAMPADD(DAY, -1, NOW()) GROUP BY realm ORDER BY authcount DESC;

Example output at day after service launch:

+-----------+-----------------------------+
| authcount | realm                       |
+-----------+-----------------------------+
|        25 | 16-15.no.hosted.eduroam.org |
+-----------+-----------------------------+

Interplay of the eduroam Managed IdP system components

eduroam Managed IdP includes multiple components which need to interwork correctly for the service as a whole to work. The following external dependencies between the components exist

eduroam Managed IdP web frontend → OCSP responder

• issues OCSP statements for each of the certificates known to the system, using a cron job. See documentation on GitHub above. Make sure the cron job is running and verify that updated statements end up in the correct directory on the OCSP responder.

eduroam Managed IdP web frontend → CAT code signing cluster

• web frontend creates installers for Windows, macOS and iOS which are to be digitally signed. The actual signature on the files is offloaded to the existing eduroam CAT code signing cluster (machines located in SURFnet premises). Make sure HTTPS traffic from the web frontend to the signing cluster is allowed.

eduroam Managed IdP RADIUS Server → OCSP responder

• makes request at OCSP responder during every user authentication. Make sure the HTTP communication between RADIUS server and OCSP Responder is possible.

Interplay of the eduroam Managed IdP system components

eduroam Managed IdP includes multiple components which need to interwork correctly for the service as a whole to work. The following external dependencies between the components exist

eduroam Managed IdP web frontend → OCSP responder

• issues OCSP statements for each of the certificates known to the system, using a cron job. See documentation on GitHub above. Make sure the cron job is running and verify that updated statements end up in the correct directory on the OCSP responder.

eduroam Managed IdP web frontend → CAT code signing cluster

• web frontend creates installers for Windows, macOS and iOS which are to be digitally signed. The actual signature on the files is offloaded to the existing eduroam CAT code signing cluster (machines located in SURFnet premises). Make sure HTTPS traffic from the web frontend to the signing cluster is allowed.

eduroam Managed IdP RADIUS Server → OCSP responder

• makes request at OCSP responder during every user authentication. Make sure the HTTP communication between RADIUS server and OCSP Responder is possible.SELECT COUNT(*) AS authcount, SUBSTR(username,LOCATE('@',username)+1) AS realm FROM eduroamauth WHERE reply = "Access-Accept" AND authdate >= TIMESTAMPADD(DAY, -1, NOW()) GROUP BY realm ORDER BY authcount DESC;+-----------+-----------------------------+ | authcount | realm | +-----------+-----------------------------+ | 288 | 9-9.lu.hosted.eduroam.org | | 54 | 16-15.no.hosted.eduroam.org | | 40 | 10-10.lu.hosted.eduroam.org | | 6 | 11-19.lu.hosted.eduroam.org | | 2 | 23-21.ua.hosted.eduroam.org | | 1 | 20-20.jp.hosted.eduroam.org | +-----------+-----------------------------+

...

This page and its child pages contains all documentation, deployment guides, instructions and manuals related to the service operations.

RESPONSIBLE: Information provided here is initially populated by the development team (during the transition phase), and revised based on the need or in a yearly service check by eduroam Managed IdP Service Manager.

...