...
These four scenarios outline diverse approaches to SAML SP testing, each tailored to its respective context and purpose and requiring a different type of deployment.
SELF - Self-testing by SP for production readiness
Summary description
Fully internal.
This scenario enables individual Service Providers (SPs) to internally validate their SAML service configuration, focusing on signature usage.
Deployment or configuration
SPs perform this self-testing independently within their organisation.
The SP deploys a test ISP, preferably as an easily configurable VM image, container image, or appliance.
Configuration of Deploy a test ISP and configure the tested SP for it .
Relational characteristics
Policy/
Deployment or configuration
!!
Arrangement and execution of tests
!!
Presentation and analysis of test results
!!
Relational or contractual arrangements
!!
includes...
Arrangement and execution of tests
Testing is initiated by a service admin or operator, triggered through command-line invocation.
The target SP for testing is specified via a command-line parameter.
Testing can occur after the service is deployed but before its production use is declared/announced, after configuration changes, or periodically via automated scheduling tools like cron.
The testing tool allows selective execution or suppression of individual tests through command-line options.
Presentation and analysis of test results
Test output verbosity can be configured using a switch.
Results are presented in plain text, offering summary and detailed formats of information about the outcome of individual tests, detected issues, and exchanged content.
Status information, issues related to SP operation, and details of both successful and failed tests are reported to standard output (stdout). Issues in the execution of the command are reported to standard error (stderr).
However, problems in both command execution and SP operation are indicated by non-zero exit statuses, facilitating use in scripts.
Relational or contractual arrangements
No formal arrangements are required as both the tester and SP belong to the same organisation.
ONBOARDING - Testing of SP deployment by FedOps during onboarding
Summary description
Options
...
This scenario is applicable during SP onboarding and may involve manual or automated testing. It is initiated upon the SP's request
...
It probably needs to be and integrated into the federation's policy and operational guidelines. However, it can be easily communicated among other requirements after the SP requests onboarding.onboarding procedure of the federation.
Deployment or configuration
The testing tool is deployed by the federation.
Details of SP configuration should be specified in onboarding guidelines.!!
Arrangement and execution of tests
Initiated upon request by the SP during onboarding.
Automation is possible as part of the onboarding process.
Specific details on conducted tests are outlined in the onboarding procedure, and this information can be communicated to SPs requesting onboarding. They may be accompanied by corresponding SP configuration guidelines that would increase the SP's chances of passing the tests. Optionally, the SP may be informed about the requirements and tests and be requested to give explicit approval and clearance for tests to be conducted.!!
Presentation and analysis of test results
For admin, for SP, web UI, email notification with access link, ...!!
Specifics regarding the presentation and analysis of test results are not provided but should be detailed in the onboarding guidelines.
Relational or contractual arrangements
!!
Periodic testing of SP deployments by FedOps
Summary description
Options
- Triggered by SPs themselves, with each SP required to invoke it in regular intervals within policy-defined periods.
- Or it could be automatically invoked by FedOps in line with predefined rules.
The testing must probably be integrated into the federation's policy and operational guidelines.
Testing may be mandated as part of the onboarding criteria when requested by SP organisations when it should be communicated as a requirement within the onboarding procedure. However, this testing will be better accepted if it is mentioned and outlined in the description of the onboarding procedure that must be read by SP organisations wishing to join the federation.
The testing process should be allowed/sanctioned into Must be aligned with the federation's policy and operational A part of guidelines.
PERIODIC - Periodic testing of SP deployments by FedOps
Summary description
Periodic testing is conducted by federation operators in pre-defined intervals aligned with the federation's policy and operational rules, ensuring ongoing compliance.
Deployment or configuration
Similar to ONBOARDING.
Please state the key differences!!
Arrangement and execution of tests
Testing execution must be aligned with the federation's policy and operating rules such as...
Tests across SPs may be spread in time and conducted during predefined high-load or low-load periods.!!
Presentation and analysis of test results
...
Relational or contractual arrangements
!!
The testing process should be allowed or mandated by the federation's policy and operating guidelines.
COMPLIANCE - Client institution testing for compliance
Summary description
Conducted by a client institution for contracted services, possibly as part of its internal compliance reviews (e.g., GDPR audits, This scenario involves the SP's client institution conducting compliance testing, often as part of broader assessments like GDPR audits or ISO 27001 security controls).
How is the practical arrangement of the test coordinated between the client institution and the SP?
Practical differences and requirements in comparison to self-testing:
...
. It often integrates with broader compliance assessments, introducing additional requirements and may involve specific compliance criteria dictated by the client institution.
...
Deployment or configuration
To best simulate the regular service usage, the testing platform can be deployed by the client organisation. However, it may also be provided by a third party specialised in compliance audits.
More!!
Arrangement and execution of tests
It is conducted by an individual client organisation to internally validate the validity of contracted SP's SAML service configuration for compliance, by internal or external auditors operating with both the organisation's and SP's approval and SP's support, if needed. However, this testing is usually done without direct involvement of the SP.
How the practical execution of tests and debugging are coordinated and conducted between the client institution and the SP is out of the scope of this description, as the SP does not have to do or deploy anything that would specifically support this testing scenario.
Presentation and analysis of test results
The use of the test by the client institution may necessitate specialised procedures and reporting. Please suggest some SLA-related metrics that would need to be supported in the generated reports!!
Does report signing need to be supported!!?
Relational or contractual arrangements
Compliance testing, as part of a broader compliance review, is likely to be included in the contractual arrangements between the client institution and the SP, possibly within the Service Level Agreements (SLAs) between the client institution and the SP.
It probably needs to be included in the SLA.
Deployment or configuration
!!
Arrangement and execution of tests
!!
Presentation and analysis of test results
!!
Relational or contractual arrangements
!!
Things/tests to look at
- https://release-check.edugain.org/
- https://access-check.edugain.org/step1
- https://medium.com/the-new-control-plane/i-need-a-saml-idp-to-test-now-477761595b60
- https://saml.oktadev.com/
- https://auth0.com/docs/authenticate/protocols/saml/saml-configuration/configure-auth0-as-service-and-identity-provider
- https://samltest.id/start-sp-test/
- https://jumpcloud.com/blog/how-to-test-saml-and-configure-sso-for-free
- https://www.samltool.com/
- https://mocksaml.com/