Conformance Testing for OpenID Federation

This page describes how to run the conformance tests for OpenID Federation entities. At the time of writing, the set of tests currently available in production are in an early stage and mostly concerned with metadata validation. Do not hesitate to register issues in the conformance suite project if you’re observing unexpected or incorrect behavior, or if you have questions regarding the tests.

Test setup

First, go to https://www.certification.openid.net and log in with your Google or Gitlab account. Next, choose to Create a new test plan.

In the Test Plan dropdown menu, select OpenID Federation: Federation entity test (alpha – INCOMPLETE/INCORRECT, please email certification team if interested), and then set Server metadata location to discovery and Client Registration Type to static_client:

In the Configure Test that appears, configure the following:

Test information

alias

Used to make any URLs used in the test unique to the user. Set it to something unique to avoid clashes with other testers.

description

You can leave it empty

publish

Leave it to “No”. Test plans can be published and shared later.

Federation

entity_identifier

The entity identifier for the primary federation entity under test, which could be a leaf, an intermediate or a trust_anchor.

trust_anchor

The intended trust anchor for the entity specified by the entity_identifier. If the federation entity under test is a trust anchor, this field will be equal to the entity_identifier field.

trust_anchor_jwks

Pre-configured public keys for the entity’s trust anchor as a JWKSet, e.g. { "keys": [ { "kty": "RSA", ... } ] }.

Finally, click Create Test Plan.

Running the tests

In the test plan overview, you are presented with a list of individual tests. Click Run Test to execute each individual test.

The current set of tests are:

openid-federation-entity-configuration

The test will fetch and validate the given entity’s Entity Configuration, and then proceed to its Immediate Superiors as specified in authority_hints. For each Immediate Superior, its Entity Configuration is fetched and validated. Following that, the test will invoke the superior’s List endpoint to confirm the presence of the original entity, and finally use the Fetch endpoint to retrieve and validate the Subordinate Statement for the entity.

openid-federation-list-and-fetch

This test validates the List and Fetch endpoints provided in the entity’s federation_entity metadata. The test will call the List endpoint, followed by a request to the Fetch endpoint for each of its subordinates, followed by validation of the corresponding Subordinate statements.

openid-federation-ensure-fetch-with-invalid-sub-fails

This test verifies that the Fetch endpoint responds with JSON and an error when the sub parameter is invalid. The test is isolated to the provided entity and will not proceed to its superiors nor subordinates.

openid-federation-ensure-fetch-with-iss-as-sub-fails

This test verifies that the Fetch endpoint responds with JSON and an error when the sub parameter references the Entity Identifier of the Issuing Entity. The test is isolated to the provided entity and will not proceed to its superiors nor subordinates.

openid-federation-preconfigured-keys-match-trust-anchors-keys

This test starts at the given entity and follows the chain up to the trust anchor. When the trust anchor has been reached, the jwks specified in its Entity Configuration are compared to the trust_anchor_jwks keys specified in the test configuration.

openid-federation-compare-trust-chain-to-resolve

The test will attempt to construct a trust chain from the configured entity to the trust anchor, and then compare it to the trust chain obtained from the trust anchor’s Resolve endpoint.

Sharing the test results

In the test plan overview, click Publish everything to make the test results visible for anyone (and not just the logged in user).

Known issues

The test openid-federation-list-and-fetch is problematic in federations with a large number of entities; it may slow down, freeze or potentially fail to execute until the end. Consider not executing this test if your entity under test has a List endpoint with more than, say, 10 or 20 entities.

Getting help

Do not hesitate to register issues in the conformance suite project if you’re observing unexpected or incorrect behavior. You can also reach out to certification@oidf.org if you have questions regarding the tests.