This page describes how to run conformance tests and gather testing results for Financial-grade API (FAPI) Read/Write Profile OpenID Providers (OPs).
Currently only FAPI-RW (i.e. FAPI part 2) has a certification program. The parts of FAPI-RW that are not currently tested are described on our wiki: https://gitlab.com/openid/conformance-suite/wikis/OP-FAPI-RW-Test-Status
There are tests available for FAPI-R (i.e. part 1); these are not yet complete. It is not necessary to run FAPI-R tests when certifying for FAPI-RW.
If your server complies with the UK OpenBanking specification, you should run only the ‘FAPI-RW-ID2: Authorization server test’ and in the ‘variant’ menu select the appropriate ‘openbankinguk-‘ one(s) for the client authentication types supported. These allow you to demonstrate compliance with both FAPI-RW and the OpenBanking UK Profile of FAPI-RW. The ‘openbankinguk’ variants are a superset of the FAPI-RW-ID2 tests, with the major difference being that OB compliant authorization servers requiring pre-registration of intent using an OB specific API for an authorization to succeed.
There are also separate tests for FAPI-CIBA; please see the FAPI-CIBA conformance instructions if you have an implementation of CIBA you would like to test.
Register two clients in your authorisation server and the necessary keys/certificates (the keys and certificates used for each client must be different). Two clients are necessary as the suite tests various mixup attacks, for example, using a valid access token from one client with the TLS client certificate from the other client.
The redirect URLs for the clients must be:
ALIAS must be replaced with a string that is unique to you/your organisation, to avoid other conformance suite users affecting your testing.
Please include the
?dummy1=lorem&dummy2=ipsum string in the redirect url exactly as written. The conformance suite will use both redirect URLs.
The IP address the conformance suite will make outgoing calls from is available within the suite on the home page after login in. (The IP address may change from time to time as the conformance suite runs on a Kubernetes cluster, in practice we wouldn’t expect it to change more than once a year.)
Instructions for Running Tests
- Open https://www.certification.openid.net/.
- Login using a Google or GitLab account, or any Op that supports WebFinger.
- Ensure “Multi test plan” is selected. (If “Single test” is shown, toggle it to “Multi test plan”).
- Select the relevant test plan from the “Select a Test Plan” dropdown menu, one with “FAPI-RW-ID2: Authorisation server test” in the name
- Select the correct ‘variant’; this depends on your client authentication type(s) and whether your server follows the openbanking uk specification or not. For example, on a FAPI-RW compliant server that supports oauth-mtls client authentication you should select variant
mtls. For a server that complies with the openbanking uk specification, pick one of the variants with ‘openbankinguk’ in the name, depending which type of client authentication you support.
- DO NOT pick a test plan that has ‘client’ in the name. These are not applicable when testing an OpenID Provider.
- Fill in the configuration form (as per the guidance given in the form fields when empty). You can switch to the “JSON” tab to view/edit the configuration in the underlying JSON format. Changes to the form are automatically reflected in the JSON and vice versa. The JSON can be copied and saved locally to be pasted back in later. The server will automatically remember the most recent JSON you successfully created a test with.A “real world” example of a full working configuration is available here: https://gitlab.com/openid/conformance-suite/wikis/Authlete-Example-Configuration.
- Note that both client JWKS sets must contain an ‘alg’ field. For FAPI-RW this is most usually ‘PS256’, but ‘ES256’ is also permitted.
- A resource server URL must be provided. This must be a simple GET endpoint that returns JSON and is used to test certificate binding of access tokens etc. (Different in OpenBanking UK tests; see below.)
- institution_id: any value is okay and is sent in a x-fapi-financial-id header. (This is a hangover from FAPI-R-ID1 and is only retained due to an error in an OpenBanking specific specification.)
- Press “Launch Test Plan”. You will be taken to a list of all the test modules in the plan.
- Press “Run New Test” on the first test module.
- Please read the description of the test in the light blue box near the top of the log page – this may contain specific instructions, for example one test requires that the user rejects the authentication process.
- If applicable, click on the “Visit” button, and login to your OP; some tests require you to do this more than once. You will be either be:
- directed back to the test suite, in which case hit the link to return to the log page, or:
- shown an error by the AS. If the error is expected (i.e. is mentioned in the light blue summary box on the log page), you should screenshot it and upload it to the conformance suite on the log detail page.
- When the test has completed, press “Continue Plan” to start the next test, or “Return to Plan” to view your progress.
If you require support, please email email@example.com. If it relates to a test failure, please include a link to the relevant log-detail.html, or if using a local install the downloaded log file.
Once you have successfully completed testing, please follow the submission instructions to complete the certification process.
About the Conformance Suite
For more information, please see about the conformance suite.
‘Plain FAPI’ specific notes
For implementors not targeting the UK / Australian ecosystems (as mentioned below), the server is expected to support the acr value ‘urn:mace:incommon:iap:silver’. This is not required by the specs, but is required for the tests to run correctly.
app2app Specific Notes
The conformance suite can also be used to test mobile apps that implement ‘app2app’, as used widely in the UK OpenBanking ecosystem and described at Implementing App-to-App Authorisation in OAuth2/OpenID Connect.
Technically this is testing the web2app scenario, but there are no significant differences on the authorization server side between web2app and app2app.
Please note that app2app tests should be run on a tablet sized device; see issue 865.
OpenBanking UK Specific Notes
The conformance suite can also run tests against implementations and live deployments compliant to the FAPI-aligned OpenBanking UK security standard. These tests are a superset of the FAPI-RW tests, and compliance with the FAPI-OB test profile also demonstrates compliance to the FAPI-RW profile.
The main differences are:
- OpenBanking UK authorization servers require an Intent ID to be created prior to authorisation using an access token obtained with the client credentials grant, which is then passed as an essential claim in the request object.
- OpenBanking UK defines ecosystem-specific ACR values.
The process of creating clients for the Conformance Suite is akin to on-boarding a Third Party, except that the suite and the credentials issued remain in the control of the ASPSP. The client credentials should be revoked after testing. The suite only requires AISP permission, as only this protocol is accessed during the flow, and only the ReadAccountsBasic permission is used. The suite does not require any significant customer data to be available; a dummy / reference account can be used.
When configuring the suite, the resource server url should be set to the URL for the AISP endpoints, for example:
Australian CDR Specific Notes
Tests can also be run against CDR compliant deployments.
The main differences are due to extra requirements in the CDR specifications, namely:
- private_key_jwt must be used
- required x-v header is sent to resource server endpoint
- Refresh tokens must be supported (the sharing_duration claim will be requested when a refresh token must be returned)
- Returned id_tokens must be encrypted
- For ACR claims, a CDR specific value is used, “urn:cds.au:cdr:2”
- x-fapi-auth-date header is included in all resource endpoint calls (it’s optional in FAPI but mandatory in CDR)
- x-cds-client-headers header is included when the x-fapi-customer-ip-address header is sent
If you would like any help, please email firstname.lastname@example.org. If it relates to a test failure, please include a link to the relevant log-detail.html, or if using a local install the downloaded log file.