This page describes how to run conformance tests and gather testing results for Financial-grade API (FAPI) Read/Write Profile OpenID Providers (OPs).
Currently only FAPI-RW (i.e. FAPI part 2) has a certification program. The parts of FAPI-RW that are not currently tested are described on our wiki: https://gitlab.com/openid/conformance-suite/wikis/OP-FAPI-RW-Test-Status
There are tests available for FAPI-R (i.e. part 1); these are not yet complete. It is not necessary to run FAPI-R tests when certifying for FAPI-RW.
If your server complies with the UK OpenBanking specification, you should run only the ‘FAPI-RW-ID2-OB’ tests. These allow you to demonstrate compliance with both FAPI-RW and the OpenBanking UK Profile of FAPI-RW. The FAPI-RW-ID2-OB are a superset of the FAPI-RW-ID2 tests, with the major difference being that OB compliant authorization servers requiring pre-registration of intent using an OB specific API for an authorization to succeed.
There are also tests under development for FAPI-CIBA; please email email@example.com if you have an implementation of CIBA you would like to test.
Register two clients in your authorisation server and the necessary keys/certificates. Two clients are necessary as the suite tests various mixup attacks, for example, using a valid access token from one client with the TLS client certificate from the other client.
The redirect URLs for the clients must be:
ALIAS must be replaced with a string that is unique to you/your organisation, to avoid other conformance suite users affecting your testing.
Please include the
?dummy1=lorem&dummy2=ipsum string in the redirect url exactly as written. The conformance suite will use both redirect URLs.
The IP address the conformance suite will make outgoing calls from is available within the suite on the home page after login in. (The IP address may change from time to time as the conformance suite runs on a Kubernetes cluster.)
Instructions for Running Tests
- Open https://www.certification.openid.net/.
- Login using a Google or GitLab account, or any Op that supports WebFinger.
- Ensure “Multi test plan” is selected. (If “Single test” is shown, toggle it to “Multi test plan”).
- Select the relevant test plan from the “Select a Test Plan” dropdown menu. The tests you should run depends on your client authentication type(s). For example, on a FAPI-RW compliant server that supports oauth-mtls client authentication you should run
- DO NOT pick a test plan that has ‘client’ in the name. These are not applicable when testing an OpenID Provider.
- Fill in the configuration form (as per the guidance given in the form fields when empty). You can switch to the “JSON” tab to view/edit the configuration in the underlying JSON format. Changes to the form are automatically reflected in the JSON and vice versa. The JSON can be copied and saved locally to be pasted back in later. The server will automatically remember the most recent JSON you successfully created a test with.A “real world” example of a full working configuration is available here: https://gitlab.com/openid/conformance-suite/wikis/Authlete-Example-Configuration.
- Note that both client JWKS sets must contain an ‘alg’ field. For FAPI-RW this is most usually ‘PS256’, but ‘ES256’ is also permitted.
- A resource server URL must be provided. This must be a simple GET endpoint that returns JSON and is used to test certificate binding of access tokens etc. (Different in OpenBanking UK tests; see below.)
- institution_id: any value is okay and is sent in a x-fapi-financial-id header. (This is a hangover from FAPI-R-ID1 and is only retained due to an error in an OpenBanking specific specification.)
- Press “Launch Test Plan”. You will be taken to a list of all the test modules in the plan.
- Press “Run New Test” on the first test module.
- Please read the description of the test in the light blue box near the top of the log page – this may contain specific instructions, for example one test requires that the user rejects the authentication process.
- If applicable, click on the “Visit” button, and login to your OP; some tests require you to do this more than once. You will be either be:
- directed back to the test suite, in which case hit the link to return to the log page, or:
- shown an error by the AS. If the error is expected (i.e. is mentioned in the light blue summary box on the log page), you should screenshot it and upload it to the conformance suite on the log detail page.
- When the test has completed, press “Continue Plan” to start the next text, or “Return to Plan” to view your progress.
If you require support, please email firstname.lastname@example.org. If it relates to a test failure, please include a link to the relevant log-detail.html, or if using a local install the downloaded log file.
About the Conformance Suite
The conformance suite is an open source project run by the OpenID Foundation, the source code can be found on gitlab: https://gitlab.com/openid/conformance-suite/
Release notes for the suite are available on gitlab: https://gitlab.com/openid/conformance-suite/releases
If you need access to bug fixes that have been merged into the conformance suite but are not yet pushed to the production server, you can use the staging environment (which automatically reflect the ‘master’ branch in git): https://staging.certification.openid.net/ (this would require you to add redirect urls for the staging server to your clients).
The master git branch is regression tested against various vendor-provided cloud environments at least once every 24 hours; a summary of the results can be viewed here: https://staging.certification.openid.net/publicPlans.html
The conformance suite can be installed locally inside docker, see: https://gitlab.com/openid/conformance-suite/wikis/Developers/Build-&-Run
A python script and library are available to allow the conformance suite to be used in a continuous integration system; it is highly recommended that authorization server developers integrate this into their development pipeline: https://gitlab.com/openid/conformance-suite/blob/master/scripts/run-test-plan.py
Enhancements, bug fixes and other contributions to the conformance suite are welcome, see: https://gitlab.com/openid/conformance-suite/wikis/Developers/Contributing
Please report bugs in the issue tracker with FULL details of what happened, links to the log page for the test run, and so on: https://gitlab.com/openid/conformance-suite/issues
If you require support, please email email@example.com.
OpenBanking UK Specific Notes
The conformance suite can also run tests against implementations and live deployments compliant to the FAPI-aligned OpenBanking UK security standard. These tests are a superset of the FAPI-RW tests, and compliance with the FAPI-OB test profile also demonstrates compliance to the FAPI-RW profile.
The main differences are:
- OpenBanking UK authorization servers require an Intent ID to be created prior to authorisation using an access token obtained with the client credentials grant, which is then passed as an essential claim in the request object.
- OpenBanking UK defines ecosystem-specific ACR values.
The process of creating clients for the Conformance Suite is akin to on-boarding a Third Party, except that the suite and the credentials issued remain in the control of the ASPSP. The client credentials should be revoked after testing. The suite only requires AISP permission, as only this protocol is accessed during the flow, and only the ReadAccountsBasic permission is used. The suite does not require any significant customer data to be available; a dummy / reference account can be used.
When configuring the suite, the resource server url should be set to the URL for the AISP endpoints, for example: