Running the conformance suite
You cannot certify using test results obtained from a local instance, you MUST use https://www.certification.openid.net/ for certification.
Using a local instance for development and testing is recommended. Please do not use https://www.certification.openid.net/ for automated testing, such as running tests as part of your internal build process.
Running the suite locally
See https://gitlab.com/openid/conformance-suite/-/wikis/Developers/Build-&-Run for instructions. (Only steps 1-6 under ‘Build & Run’ apply to RP tests)
When using a local instance of the conformance suite software, a self-signed certificate will be used for https endpoints. You may need to disable certificate validation checks in your client software.
https://www.certification.openid.net uses certificates issued by Letsencrypt, disabling certificate checks may still be necessary if your client is configured to accept certificates issued by a specific certificate authority only.
Test: A single test that is intended to test a specific test case, e.g testing how the client behaves when it receives an id_token signed using the wrong algorithm.
Plan: A set of tests, forming the conformance testing profile. The same configuration is used for all tests under a plan.
Negative test: Tests that intentionally return invalid data/response and expect the client to detect this issue.
For RP tests, the conformance suite acts as the OpenID Provider (OP) and your client will act as the Relying Party (RP).
Please note that the suite is not a fully featured OP/bank and it has its own limitations. It expects certain events to take place in a predefined order and may return errors if it encounters an unexpected situation even if it is technically acceptable.
- The suite always returns a constant consent response regardless of what was included in the consent request.
- For payment initiation requests, the suite copies the “data” element as is from the request and just adds claims like iss, iat to the returned token.
- For dynamic client registration, the suite returns the registration request as is by just adding a random client id.
- The suite does NOT support idempotency. Repeated requests may cause errors.
- As the suite cannot be added to the Open Banking Brasil Directory as a real OP and uses test keys and certificates, you may need to disable some checks in your client.
Go to https://www.certification.openid.net/ or to your local conformance suite instance, e.g https://localhost:8443/.
When using https://www.certification.openid.net/, log in using a Google or GitLab account, or any OpenID Provider that supports WebFinger.
Create a “Test Plan”
On the suite home page, click “Create a new test plan” button to create a test plan and start running tests.
- Select “FAPI1-Advanced-Final: Open Banking Brazil Relying Party (Client) Test Plan”.
- From the “Client Authentication Type” dropdown, select “mtls” or “private_key_jwt” testing depending on which type of client authentication you support.
- From the “FAPI Response Mode” dropdown, select “plain_response” or “jarm”, if you want to test using JARM. Select “plain_response” if you don’t want to test JARM.
- From the “FAPI JARM Type” dropdown, select one of “oidc” or “plain_oauth”. When “plain_oauth” is selected then the suite will not issue any id_tokens. This field is applicable only when JARM response type is also selected. (Select “oidc” by default, i.e when jarm is not selected or when you are not sure what you should select)
After configuring the standard fields, additional form fields will be displayed below. These fields may vary based on selected options.
Fill in the configuration form (as per the guidance given in the form fields when empty). You can switch to the “JSON” tab to view/edit the configuration in the underlying JSON format. Changes to the form are automatically reflected in the JSON and vice versa. The JSON can be copied and saved locally to be pasted back in later. The server will automatically remember the most recent JSON you successfully created a test with.
Server jwks configuration
- Server jwks must include private keys. Keys provided in server jwks will be used as the server keys for your tests.
- Server jwks MUST contain one key usable for signing (i.e use=sig) and one key usable for encryption (i.e use=enc).
- For payments support the server jwks MUST contain a BRSEAL signing key, an organization key that can be found on the directory. You can use the server jwks from the example configuration which contains an organization key.
You can use the server jwks from the example configuration provided at https://gitlab.com/openid/conformance-suite/-/wikis/Brazil-RP-Testing-Instructions-and-Example-Configuration
Two client configurations are required for Brazil RP tests. The first client will be used for most tests.
The second client will be used for encrypted id_token tests and jwks for the second client must contain an encryption key, e.g a key with use=enc.
Client jwks configuration
- Client jwks does not need to include private keys. Only the public keys will be used. These keys must match the keys that will be used by your client.
Dynamic Client Registration (DCR)
For dynamic client registration tests, the suite expects the configured client certificate for MTLS authentication. DCR tests does not support using a different certificate/key for newly registered clients.
Other configured fields such as client id, redirect uri, jwks (client jwks will be downloaded from the jwks_uri) will NOT be used for DCR tests.
Please note that item #1 at https://openbanking-brasil.github.io/specs-seguranca/open-banking-brasil-dynamic-client-registration-1_ID1.html#name-authorization-server-2 requires that registration requests are also performed over a mutual tls secured connection.
See https://gitlab.com/openid/conformance-suite/-/wikis/Brazil-RP-Testing-Instructions-and-Example-Configuration for additional information and an example configuration.
Creating the “Test Plan”
Once the configuration is completed, click “Create Test Plan” button.
A page with a list of tests included in this plan will be displayed.
Tests included in the plan
“FAPI1-Advanced-Final: Open Banking Brazil Relying Party (Client) Test Plan” contains the same set of tests twice. The first set is for non-PAR (plain_request) and the second is for PAR testing. Since RPs are expected to support both plain requests (non-PAR) and pushed authorization requests (PAR), the test plan contains the same set of tests twice, once for non-PAR and once for PAR testing. For example “fapi1-advanced-final-client-test” must be run twice, first without using PAR and the next time using PAR.
The suite supports both accounts and payments. The same tests can be run in either accounts mode or payments mode. Mixing accounts and payments requests (e.g first sending a request to consents/v1/consents and then later to payments/v1/pix/payments) will lead to errors.
Using the “accounts” mode
- Initial client credentials request to the token endpoint must contain scope consents
- New consent request must be sent to the consents_endpoint displayed on the UI, i.e the endpoint ending with consents/v1/consents
- Authorization request scope must contain accounts
- accounts_endpoint displayed on the UI (i.e the endpoint ending with accounts/v1/accounts) must be used as the resource endpoint (the test must end with a call to the accounts endpoint)
Using the “payments” mode
- Initial client credentials request to the token endpoint must contain scope payments
- New consent request must be sent to the payments_consents_endpoint displayed on the UI, i.e the endpoint ending with payments/v1/consents
- Authorization request scope must contain payments
- payment_initiation_path displayed on the UI (i.e the endpoint ending with payments/v1/pix/payments) must be used as the resource endpoint (the test must end with a call to the payment initiation endpoint)
1. On the Plan page (see screenshot above), click the “Run Test” button to start a test.
2. The suite will start the test and will start waiting for incoming requests.
3. Tests expect a series of requests from the client. Typically for RP tests, the following sequence of requests are expected:
- Token request using client credentials
- New consent request using the access token obtained in previous step (the suite always returns the same consent response, regardless of what was requested in the consent request)
- PAR request if it’s a PAR test
- Authorization request
- Token request
- Resource request (using the Accounts Endpoint)
- The client can also send JWKS requests and get consent requests as necessary
4. After the test starts the client must send the expected requests in the expected order. Sending requests in a different order, sending duplicate requests, sending requests after the test transitions into FINISHED state may cause errors.
5. After sending the necessary requests, wait until the test transitions into FINISHED state. Moving to a different test before the current test is FINISHED will lead to errors.
6. Negative tests, i.e tests that expect the client to detect an error and stop, will wait for 5 seconds (by default) before the test goes into FINISHED state. You must wait for the test to finish before moving on to the next one.
7. All tests under the test plan must be run one by one. The suite does not support running two tests using the same “alias” concurrently.
Please note: If your client software caches server configuration or jwks data, you may need to disable caching if you run into problems related to caching. Some server configurations (e.g supported algorithms, methods or similar returned in the discovery document), keys or similar might vary per test.
To report bugs or to request enhancements, submit an issue at https://gitlab.com/openid/conformance-suite/-/issues. Please use the template and provide all required information.
For support requests and questions please send an email to firstname.lastname@example.org.
Follow the instructions at https://openid.net/certification/instructions/ and https://openid.net/certification/fapi_rp_submission/.
Please note that you cannot certify with results obtained from a local instance, you must use https://www.certification.openid.net/ for certification.
For RP tests only, evidence demonstrating the behavior of the relying party for each test must be included in certification submissions. This can take the form of RP log files, screen captures (image files), or both. These files must be placed inside a folder named “client-data” (without the quotes). Files should be named consistently using the test name as the file name prefix. For example the log file for fapi-rw-id2-client-test should be named fapi-rw-id2-client-test.log or fapi-rw-id2-client-test-logs.txt or similar.
If more than one file needs to be included for a test then they should be named like fapi-rw-id2-client-test-1.log, fapi-rw-id2-client-test-2.log or similar.
Please make sure that client data contains enough details to allow a reader to see how the client behaved. For example, if your client was set to log only fatal issues and the client data included in your submission package did not contain any details then you will be asked to re-test.
Please also see:
- Recording of joint OIDF & Chicago Advisory workshop discussing RP certification
An example client can be found at https://gitlab.com/openid/sample-openbanking-client-nodejs. This is a nodejs application and requires nodejs and npm installed on your system.
You can use this client to see how your client implementation should interact with the conformance suite. Please note that this is just an example client and is neither production quality nor normative.
See instructions at https://openid.net/certification/fapi_rp_testing/ for more information.