Conformance Testing for OpenID AuthZEN Authorization API 1.0

This page describes how to run the conformance tests for OpenID AuthZEN PDP. At the time of writing, the set of tests currently available in production are in an early stage. Do not hesitate to register issues in the conformance suite project (https://gitlab.com/openid/conformance-suite) if you're observing unexpected or incorrect behavior, or if you have questions regarding the tests.

The OpenID Foundation conformance suite tests a Policy Decision Point (PDP) Server that implements the AuthZEN Authorization API 1.0. The suite acts as the Policy Enforcement Point (PEP): it calls your PDP’s API endpoints and checks the responses against the specification and the AuthZEN certification profile.

References

The AuthZEN tests are at an early stage. Please report anything unexpected (see Getting help). The six plans under Available test plans are the certification plans.

Getting started

  1. Go to https://www.certification.openid.net and log in with your Google or GitLab account.
  2. Click Create a new test plan.
  3. Choose one of the AuthZEN test plans listed under Test an AuthZEN PDP server (see Available test plans).
  4. Select the variants that describe your PDP (see Test plan variants).
  5. Fill in the configuration (see Configuring the test).
  6. Click Create Test Plan and continue to Running the tests.

Prepare your PDP: the certification fixture

The certification profile (section 4) defines a small fixture of subjects, resources and actions that your PDP must load before testing. The suite only validates decision values for the requests the fixture defines; every other request is checked for structure and status codes only.

EntityTypeIdentifiers and properties
Subjectsuseralice; bob (property role = "admin" when sent as a property)
Resourcesrecordrecord-1 (property status = "active"); record-2 (property status = "archived")
Actions—read; write; delete (property soft = true or false)

Decisions the suite validates (certification profile section 4.4)

#SubjectActionResourceDecision
Core (identifiers only)
1alicereadrecord-1true
2alicewriterecord-1true
3bobreadrecord-1true
4bobwriterecord-1false
Properties
5alicewriterecord-2 (status=archived)false
6bob (role=admin)writerecord-2 (status=archived)true
7alicedelete (soft=true)record-1true
8alicedelete (soft=false)record-1false

Search results the suite expects (certification profile section 4.5) include, for example: users who can read record-1 include alice and bob; records alice can read include record-1; actions alice can perform on record-1 include read and write. The Properties search scenarios add the admin and archived-record cases. Your PDP may return additional entities beyond the fixture.

Available test plans

Plans 1 to 5 each test one AuthZEN API endpoint, and plan 6 tests the PDP metadata. Run every plan that matches what your PDP implements.

1. AuthZEN 1.0: PDP server test for evaluation

Test plan name
authzen-pdp-evaluation-test-plan
Endpoint
Access Evaluation API (spec section 6)
Profile level
Basic (certification profile section 5)

Covers single decisions for the fixture rules, request validation (missing or malformed subject, action and resource), unknown fields being ignored, transport behaviour (HTTPS POST, JSON content type, malformed JSON, empty body), X-Request-ID echo (on error responses too), idempotency, and, when the PDP authenticates callers, 401 responses for missing or invalid credentials.

2. AuthZEN 1.0: PDP server test for batch evaluations

Test plan name
authzen-pdp-evaluations-test-plan
Endpoint
Access Evaluations API (spec section 7)
Profile level
Batch (certification profile section 6); the profile requires the Basic level as a prerequisite

Covers batch requests with an evaluations array, inheritance of top-level subject, action, resource and context defaults, per-evaluation overrides, the execute_all semantic, decision ordering, the backwards-compatible single-evaluation forms, transport behaviour, X-Request-ID, idempotency, and, when the PDP authenticates callers, 401 responses for missing or invalid credentials.

3–5. AuthZEN 1.0: PDP server tests for Search

PlanTest plan nameEndpoint
Subject Searchauthzen-pdp-subject-search-test-planSubject Search API (spec section 8.4)
Resource Searchauthzen-pdp-resource-search-test-planResource Search API (spec section 8.5)
Action Searchauthzen-pdp-action-search-test-planAction Search API (spec section 8.6)

Together these make up the Search level (certification profile section 7). They cover the expected search results, request validation, ids that must be ignored in a search request, unknown fields, empty results for unknown entities, response and page object structure, pagination (the subject search plan follows page.next_token across pages with page.limit set), transport behaviour, X-Request-ID, idempotency, and, when the PDP authenticates callers, 401 responses for missing or invalid credentials.

6. AuthZEN 1.0: PDP server test for Discovery

Test plan name
authzen-pdp-discovery-test-plan
Endpoint
PDP metadata at /.well-known/authzen-configuration (spec section 9)
Profile level
Discovery (certification profile section 9). Discovery is an independent level: it does not require, and is not required by, any other level.

Has a single test, which fetches your PDP’s metadata and checks it:

  • HTTP 200 with Content-Type: application/json and a JSON object body;
  • policy_decision_point equal to the PDP Identifier;
  • access_evaluation_endpoint present, and every endpoint an HTTPS URL;
  • capabilities, if present, an array of strings;
  • signed_metadata, if present, a JWT with an iss claim whose signature verifies against the PDP JWK Set.

It calls no API endpoint. The metadata endpoint must be available: an HTTP 404 fails the test. The plan has no variants, and the suite sends no credentials when it fetches the metadata, as with other OpenID discovery documents: the metadata endpoint must be reachable without authentication.

The discovery variant on plans 1 to 5 runs the same metadata checks before it takes the endpoints from the metadata, but those plans certify their API level, not Discovery. To certify Discovery, run plan 6.

Not part of the certification program

AuthZEN 1.0: PDP server test for batch evaluations, including optional evaluation semantics

Test plan name
authzen-pdp-evaluations-comprehensive-test-plan
Endpoint
Access Evaluations API (spec section 7)

Runs everything in plan 2, plus the optional short-circuiting evaluations_semantic values (deny_on_first_deny, permit_on_first_permit) and the rejection of an unknown evaluations_semantic value. A PDP does not have to support these (spec section 7.1.2.1), so the plan reports no certification profile. Use it to check a PDP that claims these semantics; use plan 2 to certify.

Certification profile names

When you publish results for certification, each plan reports the profile it certifies. For plans 1 to 5 it depends on the AuthZEN Specification Support variant:

Planauthzen_support=coreauthzen_support=properties
evaluationAuthZEN EvaluationAuthZEN Evaluation w/properties
batch evaluationsAuthZEN EvaluationsAuthZEN Evaluations w/properties
Subject SearchAuthZEN Subject SearchAuthZEN Subject Search w/properties
Resource SearchAuthZEN Resource SearchAuthZEN Resource Search w/properties
Action SearchAuthZEN Action SearchAuthZEN Action Search w/properties

The Discovery plan reports AuthZEN Discovery.

Test plan variants

Plans 1 to 5 each have three dropdowns. Create a separate plan for each combination your PDP supports. The Discovery plan has none.

PDP Authentication Type (pdp_auth_type)

How the suite authenticates to your PDP. The AuthZEN specification leaves PEP authentication out of scope (section 11.2) and the certification profile does not require it, so choose what your deployment uses. If your PDP supports several methods, test each one in its own plan.

ValueMeaning
noneThe PDP does not authenticate callers.
client_secret_basicHTTP Basic authentication with a client id and secret.
api_keyAn API key, sent as a Bearer token in the Authorization header.
mtlsMutual TLS: the suite presents a client certificate in the TLS handshake.

With client_secret_basic and api_key, each plan for an API endpoint (plans 1 to 5) also checks that the PDP returns HTTP 401 for a request with missing credentials and for one with invalid credentials (spec section 11.3).

PDP server metadata location (pdp_server_metadata)

ValueMeaning
discoveryThe suite fetches your PDP’s metadata from /.well-known/authzen-configuration and takes the endpoints from it (spec section 9). Recommended if your PDP supports it, as it needs less configuration and also tests the metadata. To certify the metadata itself, also run the Discovery plan. If the metadata does not contain the endpoint a plan tests, the PDP does not support that API (spec section 9.1.1) and the plan’s tests stop; add the endpoint to the metadata, or use static. The suite fetches the metadata without credentials, as with other OpenID discovery documents, so the metadata endpoint must be reachable without authentication even when your PDP authenticates API calls.
staticYou enter the endpoints in the test configuration and the suite never calls the metadata endpoint.

AuthZEN Specification Support (authzen_support)

ValueMeaning
coreYour PDP makes decisions from entity identifiers (type, id, name) only. Requests may still carry properties, but they must not change the decision.
propertiesYour PDP also bases decisions on the properties of the subject, resource and action (spec sections 5.1.1, 5.2.1, 5.3.1). This adds the Properties fixture rules 5 to 8 and the property-based search scenarios to each plan, and certifies the “w/properties” profile. Choose it if your PDP supports the Properties sub-level of the certification profile. In the profile, Batch Properties requires Basic Properties, and Search Properties requires Search Core.

Configuring the test

The configuration form shows only the fields that apply to the plan and variants you selected. You can fill in the form or edit the JSON directly.

Test information

alias

Makes any URLs used in the test unique to you. Set it to something unique, such as your company name, using only URL-safe characters.

description

Optional free text to identify the run. It is public if you publish the results.

publish

Leave as “No”. You can publish later (see Publishing the results). “Everything” also publishes your configuration, including any secrets in it.

AuthZEN (the pdp object)

PDP Identifierpdp.policy_decision_pointall plans

Required for every plan. The base HTTPS URL of your PDP, as defined in spec section 9.1.1. It must use https, have a host, and contain no query or fragment. Do not add a trailing slash unless your PDP’s identifier has one.

With the Discovery plan or the discovery variant, the suite derives the metadata URL from it by inserting /.well-known/authzen-configuration between the host and any path:

PDP IdentifierMetadata URL
https://pdp.example.comhttps://pdp.example.com/.well-known/authzen-configuration
https://pdp.example.com/tenant1https://pdp.example.com/.well-known/authzen-configuration/tenant1

The policy_decision_point in the returned metadata must equal this value exactly (spec section 9.2.3), or the test stops.

API endpointsstatic only
FieldKeyUsed by
Access Evaluation API Endpointpdp.access_evaluation_endpointEvaluation plan
Access Evaluations API Endpointpdp.access_evaluations_endpointBatch evaluations plan
Subject Entity Search API Endpointpdp.search_subject_endpointSubject Search plan
Resource Entity Search API Endpointpdp.search_resource_endpointResource Search plan
Action Entity Search API Endpointpdp.search_action_endpointAction Search plan

If you leave an endpoint field empty, the suite uses the default path from spec section 10.1 under the PDP Identifier:

<PDP Identifier>/access/v1/evaluation
<PDP Identifier>/access/v1/evaluations
<PDP Identifier>/access/v1/search/subject
<PDP Identifier>/access/v1/search/resource
<PDP Identifier>/access/v1/search/action

The suite warns if an endpoint path does not contain a version segment such as /v1 (spec section 4).

PDP JWK Setpdp.jwksDiscovery plan and discovery variant

Required when your PDP’s metadata contains signed_metadata (spec section 9.1.3); leave it empty otherwise. A JSON Web Key Set, starting with { "keys": [, holding the public key the suite uses to verify the signed_metadata JWT. Do not include private asymmetric keys.

If your PDP signs with an HMAC algorithm such as HS256, the set holds the shared secret as an "oct" key. Anyone who can see your configuration can then sign metadata as your PDP, so use a secret that is only for testing and do not publish “Everything”.

Signed Metadata Issuerpdp.metadata_issuerDiscovery plan and discovery variant, optional

The expected iss claim of the signed_metadata JWT. Set it only when a party other than the PDP signs the metadata. When empty, iss must equal the PDP Identifier.

Client credentials (pdp_auth_type=client_secret_basic)

client_idclient.client_id

The client identifier your PDP issued to the PEP.

client_secretclient.client_secret

The matching secret.

The suite sends Authorization: Basic base64(client_id:client_secret). This is the OAuth 2.0 client_secret_basic method; the AuthZEN specification recommends OAuth 2.0 support (spec section 2).

As RFC 6749 section 2.3.1 requires, the id and secret are each form-URL-encoded before they are joined and base64-encoded. Letters, digits and - . _ * are unchanged, a space becomes +, and any other character is percent-encoded: a+b/c= is sent as a%2Bb%2Fc%3D. Your PDP must decode the credentials the same way. A PDP that decodes them as plain HTTP Basic credentials (RFC 7617) rejects an id or secret containing such characters with 401, so for that PDP use an id and secret made only of letters, digits and - . _ *. RFC 6749 allows only printable US-ASCII in a client id or secret (Appendix A.1, A.2).

API key (pdp_auth_type=api_key)

PDP API Keyclient.api_key

The API key your PDP expects. The suite sends it as Authorization: Bearer <key>.

Mutual TLS (pdp_auth_type=mtls)

KeyValue
mtls.certThe TLS client certificate, in PEM format or as base64-encoded DER without headers. Your PDP must trust it.
mtls.keyThe matching unencrypted private key, in PEM format or as base64-encoded DER without headers.
mtls.caOptional. Certificates for the client certificate’s chain, in the same formats. Omit it for a self-signed certificate.

The certificate is presented in the TLS handshake of every API call. The missing-credentials and invalid-credentials 401 tests do not run with mtls.

Example configurations

Static metadata, no authentication:

{
    "alias": "my-company",
    "description": "Example PDP, static endpoints",
    "pdp": {
        "policy_decision_point": "https://pdp.example.com",
        "access_evaluation_endpoint": "https://pdp.example.com/access/v1/evaluation",
        "access_evaluations_endpoint": "https://pdp.example.com/access/v1/evaluations",
        "search_subject_endpoint": "https://pdp.example.com/access/v1/search/subject",
        "search_resource_endpoint": "https://pdp.example.com/access/v1/search/resource",
        "search_action_endpoint": "https://pdp.example.com/access/v1/search/action"
    }
}

Discovery plan, with signed metadata:

{
    "alias": "my-company",
    "description": "Example PDP, metadata",
    "pdp": {
        "policy_decision_point": "https://pdp.example.com",
        "jwks": {
            "keys": [
                { "kty": "EC", "crv": "P-256", "kid": "metadata-key",
                  "x": "...", "y": "..." }
            ]
        }
    }
}

Discovery variant on an API plan, with signed metadata and an API key:

{
    "alias": "my-company",
    "description": "Example PDP, discovery",
    "pdp": {
        "policy_decision_point": "https://pdp.example.com",
        "jwks": {
            "keys": [
                { "kty": "EC", "crv": "P-256", "kid": "metadata-key",
                  "x": "...", "y": "..." }
            ]
        }
    },
    "client": {
        "api_key": "test-api-key"
    }
}

Running the tests

In the test plan overview you are presented with a list of individual tests. Click Run Test to execute each one. Each test calls your PDP and finishes on its own; none of them needs any action in a browser.

Status codes for rejected requests

Several tests send a request your PDP should reject. These statuses follow the AuthZEN working group’s guidance:

RequestExpected status
Wrong method (GET or PUT; the API uses POST, spec section 10.1)405 Method Not Allowed with an Allow: POST header (preferred; RFC 7231 section 6.5.5 requires the Allow header on any 405), or 400. A 404 is reported as a warning: it is only expected when your framework routes on method and path, so check it is not a routing problem. A PDP that accepts the request is also reported as a warning.
Wrong or missing Content-Type (spec section 10.1, certification profile section 5.4.3)400 or 415 Unsupported Media Type. Any other status fails, including 404, as an unsupported media type is a problem with the content, not a missing route.
Malformed request (missing or wrongly typed fields, malformed JSON, an empty body)400 (spec section 10.1.1). A batch in which an evaluation has no subject, action or resource at any level may instead get 200, with that evaluation returned as "decision": false.
Missing or invalid credentials, when your PDP authenticates callers401 (spec section 11.3); a WWW-Authenticate header is recommended.
ResultMeaning
PassedEvery check succeeded.
WarningThe PDP did something the specification recommends against (a SHOULD), or the suite could not check something; review it.
FailedThe PDP violated a requirement. Open the test’s log to see the request the suite sent, the response, and the requirement that failed, with a link to the specification section.
SkippedThe test could not be exercised; its log says why. Tests that do not apply to the variants you chose are not listed in the plan.

Run every test in the plan. For certification, all tests must have finished without failures.

Publishing the results

In the test plan overview, click Publish everything to make the results visible to anyone, not just logged-in users. Use Publish summary if your configuration contains secrets (client_secret, API key, mTLS key, or an HMAC signing key) that must stay private.

To apply for certification, follow the OpenID Foundation certification process at https://openid.net/certification/ with the published results and the certification profile name for each plan.

Getting help

Do not hesitate to register issues in the conformance suite project if you see unexpected or incorrect behaviour. You can also contact certification@oidf.org.