Architecting Secure APIs: The Phantom Token Pattern

In modern API security, architects often face a dilemma when choosing an Access Token format: Reference Tokens (opaque strings) versus JSON Web Tokens.

JWTs offer stateless verification and performance benefits for microservices but expose sensitive claims to public clients and are difficult to revoke immediately. Conversely, Reference Tokens do not contain sensitive data and are revocable but introduce latency by requiring a database lookup (introspection) for every API request.

This article details the Phantom Token Pattern, an architectural solution that utilizes the OAuth 2.0 Introspection Endpoint to provide the best of both worlds. It ensures that public clients only ever handle opaque identifiers, while internal services benefit from the stateless utility of JWTs.

When a public client receives a JWT, the contents of that token are effectively public. Even if signed, the payload is just Base64-encoded.

Direct JWT usage presents several risks:

1. Content Leakage: Tokens often contain user IDs, emails, or internal roles. Exposing this can damage users’ privacy.
2. Coupling: Frontend apps may rely on the token structure, making it difficult to change the internal identity schema later.
3. Revocation Difficulty: Once a JWT is issued, it is valid until it expires. Revoking it requires complex mechanisms.

The Phantom Token Protocol

The Phantom Token approach separates the external token format from the internal token format.

System Components

• Public Client: Receives and sends a random string (Reference Token). It knows nothing about the user’s data.
• API Gateway or BFF: The boundary between the public internet and the private network. It acts as a reverse proxy.
• Authorization Server: Issues tokens and provides an Introspection Endpoint.
• Internal APIs: Microservices that expect a JWT to authorize requests statelessly.

The Exchange Flow

1. Token Issuance: The client authenticates and receives an opaque Access Token (e.g., session ID).
2. API Request: The client calls the API Gateway with the HTTP Authorization header Authorization: Bearer cc445....
3. Introspection: The API Gateway intercepts the request. It sends the opaque token to the Authorization Server’s Introspection Endpoint.
4. Exchange: The Authorization Server validates the token. If valid, it generates a signed JWT containing the user’s claims and returns it to the Gateway.
5. Forwarding: The Gateway discards the opaque token, injects the JWT into the Authorization header, and forwards the request to the downstream API.
6. Caching: Optionally, the Gateway may cache the JWT with some TTL (it can be the TTL of the issued JWT), so on subsequent requests, it won’t need the introspection endpoint to perform the exchange step.

The Introspection Endpoint Role

The core of this pattern lies in the customization of the Introspection Endpoint. Standard introspection returns a JSON object indicating token status:

{
  "active": true,
  "scope": "read write",
  "client_id": "gateway"
}

To support Phantom Tokens, the Authorization Server must be configured to either return the JWT directly or embed it in the response.

Approach A: The Token Procedure

During introspection, the Authorization Server looks up the data associated with the opaque token and creates a new JWT on the fly. Below is a conceptual implementation of a Token Procedure that modifies the introspection response:

// https://datatracker.ietf.org/doc/html/rfc7662
const introspectionSchema = z.object({
  token: z.string(),
  token_type_hint: z.string().optional(),
});

const requireAuth = () => async (c, next) => {
  const [clientId, clientSecret] = getAppCredentials();
  const isValid = await verifyClientCredentials(clientId, clientSecret);

  if (!isValid) {
    throw new HttpException();
  }

  await next();
};

app.post(
  '/introspect',
  // Middlewares for application/x-www-form-urlencoded data validation and for authorization.
  zValidator('form', introspectionSchema),
  requireAuth(),
  async (c) => {
    // `token` is a user's opaque token that was sent do Gateway using Authorization header.
    const { token } = c.req.valid('form');
    const tokenData = tokenStore.get(token);

    if (!tokenData || !tokenData.active) {
      return c.json({ active: false });
    }

    // Using public key cryptography to allow check if the signature is correctly created
    // without need to share a secret.
    const { privateKey, kid } = await keyVault.get();

    const phantomToken = await new SignJWT({
      client_id: tokenData.client_id,
      username: tokenData.username,
      scope: tokenData.scope,
    })
      .setProtectedHeader({
        alg: ALG.RS256,
        typ: 'JWT',
        kid,
      })
      .setIssuer(issuer)
      .setAudience(audience)
      .setSubject(tokenData.username)
      .setIssuedAt()
      .setExpirationTime(tokenData.exp)
      .sign(privateKey);

    return c.json({
      active: true,
      client_id: tokenData.client_id,
      username: tokenData.username,
      scope: tokenData.scope,
      exp: tokenData.exp,
      phantom_token: phantomToken,
    });
  },
);

Approach B: MIME Type Negotiation

Alternatively, the API Gateway can request the JWT directly by setting the Accept header to application/jwt during the introspection call.

Gateway Request:

POST /introspect HTTP/1.1
Host: idp.example.com
Content-Type: application/x-www-form-urlencoded
Authorization: Bearer 23410913-abewfq.123483
Accept: application/jwt

token=cc445-85a0-759c...

Server Response:

If valid, the server returns the raw JWT string as the body. If invalid, it returns a 204 or 404. This simplifies the Gateway logic, as no JSON parsing is required to extract the token.

Gateway Implementation

The API Gateway is responsible for the exchange. It should cache the introspection result to avoid overloading the Authorization Server. To respect the possibility of users’ opaque token revocation, the cache must be short-lived and it should also be possible to invalidate the cache, e.g. using a central cache system such as Redis.

Here is a conceptual configuration logic for handling the exchange:

app.use('*', async (c, next) => {
  const opaqueToken = getOpaqueTokenFromAuthorizationHeader();
  let phantomToken = await redis.get(PHANTOM_TOKEN_CACHE_KEY);

  if (!phantomToken) {
    // Cache Miss: Introspect the token at the Authorization Server
    const response = await fetch(`${AUTHORIZATION_SERVER}/introspect`, {
      method: 'POST',
      headers: {
        'Content-Type': 'application/x-www-form-urlencoded',
        Authorization: `Bearer ${credentials}`,
        Accept: 'application/json',
      },
      body: new URLSearchParams({ token: opaqueToken }),
    });

    if (!response.ok) {
      throw new HttpException();
    }

    const data = await response.json();
    if (!data.active || !data.phantom_token) {
      throw new HttpException();
    }
    phantomToken = data.phantom_token;
    // Save to Redis with a short TTL
    await redis.set(
      cacheKey,
      phantomToken,
      'EX',
      PHANTOM_TOKEN_CACHE_TTL_SECONDS,
    );
  }

  // Swap the token: Replace the Opaque Token with the Phantom Token
  // Downstream services will now receive the JWT
  c.req.raw.headers.set('Authorization', `Bearer ${phantomToken}`);

  // Continue to the intended downstream route
  await next();
});

Security Analysis

The Phantom Token pattern significantly alters the security model of the application.

Benefits

Privacy and Storage: The public client holds a token that is mathematically unrelated to the user’s identity. If an attacker gets the token, they find no user information, only a random string. To maximize protection against token theft, the Gateway must store these opaque tokens in HttpOnly cookies.

Centralized Control: Because the Gateway must introspect the token, the Authorization Server remains the single source of truth.

Revocation Ability: If a user clicks Log Out or an administrator revokes a session, the opaque token becomes invalid immediately. The next time the Gateway tries to introspect it, the Authorization Server returns active: false, and access stops instantly.

Internal Performance: Microservices do not need to call the Authorization Server. They perform local, stateless validation of the JWT signature (using public keys via JWKS).

Token Replay Mitigation: This pattern also mitigates attacks involving token replay within the internal network. Since the JWT is generated at the Gateway, it can be issued with a very short lifespan and restricted audiences, whereas the external opaque token might live for hours. Even if an internal service is compromised and leaks the JWT, its utility to an attacker is strictly limited in time and scope.

Demonstrating Proof-of-Possession: Both the user agent and the Gateway can use DPoP to prevent the use of stolen opaque tokens.

Drawbacks

Latency Overhead: The API Gateway must perform an introspection request to the Authorization Server for every new opaque token. This adds latency to the initial API call and it can degrade the user experience if not properly mitigated through caching.

System Complexity: Implementing this pattern requires additional infrastructure components. The architecture must support an introspection endpoint, a shared caching layer, and a secure mechanism for the Gateway to authenticate itself (e.g. using DPoP).

Stateful Bottleneck: Because opaque tokens require server-side validation, the Authorization Server becomes a potential bottleneck. It must be scaled to handle the volume of incoming introspection requests from the API Gateway.

Cache Invalidation Trade-offs: While caching reduces the load on the Authorization Server, it introduces a delay in token revocation. Achieving true immediate revocation requires an active cache invalidation system.

Conclusion

The Phantom Token pattern creates a balance between external security and internal statelessness. Instead of only checking if a token is valid, the system uses the introspection endpoint to actually swap the token. This allows applications to use safe, random tokens on the public internet while using stateless JWTs inside their internal network.