跳到主要内容

Retries and automatic failover

Many models on OceanToken have more than one channel, and every channel is an official or authorized source of the same model. When one fails, the gateway retries the request and, if the model has another channel, moves the request there. You get the answer from the model you asked for, at the same price, a little later. Most of the time you will not see the failure at all.

This page explains what happens in each case and which errors still reach your code.

How a request is served​

  • Failover never changes the model. A request for a model is answered by that model, whichever channel serves it. A different model is tried only if you ask for one with fallbacks.
  • Channels are ranked. The first healthy channel serves the request; the next one takes over when it fails.
  • Coverage. The models with more than one channel include the current Claude and GPT models and several image and video models. Other models have a single channel today: they still get one retry after a temporary error, but there is nowhere to switch to.
  • The channel does not change the result. The model, the price and the response format are the same whichever channel answers. The response does not say which channel answered.

When a channel fails​

What happens upstreamWhat the gateway doesWhat you see
A temporary error: a server error, a dropped connection, a timeout on that channelRetries once on the same channel, then moves to the next channelA normal response, a little later
The channel is at capacity (it answers "too many requests")Moves straight to the next channel, without retrying the busy oneA normal response
A channel keeps failingTakes it out of rotation for a short time, so the next requests go straight to a working channelNothing
Every channel has failedReturns the last error5xx provider_error, or 429 provider_overloaded when every channel is at capacity. Nothing is charged.

You do not need to retry the failures the gateway handles for you. Keep a retry in your client for the errors that do reach it: 429 and 5xx with exponential backoff, as shown in Errors → Retries.

What is not retried​

Some failures would happen on every channel, so moving the request does not help. These come back to you as errors:

FailureCodeWhat to do
The request itself is wrong: a missing field, a value the model does not accept, a prompt longer than the model's context window400 invalid_requestFix the request.
A safety check refused the request or the answer400 content_filteredChange the input. A refusal is never sent on to another channel.
Your key, account or balance: an unknown model, a model outside your account, an invalid or expired key, no credits left, the key's own limit reached400, 401, 403, see ErrorsFix the key or the account.
Your key's own rate limit429 rate_limitedWait for Retry-After. See Limits.

A channel that goes quiet is abandoned. On a streamed request, if a channel sends nothing for 120 seconds before the first token, the request moves to the next channel. A non-streamed request still running after 240 seconds is closed with 504 timeout (Limits → Timeouts). For long answers, stream.

Streaming​

  • Before the first token, failover works exactly as above. The stream starts a little later.
  • After the answer has started, a clean switch is not guaranteed. Depending on the endpoint, the gateway either finishes the answer on another channel or ends the stream with an error event, as described in Errors → Errors while streaming.

Check that every stream ends with a finish reason: finish_reason on Chat Completions, stop_reason on Messages, a response.completed event on Responses. A stream without one is incomplete; retry it.

Video jobs​

Failover happens when a job is submitted. If the first channel refuses or fails the submission, the next channel takes it. Once a job has been accepted, it runs to the end on that channel. If it fails, it is reported as failed and its hold on your balance is released (Limits → Spending). To retry a submission safely after a network error, send an Idempotency-Key (Errors → Retrying a request that costs money).

Billing​

  • You pay the model's listed price, whichever channel answers. The price on the Models page applies to every channel.
  • Retries and switches are free. An attempt that fails before the model has produced any output is not billed; you pay for the answer you get. A stream that breaks after output has started is billed for its prompt and for the output streamed up to that point, reasoning included, and for nothing generated after it stopped (Errors → Errors while streaming).
  • Prompt caching does not carry over. Cached input is kept per channel. The first request served by a different channel may bill input at the regular rate where it would otherwise have been a cache hit.

Your own backup models​

You can also name other models to try if the model you asked for fails on all of its channels. Add fallbacks to the request body, as a list of model names in the order to try them. It works on /v1/chat/completions and /v1/responses.

# Your own backup models. If openai/gpt-5-mini fails on every one of its
# channels, the same request goes to anthropic/claude-haiku-4.5. `fallbacks` is
# not part of the OpenAI SDK's signature, so it travels in extra_body.
import os
from openai import OpenAI

client = OpenAI(
    base_url=os.getenv("OCEANTOKEN_BASE_URL", "https://api.oceantoken.ai/v1"),
    api_key=os.environ["OCEANTOKEN_API_KEY"],
)

response = client.chat.completions.create(
    model="openai/gpt-5-mini",
    messages=[{"role": "user", "content": "Say hello in one short sentence."}],
    extra_body={"fallbacks": ["anthropic/claude-haiku-4.5"]},
)
# `model` names the model that answered, and you are billed at its price.
print(response.model, "->", response.choices[0].message.content)
  • Your backup models are tried after every channel of the requested model has failed. Unlike the gateway's own failover, they are tried for any error the model returns, including a prompt that is too long for its context window.
  • Errors about your key, account, balance or rate limit, and refusals by OceanToken's own safety checks, are returned without trying them.
  • Every name must be a model your key can call. An unknown name is refused with 400 model_not_found before anything runs.
  • You are billed at the price of the model that answered. The x-oceantoken-model response header names it, and so does model in a non-streaming response.
  • Retry 429 and 5xx with exponential backoff and jitter, and honour Retry-After (Errors → Retries).
  • Let a non-streaming request run for the full 240 seconds before your client gives up, or stream.
  • Send an Idempotency-Key on image, video and speech requests, so a retry never runs and bills twice.
  • Log the x-oceantoken-request-id header. It finds the request in the console under Logs, and it is what to quote when you write to us.

If errors persist​

Failures are retried and moved automatically as described above. If a model keeps returning errors after your own retries, write to support@oceantoken.ai with the x-oceantoken-request-id of a few failed requests.