Routing Engine¶
The Worker uses two separate NVIDIA stages plus a Puter fallback:
Classifier Stage¶
The classifier model is selected randomly from a fast verified NVIDIA NIM subset. This keeps routing quick and avoids wasting time on large models before the real answer request starts.
What it receives¶
What it should return¶
Failure fallback¶
If classification fails, returns invalid JSON, or returns an invalid tier, the Worker falls back to:
Timeout¶
Classifier NVIDIA calls have a short timeout:
Answer Stage¶
The Worker chooses one NVIDIA NIM answer model from the selected tier.
Model rotation¶
It records recent NVIDIA model usage in the Durable Object so it can avoid repeating recently used models.
For speed, the Worker first chooses from a fast-preferred NVIDIA pool inside the selected tier. If those models are exhausted by recent-use exclusions, it can still use the full verified NVIDIA pool for that tier.
Execution order¶
NVIDIA model #1
NVIDIA model #2 from same tier if #1 fails
Puter model #1 from same tier if both NVIDIA attempts fail
Puter retry models if needed
Timeout¶
Each NVIDIA answer attempt has a timeout:
So if NVIDIA is slow or not responding, the Worker moves to the next NVIDIA model quickly, then to Puter fallback, instead of hanging.
Empty responses¶
For non-streaming requests, if a provider returns HTTP 200 but empty content, the Worker treats that as a bad answer and tries the next available provider/model.
Puter Fallback¶
Puter is not the first-choice provider anymore — it is the fallback provider.
When Puter is used¶
both NVIDIA answer attempts fail
or NVIDIA returns empty non-streaming content and the second NVIDIA attempt also fails/returns empty
Where models come from¶
The Puter fallback model comes from src/model-tier-pools.js.
The Puter call¶
with:
{
"interface": "puter-chat-completion",
"driver": "ai-chat",
"method": "complete",
"args": {
"model": "...",
"messages": []
}
}
Token rotation¶
The Worker has a Puter token pool and retries another token on 401 or 500.