TensorZero

Overview

TensorZero is an open-source LLM gateway and operations stack for teams that want provider routing, inference data, and experimentation inside infrastructure they control. Its breadth is compelling for a self-hosting team, but the archived project is not a sound choice where ongoing maintenance matters.

A unified API covers major model providers and OpenAI-compatible endpoints, with routing, retries, fallbacks, prompt templates, schemas, and A/B testing. TensorZero also collects inference and feedback data in a database chosen by the user, tying gateway traffic to observability and optimization workflows. This is a substantial set of capabilities for a free, self-hosted platform; in practice, teams must also own deployment, database operations, and any upkeep the project no longer provides.

Key features

Routing and workflows

Fallback routing, retries, and traffic routing for A/B tests give teams ways to direct requests across providers and compare alternatives. Multi-step workflows support compositions beyond a single model call, and feedback can be attached to individual inferences or episodes. The gateway reports sub-millisecond P99 latency overhead under extreme load, though that performance claim does not offset the project’s maintenance risk.

Data and integrations

Inference and feedback data go to the configured database, where teams can query them directly with SQL. That approach favors organizations that want control over data location and access to their own analytics workflows rather than a separate vendor-controlled data store. Documented integrations include Anthropic, AWS Bedrock, Azure, Google, OpenAI, OpenRouter, Together AI, vLLM, and xAI, alongside OpenAI-compatible APIs. Support for the OpenAI SDK, OpenTelemetry, and clients for major programming languages helps fit existing development setups.

Deployment and security

The gateway and UI can run through Docker, Docker Compose, Kubernetes with Helm, or a source build. API key authentication can protect gateway endpoints other than status and health checks, so teams can secure operational routes while keeping health checks accessible. Pseudonymous aggregate usage analytics exclude application data and inference inputs and outputs; collection can be disabled. These controls are useful, but security configuration and deployment remain the operator’s responsibility.

Pricing

TensorZero LLMOps platform: free (100% self-hosted · open-source). The Apache 2.0 license and free plan make it a fit for teams able to run the stack themselves without a hosted service bill. There is no paid tier or managed offering to weigh against the work of operating it: the FAQ described no revenue model at the time and planned a managed service for the future, while the current project status is no longer maintained. Teams that need maintained software or a supported managed deployment should look elsewhere.

Platforms

TensorZero supports API, Linux, self-hosted, and web deployments. Docker and Docker Compose suit container-based setups; Kubernetes and Helm address teams already operating Kubernetes, while source builds leave deployment choices to the team.

Who it's for

TensorZero is best for technically capable teams that need a self-hosted gateway, want inference data in their own database, and can accept an archived codebase. Its routing and experimentation tools suit builders comparing models or workflows within an existing stack. It is a poor fit for teams that depend on active maintenance, a hosted service, or a project with a current commercial support model.

Pros and cons

Pros

• Broad gateway and experiment toolkit: Provider integrations, fallbacks, retries, workflows, and A/B routing bring request management and evaluation-related work into one stack.
• Direct data control: Inference and feedback data stay in the user’s configured database and can be queried with SQL.
• Flexible self-hosting: Docker, Docker Compose, Kubernetes with Helm, and source builds accommodate several deployment approaches.
• Free and open source: The free plan and Apache 2.0 license avoid a platform subscription.

Cons

• No ongoing maintenance: The site says the project is no longer maintained, and the GitHub repository is archived and read-only, making it risky for production stacks that need continuing fixes or development.
• Operational burden stays with the user: The plan is 100% self-hosted, so teams must run the gateway, UI, and database themselves.
• No present managed-service path: The FAQ described a future managed service, not an available hosted option, so teams seeking a vendor-operated deployment should choose another product.

Alternatives

• LLM gateways is the category directory for comparing gateway options.
• Kong Gateway is a paid alternative with a free 30-day trial that requires no credit card and offers no gateway limits during the trial.
• Eden AI suits teams preferring an API gateway without a subscription; API calls are charged separately at provider prices, with a 5.5% platform fee when purchasing credits.
• Envoy AI Gateway is another free, self-managed open-source project, with a local CLI for Linux and macOS and Kubernetes deployments requiring compatible dependencies.
• Helicone is an open-source alternative with a free Hobby plan and a freemium model.
• LangSmith offers a free Developer plan with up to 5k base traces per month, one seat, and community support, followed by pay-as-you-go usage.
• LiteLLM is a free-forever self-hosted option with 100+ providers, virtual keys, spend tracking, budgets, and rate limits.
• LLM Gateway provides a free tier with 200+ models across 40+ providers and a 20-requests-per-minute limit on free models; paid models are not free.
• Manifest offers unlimited self-hosting and 10,000 routed requests per month on its free cloud plan, with seven-day dashboard retention.

Verdict

Choose TensorZero if you need a capable, free gateway and experimentation stack, want to keep its data in your own database, and can take responsibility for operating the software. Its breadth and direct data control are strong reasons to consider it; its archived, unmaintained state is the overriding reason to prefer a maintained alternative for production systems.