LangSmith alternatives for tracing, evaluation, and more

LOpen-source tracing and prompt managementAPIOpenFree

Langfuse is an open-source AI engineering platform for debugging and analyzing agent applications. Its tracing captures prompts, responses, token usage, latency, tool calls, retrieval steps, and metadata, while Prompt Management stores, versions, deploys, and tests prompts. LangSmith also traces agents, tracks costs, and supports prompt experimentation, evaluations, and deployment.

Suits: Teams seeking open-source tracing and versioned prompt management.

AOpen-source agent observability with run replayAPIOpenFree

AgentOps provides observability for OpenAI, CrewAI, AutoGen, and 400+ LLMs and frameworks. It visually tracks LLM calls, tools, and multi-agent interactions, and lets users rewind and replay agent runs. LangSmith also traces agent runs and monitors errors, latency, token use, and costs, with evaluations, prompt tools, and deployment.

Suits: Teams that want visual tracking and replay for agent runs.

DAI testing and production monitoringAPIOpenFree

Deepchecks LLM Evaluation focuses on testing, observability, and monitoring AI systems in production. It offers automated scoring, version comparison, custom and off-the-shelf properties, and golden set management. LangSmith also evaluates and monitors production interactions, with human review, LLM-as-judge checks, and custom evaluators.

Suits: Teams prioritizing automated scoring and version comparisons.

DOpen-source evaluation framework for AI systemsAPIOpenFree

DeepEval is an open-source LLM evaluation framework for building evaluation pipelines. Its evaluations are Pytest-native and can run in CI/CD or as Python scripts. LangSmith supports evaluations with human, heuristic, LLM-as-judge, pairwise, and custom Python or TypeScript evaluators, and integrates with pytest, Vitest, and GitHub workflows.

Suits: Teams that want open-source evaluations in Python tests or CI/CD.

EAOpen-source evaluation and monitoringAPIOpenFree

Evidently evaluates, tests, and monitors LLMs, RAG applications, AI agents, and machine-learning models. Its Python library provides more than 100 evaluation metrics, and automated evaluations measure output quality, safety, and reliability with pipeline-ready visual reports. LangSmith combines evaluation with agent tracing, cost tracking, prompts, and deployment.

Suits: Teams focused on quality, safety, and reliability metrics with reports.

PPrompt evaluation and AI red teamingAPIOpenFree

Promptfoo evaluates prompts across multiple models and provides a web view for comparing outputs. Its red teaming simulates users to look for prompt injections, jailbreaks, data leaks, business rule violations, and insecure agent tool use. LangSmith also supports prompt comparisons and evaluations, alongside tracing, monitoring, and deployment.

Suits: Teams comparing prompts across models or checking for AI security risks.

DOpen-source visual builder for AI applicationsAPIOpenFree

Dify is an open-source platform for building AI applications with agentic workflows, retrieval pipelines, model management, observability, and tools. Its Workflow Studio puts model calls, knowledge retrieval, tools, code, branching, triggers, and human review on a collaborative visual canvas. LangSmith is framework-agnostic and emphasizes observing, evaluating, and deploying agents rather than a visual workflow canvas.

Suits: Teams building AI apps with a collaborative visual workflow canvas.

LLM Gateway offers one OpenAI-compatible API for requests across 250+ models from 40+ providers. It can retry a request with another provider when there is an error, timeout, or rate limit, and its dashboard tracks requests, tokens, spend, latency, errors, and cache metrics. LLM Gateway is a fit when routing and usage analytics matter more than LangSmith's agent evaluation toolkit.

Suits: Teams routing requests across providers and tracking usage in one dashboard.

Manifest is an open-source LLM gateway with one OpenAI- and Anthropic-compatible endpoint for models and subscriptions. It routes requests, retries through another provider when one goes down, repairs malformed requests with Autofix, and tracks spending. Manifest suits teams focused on request reliability across providers, not replacing LangSmith's wider agent toolkit.

Suits: Teams that need request routing, provider fallbacks, and malformed-request repair.

AOpen-source workspace for AI coworkersAPIOpenFree

Agenta describes an open-source workspace where teams can build, use, and automate AI coworkers. Its homepage presents pre-built templates for engineering, customer support, sales, company knowledge, and operations, and says teams can work with AI coworkers in Slack or Telegram. LangSmith focuses on building, testing, deploying, monitoring, and improving AI agents, with tracing and evaluations.

Suits: Teams looking for AI coworker templates and Slack or Telegram use.