AI Tools·Standings
LLM Developer Tools
LLM developer tools help you build, evaluate, and monitor language-model apps; the choice turns on deployment needs and access to an API or free plan.
Use the comparison marks to check free and paid options, deployment type, supported model providers, evaluation, observability, SDK languages, and open-source code. The signal score is out of 10; API, open-source, and free-tier marks make those details quick to scan.
- 26 platforms
- 1 with a public API on this page
- 22 with a free tier
- Checked 7 Oct 2026
- 1 D
DeepEvaldeepeval.comFree open-source framework for Pytest-based AI evaluations in scripts or CI/CD.
- 2 P
Promptfoopromptfoo.devAn open-source tool for comparing prompt results across models and checking AI apps for security risks.
- 3 O
Opikcomet.comOpen-source agent logging and evaluation platform with free plan, trial, and production checks.
- 4 L
Langfuselangfuse.comOpen-source AI platform with free plan for tracing applications and managing prompt versions.
- 5 L
LangSmithlangchain.comAgent engineering platform with free plan for dataset evaluations and live production scoring.
- 6 B
Braintrustbraintrust.devAI agent observability service with traces, experiments, evaluations, and free or paid plans.
- 7 R
Ragasragas.ioA free library for evaluating AI applications, with measures for retrieval, agents, SQL, and custom tasks.
- 8 H
Haystackhaystack.deepset.aiAn open-source tool for linking reusable components into pipelines for agents and AI applications.
- 9 PPortkeyportkey.ai
An AI gateway that connects to models from different providers and supports routing, retries, and failovers.
- 10 O
OpenRouteropenrouter.aiA freemium API that connects to many AI models through one endpoint, with backup providers available.
- 11 RReplicatereplicate.com
A service for running, fine-tuning, and deploying models through an API, including image, speech, and video tools.
- 12 L
LlamaIndexllamaindex.aiDocument workflow platform that parses 50-plus file types, extracts data, and builds retrieval indexes.
- 13 L
LangChainlangchain.comFree open-source framework for configurable agents combining models, tools, prompts, and middleware.
- 14 APArize Phoenixarize.com
Free platform that traces, evaluates, and improves AI applications across web, API, or self-hosted setups.
- 15 CACrewAI AMPcrewai.com
An agent management service with prompt, visual, and code-based workflow builders and a free plan.
- 16 DDSPydspy.ai
Free MIT-licensed Python framework using typed task signatures and optimizable programs.
- 17 H
Heliconehelicone.aiAI gateway with free plan, routing, caching, logs, and debugging across over 100 models.
- 18 L
LiteLLMlitellm.aiOpen-source gateway offering one API to over 1,800 models with routing and spend controls.
- 19 P
PromptLayerpromptlayer.comA visual workspace for editing and versioning prompts, comparing models, and reviewing evaluation results.
- 20 AAutoGenmicrosoft.github.io
A free local toolkit for connecting models and APIs in agent projects on Linux, macOS, and Windows.
- 21 HF
Hugging Face Inference Endpointsendpoints.huggingface.coPaid managed service deploying Hugging Face models through APIs on major cloud providers.
- 22 CNCohere Northcohere.com
Enterprise AI workspace for automations, reusable agents, and apps grounded in company data.
- 23 FA
Fireworks AIfireworks.aiPaid cloud for training and serving open or specialized models with flexible deployments.
- 24 GGGoogle Gemini APIai.google.dev
Free cloud API service for Gemini models with Python, JavaScript, Go, and Java SDKs.
- 25 GGroqCloudconsole.groq.com
Freemium cloud service providing model and API access plus agent workflows through the web.
Places #1, #2, #3, #4, #5 on this board are open to sponsors. Take #1 · Other ways to advertise
How to choose
Which capabilities should I compare?
Compare free plan, paid from, deployment, LLM providers, LLM evaluation, LLM observability, SDK languages, and open-source status.
Evaluation means checking how well a model or app performs. Observability means seeing what happens while it runs.
Which platforms does it need to run on?
Choose the platform that matches your setup: API, web, self-hosted, Linux, macOS, or Windows.
An API lets another app connect to the tool. Self-hosted means running it on infrastructure you control; web means using it in a browser.
Can I try an LLM developer tool for free?
Check the Free plan and free-trial marks before paying. Pricing notes show whether a free plan, a paid starting price, or a time-limited trial is available.
Braintrust lists a free plan and paid plans from $249 per month. Helicone lists a free plan, paid plans from $79 per month, and a 7-day trial.
Do I need open-source code?
If you need to read or change the code, check the open-source mark and licence for the tool you are considering.
DSPy and Haystack are listed as open source.
How should I compare paid pricing?
Use the Paid from field when a starting price is published, and look for pricing-on-request notes when it is not.
Cohere North lists pricing on request. Fireworks AI has no free plan, so check its paid terms before choosing it.
Questions
What are LLM developer tools used for?
They support language-model app development, model evaluation, and observability, with comparisons covering providers, SDK languages, deployment, pricing, and open-source status.
Which LLM developer tools have a free plan?
Use the Free plan mark to find them. The list also distinguishes free software, freemium plans, paid products, and free trials.
What does self-hosted mean for an LLM tool?
Self-hosted means you run the tool on infrastructure you control rather than using it solely as a hosted web service.
Which LLM tools are open source?
Check the Open source mark. DSPy and Haystack are examples listed as open source.
How do I compare LLM tool pricing?
Check the pricing model, Free plan mark, Paid from field, and any trial or pricing-on-request note.
Some top positions on this board can be paid placements, standing where they were sold. The facts on each platform are not for sale. How we rank · Disclosure