LangSmith

Trace, evaluate, and monitor LLM apps: nested spans, datasets, prod feedback

MLOps & LLMOps · Intermediate · ~5 min

In plain English

A place to see what your LLM app actually did: the prompts, the chains, the tool calls, the costs, and how a change scored on your test set.

Why it's worth your time

Tracing and evals are the two things that turn prompt work from guessing into engineering.

If you remember three things

Traces show the rendered prompt, not the code that built it
Datasets plus evaluators give you a regression gate
Production traces become tomorrow's test cases

Overview

LangSmith is the observability and evaluation platform for LLM apps, from the LangChain team but usable with or without LangChain. It captures every chain or agent run as a trace of nested spans with latency, token, and cost data, so you can see exactly what happened inside a request. It also provides datasets and evaluators for offline testing, a prompt hub for versioning prompts, and production monitoring with feedback capture.

In an interview

LangSmith is observability and evals for LLM apps. Every request becomes a trace of nested spans showing latency, tokens, and cost, so you can debug what the model and tools actually did. Offline, you build datasets and run evaluators — LLM-as-judge, heuristics, or human — to score changes before shipping; online, you monitor production and collect feedback that loops back into new test cases. It works with any stack, not just LangChain.

Production defaults

Trace: everything in dev, 100% of failures plus a sample of successes in production
Datasets: build them from real traces. Curated production failures beat invented examples
Gate: run the eval suite in CI on every prompt change
Privacy: redact before sending. Traces contain user data by definition

What breaks

Traces are noisy and unused — Sample successes, keep all failures, and tag by feature so you can filter.
Evals pass but users complain — The dataset isn't from real traffic. Refresh it from production.

Watch it explained

What Is LangSmith? Explained in 5 Minutes — LangChain, 5:24