Decision Guide · Source-verified
Agent observability is not evaluation
A trace can show what an agent did. An activity metric can show how often it ran. Neither establishes whether the result was correct, useful, or safe for the workload.
Runtime signals and quality judgment are different loops
What the sources establish
- OpenTelemetry publishes generative AI semantic conventions for instrumenting model and agent operations.
- LangSmith documents an observability surface for traces and runs.
- GitHub's usage metrics API includes agent app activity.
- AWS documents a GitHub Actions pipeline that evaluates AgentCore workloads and fails a pull request when evaluation scores regress below a threshold.
- AgentCore Evaluations provides tool-level SkillSelectionAccuracy and SkillInstructionFollowing evaluators and emits one result per detected skill invocation.
- AWS's September 2026 release notes add AgentCore Evaluations support for the TypeScript versions of Strands Agents, LangGraph, OpenAI Agents, and Vercel AI SDK; Vercel AI SDK is TypeScript-only.
Three different questions
Activity asks how much. Observability asks what happened. Evaluation asks whether the result met a defined standard.
Evaluation as a release gate
A release gate turns a named dataset, evaluator, and threshold into a merge condition. AWS's AgentCore and GitHub Actions example demonstrates the mechanism; each team still owns the cases, ground truth, threshold, variance policy, and override procedure.
Skill evaluation depends on trace evidence
AgentCore's Builtin.SkillSelectionAccuracy asks whether the selected skill fits the task, while Builtin.SkillInstructionFollowing evaluates adherence to the loaded SKILL.md steps. Both operate at tool-call level and emit one result per detected invocation. When neither invoked-skill nor skill-content signals are present, AWS says the evaluators skip the call; missing results are therefore an instrumentation question before they are a quality conclusion.
TypeScript compatibility is framework-scoped
AWS's September 2026 release notes document AgentCore Evaluations compatibility with the TypeScript versions of Strands Agents, LangGraph, OpenAI Agents, and the TypeScript-only Vercel AI SDK. Treat this as support for those four evaluation framework paths, not as blanket TypeScript support across AgentCore or proof of TypeScript custom-evaluator authoring parity. Telemetry requirements still determine whether an evaluation can interpret a run.
Agent output is not task completion
Require an evidence chain: agent output, executable verification, tests/build/scripts, retained artifacts, and human or policy review. A successful run or a vendor-reported uplift is not proof that the repository outcome is correct.
Management trap
A dashboard full of agent activity can look like adoption while hiding review cost and failed outcomes.
Decision detail
- Instrument traces to diagnose execution and cost.
- Build workload-specific evaluations to judge outcomes and regressions.
- Use a representative evaluation set as a release gate when a regression must block a pull request.
- Do not use run counts, token volume, or trace presence as a quality score.
- Before adopting skill evaluators, verify that traces expose the invoked skill or loaded SKILL.md content required by the selected evaluator.
- Treat TypeScript support as evaluation compatibility for the four documented framework paths, then verify telemetry and evaluator workflow requirements separately.
Evidence
These claims are source-verified. We do not label this page hands-on or benchmarked because no reproducible test artifact is attached.
- OpenTelemetry generative AI semantic conventionsOpenTelemetry · checked 2026-09-13
- LangSmith observabilityLangChain · checked 2026-09-13
- Copilot usage metrics API adds agent app activityGitHub · checked 2026-09-13
- Automated agent evaluation with AgentCore and GitHub ActionsAWS · checked 2026-09-13
- AgentCore skill evaluatorsAWS · checked 2026-09-18
- AgentCore Evaluations TypeScript agent framework supportAWS · checked 2026-09-18