Claude Code Skills: How to Build, Test, and Optimize AI Agent Skills

> Speaker: Nate Herk | Duration: 16:15

Key Ideas

1. Skills are just text recipes for your AI agent — A skill is a set of text instructions that tells Claude how to perform a specific task consistently. It's like a prompt or SOP that anyone can read and understand. When invoked, the agent follows the recipe and produces reliable output every time.

2. Two types of skills: Capability Uplift vs Encoded Preference — Capability Uplift skills teach Claude something it doesn't do well by default (e.g., front-end design with good fonts and layouts). Encoded Preference skills encode your specific multi-step workflow (e.g., idea mining across YouTube, X, and web). The distinction matters because they age differently.

3. Capability Uplift skills may become obsolete — As newer, more capable models release, they may natively outperform skills designed for weaker models. Be ready to retire these when the base model surpasses them.

4. Encoded Preference skills are durable — Because they encode your unique processes, business logic, and preferences, they remain valuable across model generations. A new model won't be trained on your specific workflows.

5. Anthropic's Skill Creator is a meta-skill — It packages all of Anthropic's best practices into one skill that can create, modify, evaluate, benchmark, and optimize other skills automatically.

6. Evals measure and improve skill quality — Provide examples of ideal output, and the eval system tests your skill against them. Evals catch regressions (skill works worse with new model) and spot growth (skill becomes unnecessary).

7. Benchmarks give quantitative metrics — Pass rate, execution time, and token usage with and without a skill loaded. Run these whenever a model updates or after iterating on a skill.

8. Trigger tuning fixes misfires — When you have many skills, the agent may call the wrong one. Trigger tuning rewrites skill descriptions so they're invoked more accurately from natural language.

9. Skills improve with use and feedback — Each feedback cycle makes a skill stronger. Projects with rich context (existing scripts, brand assets, channel data) amplify skill effectiveness.

10. The future is natural language skill specs — Anthropic expects that eventually a plain description of what you want will be enough, with the model figuring out steps, rules, and format on its own.

11. Skill files use YAML + Markdown — A skill .md file has YAML frontmatter (name, description, invocation settings, argument hints) and Markdown body with context, instructions, and references to scripts.

Transcript

What is a Skill?

Skills are text-based recipes that instruct an AI agent on how to complete a specific task consistently. They are stored as Markdown files and can be used in Claude Code, Claude Desktop, or VS Code. A skill might tell Claude how to write internal communications, mine video ideas, or generate reports. The key insight is simplicity: skills are just text that anyone — even a non-technical person — can read and understand.

Two Types of Skills

There are two fundamentally different types of skills:

Capability Uplift Skills are prompts that teach Claude to do something better than its default behavior. For example, a front-end design skill provides knowledge about good fonts, color schemes, layouts, and background elements. Without it, Claude might produce generic, "AI slop" websites. With it, results look significantly more professional.

Encoded Preference Skills are sequential workflows specific to you. For example, an "idea mining" skill first checks YouTube comments, then looks at niche videos, then analyzes AI trends on X and the web. It spins up two parallel agents (YouTube and research), sends their outputs to a main agent for scoring and cross-referencing, and finally generates video ideas. This process is unique to the user's business.

The critical difference: capability uplift skills may fade as models improve (Opus 5 might be better at design without a skill than Opus 4.6 with one), while encoded preference skills stay durable because they encode your personal process.

Anthropic's Skill Creator

Anthropic released an official Skill Creator skill that consolidates all their best practices — including content from a 33-page PDF on skill fundamentals, planning, testing, iteration, distribution, and troubleshooting. It can:
Create new skills from scratch
Modify and improve existing skills
Measure skill performance via evals
Run benchmarks
Optimize skill descriptions for trigger accuracy

Evals

Evals let your agent evaluate skill quality and make improvements. You provide examples of ideal output, the system tests prompts against your skill, and compares results. This shortcuts the manual feedback loop.

Two reasons to use evals:
1. Catch regressions — A model update might cause a skill to perform worse. Evals provide early warning signals.
2. Spot growth — A new model might perform better without the skill entirely. Evals help you decide when to retire a skill.

Benchmarks extend this with quantitative metrics: pass rate, execution time, and token usage — with and without the skill loaded.

Trigger Tuning

With 10+ skills in a project, you may experience false triggers or misfires. Trigger tuning analyzes skill descriptions, tests various natural language prompts, and rewrites descriptions for more accurate invocation. Testing showed significant improvement in both test and train scores after trigger tuning, though results aren't yet perfect.

Building a Skill Live: YouTube Weekly Roundup

Nate demonstrated building a "YouTube Weekly Roundup" skill from a vague natural language prompt. The process:

1. Vague prompt: "Create a skill that looks at my videos from the past week, analyzes comments, views, engagement, and gives me a PDF report with insights, strengths, weaknesses, threats, opportunities."
2. Clarifying questions: The Skill Creator asked about time window (rolling 7 days), report sections, and PDF styling (use brand assets).
3. Plan generation: Without any technical specification, the system created a full implementation plan including scripts, templates, and an eval step.
4. First iteration: Produced a PDF with correct structure but inaccurate data and missing sections.
5. Feedback and second iteration: After honest feedback about data quality, the system revised its data fetching approach, reused existing project scripts, and produced a significantly better report with accurate stats, SWOT analysis, comment analysis, competitor context, and AI trend tracking.

The final skill file included YAML frontmatter, referenced external scripts (fetch YouTube data, prepare data, render report), and could be invoked simply by saying "do a YouTube weekly roundup."

Final Output

The polished report included: video statistics, executive summary with key takeaways, per-video breakdown with metrics, SWOT analysis, top comments and audience signals (including video requests and pain points), competitor context with channel comparisons, and trending AI topics. Built in approximately 20 minutes.

Practical Exercises

Exercise 1: Install the Skill Creator
Open Claude Code. Run , manage plugins, search for , and install the official Anthropic version. Restart Claude Code and verify by asking: "Do you have the skill creator skill? What does it do?"

Exercise 2: Build an Encoded Preference Skill from a Vague Prompt
Switch to plan mode. Describe a recurring workflow in 2-3 sentences of natural language — no technical details. Let the Skill Creator ask clarifying questions, generate a plan, build the skill, and run evals. Observe the full lifecycle.

Exercise 3: Run Evals and Benchmarks
Prepare 3-5 examples of ideal output for a skill. Ask Claude to run the eval process. Then benchmark with and without the skill loaded. Compare pass rate, time, and token usage.

Exercise 4: Trigger Tune Multiple Skills
With 3+ skills in a project, run trigger tuning. Write 5 natural language prompts per skill. Compare before/after descriptions and test invocation accuracy.

Exercise 5: Iterate with Feedback Loops
Run a skill, identify 3 specific issues in the output, provide honest feedback in plan mode, and let Claude revise. Repeat at least twice to see improvement.

Best Quotes

> «It's basically just a recipe. So, that when you ask your agent to make you, for example, a LinkedIn post, it will read the recipe and it will get it right every single time.» — Nate Herk

> «It is literally just text that you could read that an intern could read. Anybody could read and understand what's going on in the skill.» — Nate Herk

> «Over time, a natural language description of what the skill should do may be enough with the model figuring out the rest.» — Anthropic (quoted by Nate Herk)

> «And I really think that this word 'may' should actually have been 'will'.» — Nate Herk

> «If you ask Claude without a front-end design skill to build you a website, it could do it, but it might just look very generic. It might look AI slop as they call it.» — Nate Herk

> «With an encoded preference skill, these will probably stay pretty durable and accurate because the process is very specific usually to you, which Opus 5 won't be trained on most likely.» — Nate Herk

> «You could either take time and learn this or you could just give your agent the skill creator skill and all that information is already in there.» — Nate Herk

> «This is such a good example of why using a project more and using a skill more makes it stronger because some of the pieces that I already had in this project it's able to reuse.» — Nate Herk

> «Most people that are using skills right now are actual just like executives and managers and operators. They're not engineers, which means we're really good at being able to explain what we want, the metrics we need to hit, and why we need that, but maybe not all of those technical nitty-gritty details.» — Nate Herk