What will Skills do as models improve?

Turn possible future value into testable questions instead of predictions stated as facts.

Suggested learning time: 60 minutes.

1. 1Old model with and without Skill
2. 2New model with and without Skill
3. 3Task-level differences
4. 4Hypothesis review

Original learning map: arrows show the reading or decision sequence, not a measured execution trace.

Prerequisites

• Maintain comparisons across updates

Evidence and exercise status

This chapter is an editorial learning guide. Reading sources is distinct from executing a Skill and measuring its effect. Exercise status: not-run. No experiment logs or model outputs exist.

Learning goals

• Distinguish supplementing capability from supplying specific context.
• Separate future forecasts from measurements.

Roles in the work

• Learner: define hypotheses and grading criteria.
• AI: assist with reading and deliverable creation within the authorized scope.
• Reviewer: inspect outcomes and logs separately.

Inputs

• The input examples specified in this chapter.
• The official material and versions to verify.

Consider three hypotheses

These are future scenarios, not an established roadmap. First, Skills teaching detailed general procedures may become less valuable if a new model naturally does the same work. Second, organization-specific output formats, private procedures, and current quality criteria may still need external input even as general model capabilities increase. Third, Skills bundling scripts and validators may become more important as reusable working components than as written instructions.

Include the possibility that progress has adverse effects

Detailed reasoning guidance for an older model may be redundant for a newer one. An old workaround for a particular tool may become incorrect after an update. At the same time, agreeing explicitly on delivery format and change scope may matter more as more decisions are delegated. Both directions are conjectures. Check them after each model update by comparing disabled, current, and shortened Skill versions on the same representative tasks.

Cross models with Skill conditions

A 2×2 design tests both an old model A and a new model B with and without the Skill, separating the model's own improvement from the additional contribution of the Skill. Effectiveness on an old model does not establish effectiveness on a new one, and a new model's high score alone does not establish that the Skill is unnecessary. Compare cost, speed, safety, and difficult-case performance for every condition. Adding a shortened version creates another condition.

Can you explain your Skill in one sentence?

Finally, state who uses the Skill, with which input, to create which deliverable, and to reduce which failure. If the only answer is “make the AI better,” remove content and find a clearer focus. With a clear answer, a Skill becomes a unit for sharing and checking work agreements rather than a bag of knowledge. The design habit of stating inputs, constraints, deliverables, and verification remains reusable even as formats and runtimes change.

Forecast status

The future scenarios in chapter 10 are hypotheses, not measurements, guarantees, or a roadmap.

Workflow

1. Select representative tasks to rerun when a new model appears.
2. Write conditions under which the Skill becomes unnecessary and conditions under which you retain it.
3. Label forecasts as hypotheses until results exist.

Outputs

• A 2×2 comparison plan and operational decision criteria.

Quality checklist

• Future forecasts are explicitly labeled as hypotheses.
• You do not say that a Skill permanently changes model weights.
• You planned a condition that removes the Skill too.

Failure diagnosis

• Symptom: Recording success without observing the effect.
• Cause: Confusing expected judgments with actual outputs.
• Fix: Keep unexecuted work as not-run, clear measurement fields, and obtain raw outputs and logs before scoring.

Exercise: Design reevaluation after a model update

Follow the workflow above in order and create the stated deliverable.

Completion criteria: Keep model-update effects separate from Skill effects and avoid assertions about the future.

Status: not-run.

Source scope

Sources support feature descriptions and distributor statements in the text and catalog. They are not evidence of measured effects or popularity ranks. Verification dates record reading public sources, rather than publication or update dates. Rolling references such as main are not pinned experimental versions.

Offline experiment worksheet

Methodology reference and limits

Anthropic’s authoring guide describes baseline evaluation and iteration. It supports that general evaluation approach; the protocol, record format, and future hypotheses here are independently authored teaching designs. Their task counts and scores are proposals, and no experiment has been executed.