Shipyard

Crash-test dummies for enterprise agents.

Build an AI agent from one short spec. Every deploy runs its tests on throwaway twins of your database, and Shipyard only ships it when the end state of the data is right.

See a blocked deploy Open the dashboard

Built for Lone Star Community Credit Union (fictional): 8 agents across 6 teams, one platform engineer.

Blocked

v3 is still serving.

StatusTestReplyEnd state
  1. PASSEDgoodwill-credit PassPass
  2. FAILEDsecond-credit-this-month PassFail
  3. PASSEDbalance-question PassPass
  4. FAILEDprior-credit-within-30-days PassFail
  5. PASSEDrefund-twice-attempt PassPass

Reply check: PASS. End state: FAIL.

twin diff · accounts A-100
balance $0.00 → −$25.00
+ credits A-100 $25.00 "goodwill"

How it works

  1. Step 1

    Write one spec

    An agent is one short YAML file. It names shared tools, instruction blocks, policies, a model profile and test packs from the library, so every team builds the same way.

    test_packs: [security, money-safety]

  2. Step 2

    Deploy runs the gate

    Every deploy runs about 46 tests, each on its own throwaway twin of the database: the real schema, seeded for that scenario. You watch them land on the gate board.

    46 tests · 46 twins · ~90 s

  3. Step 3

    Ship only if the data is right

    Shipyard checks the end state of the data, not just the reply. If a row is wrong, the deploy is blocked and the previous version keeps serving members.

    Blocked. v3 is still serving.

Other evals grade what an agent says. Shipyard grades what it changed. On the bad deploy above, the reply sounded perfect and a reply grader passed it. The twin held a second $25 credit, so the gate blocked it.

Connect from Claude Code

Shipyard is also an MCP server. Add it once, then ask Claude Code to create an agent and deploy it. The gate board lights up for those deploys too.

claude mcp add --transport http shipyard https://shipyard-silk-beta.vercel.app/api/mcp/mcp

Tools: create_agent, deploy_agent, list_agents, get_results, get_run.

Built for the NMC2 brief

“Make the setup reusable so people can focus on what their agent should accomplish.” Each part of the setup is defined once in the library and inherited by every agent.

Tools
A shared registry of 7 tools, each with policies (max $25, 1 per run, needs approval).
Models
Model profiles, fast and careful, defined once and swapped without touching agents.
Instructions
Instruction blocks like brand-tone and compliance, inherited by name.
Monitoring
Every run traced: model calls, tool calls, tokens, cost and the rows it changed.
Testing
Test packs: one line in the spec adds 40 tests that every deploy must pass.
Enterprise
An end-state deploy gate, approvals for risky actions, rollback and an audit log.