Check an agent change
before merging it.
Use your agent’s test results to check for repeated actions before merging a change in GitHub. Ship-Check checks the recorded tool calls against your rules.
Start with a recorded test run
- Run your agent tests and record the tool calls.
- Configure Ship-Check in GitHub to check that evidence against your rules.
- Require a passing check before merging.
Example: catch a new retry loop
You change an agent’s prompt. Your tests record it repeating the same lookup. Ship-Check checks that trace against your rules; a failed check can block the merge when you require it in your repository settings.
This is a developer setup, with no hosted browser demo. It requires your evaluation files, a RunLog API key and a trusted gate signing key. Get setup help.
Setup requirements
What you need
- A RunLog API key, stored in your repository's protected secrets.
- Your evaluated suite, workflow definition, measured report with a real tool-execution trace, and an evidence file binding those files and commits.
- A trusted issuer public key supplied independently of the pull request. Contact support if you need help obtaining and verifying the gate key.
- A reviewed Ship-Check action release pinned in your GitHub workflow.
The action guide documents exact inputs, evidence fields, and verification steps. Keep gate credentials out of untrusted pull-request code.
Ship-Check does not run your agent or generate evaluation evidence. A RunLog Benchmark report alone does not contain the tool-execution trace required for a release check.
What an approval verifies, and its limits
Understand the result
Version 0.3.0 uses the runlog.trace-gate.v3 contract. The signature covers the submitted scenarios’ loop-policy decision, commit bindings, and supplied artifact hashes. The service does not independently execute your agent or verify benchmark quality; your trusted CI harness must establish those facts.
The action fails when evidence is invalid, hashes or commits do not match, the signature cannot be verified, the decision is too old, or the verdict is not approved. Configure your repository rules to require the check before merging.
The legacy recorded-trace mode checks caller-provided records only. It does not establish measured benchmark quality. Ship-Check does not automatically post pull-request comments.
After merging, add Loop-Guard to check calls while your agent runs. For a local setup, see downloads and examples.