Testing an agent against 20 negative constraints
I gave the agent a task plus 20 "do not" rules (do not add dependencies, do not touch the public API, do not rename files…) and counted violations over 30 runs.
Constraint type | Violations |
|---|---|
File-scope ("do not edit X") | 1 / 30 |
Dependency rules | 3 / 30 |
Naming rules | 7 / 30 |
Style rules | 12 / 30 |
Hard boundaries held well. Soft style rules were the first to go. Rewording style rules as concrete examples cut violations by half.
6 days later
Related topics
- How to structure multi-project rules files to avoid prompt bloat4 replies
- How do I add a second repository to a project?2 replies
- Writing a compiler with an agent: what worked and what did not2 replies
- Using Tessel for data science: notebooks, SQL and plots1 reply
- Share a chat as a read-only link1 reply
- Keyboard-only workflow: my shortcuts2 replies