How well do agents use test and verification techniques? via danluu.com

Another banger of an experiment by Dan Luu on LLM behavior against conventional wisdom.

My main takeaways here are:

• generic tutorials-as-skills are not doing what you think and are probably not helpful
• specific workflow skills are helpful
• different models on different settings behave different, you have to constantly recalibrate (e.g. the follow-up to this very article, based on results from Astra)
• measure, benchmark, validate

The specific results are interesting but if you’re in a hurry you can jump down to the “Skills” heading and the subsequent section.