You’ll sometimes hear objections about how random testing can only find shallow bugs because random tests are too dumb to find really complex bugs. For one thing, that assumes that you don’t specify constraints that allow the random generator to generate intricate test cases. But even then, this paper analyzed production failures in distributed systems, looking for “critical” bugs, bugs that either took down the entire cluster or caused data corruption, and found that 58% could be caught with very simple tests. Turns out, generating “shallow” random tests is enough to catch most production bugs.
People overestimate their abilities and overestimate how well they understand the system and so they overestimate how complex bugs in that system will be.
Fuzz testing (or mutation testing) can be unreasonably effective at finding those bugs.