A couple of weeks ago, some colleagues of mine and I went to a bar after work. It was a typical bar atmosphere, perhaps slightly amplified by it being a Friday, with loud music and louder patrons. Despite some difficulty hearing each other, we got chatting over beers. At some point Adam, someone I’d worked with closely in the preceding weeks, was describing a process by which coding assignments - programs written by students to solve an assigned problem - were automatically checked at NYU. The process automatically ran the submitted programs, systematically trying malformed inputs and edge cases; things like replacing an expected number with a string, or a divisor with zero. This piqued my interest, and I responded enthusiastically, “Oh, they’re fuzzing ‘em!” He looked at me quizzically, and kindly replied “No… It’s actually the opposite of fuzzing.” While ‘opposite’ perhaps too strongly implies dichotomy, he was absolutely correct. And I knew that. So what happened?
For the past year, I’d been conducting a review of fuzzing technologies, reading papers and analyzing them in a variety of ways. It’s valuable here to note that nearly every paper I’d read introduced fuzzing as a form of software testing, and made a point of explaining that the only real definitional component of fuzzing is the random generation of inputs. So despite being well aware of fuzzing’s random nature, and despite Adam’s description of the NYU system as very much not random, my knee jerk reaction was in the ballpark, but not a hit. I’d gotten the superclass correct, but not the subclass. Software testing, yes; fuzzing, no. At this point, I’d consider myself moderately ‘trained’ in fuzzing, and as we are what we do, and in the face of the noise, the socially aggressive nature of bars and the speed of the interaction, I fell back on that training… and missed the mark.
In social situations, missing the mark, when committed in kindness, is rarely a serious issue. However in other situations, like during physical combat, it can be quite serious; the mark to miss might be a fist flying toward your face. Yet despite the different scenarios, there’s a parallel which I failed to observe prior to thinking about my response to Adam. When you learn how to block a punch, the most immediate indicators of success are whether or not you made contact with your opponent's hand, and whether that hand ended up making contact with its target. However this is somewhat naive. A punch originates from an arm, which originates from a torso, which is based on legs, which in turn provide a connection to the great base; the ground. A more sophisticated view of a punch would place it within this greater system, and suggest much more valuable indicators of success. Did my block disrupt their connection to the ground? Did my block leave that system in a state from which it can easily attack again, or have I substantively degraded its ability to cause damage?
Yet for a variety of reasons, this is not a particularly common way of viewing the act of blocking. Training tends to emphasize contact with the hand, and rote memorization of movement, complete with an opponent’s expected reaction. And that opponent is at the same time quite likely being trained in how to throw a will be blocked punch. Despite these shortcomings, this type of activity is not without value. In the event of being surprised by an incoming punch, it is much better to throw up something, than nothing, even if the movement may be based on misconceptions, or upon an incomplete view of the situation.
A punch involves most obviously a hand, but as connected components range away from the hand and toward the ground, the obviousness of their relation to the activity diminishes. Despite this, the hand is member of a single system. A punch must therefore obey the rules of the entire system, rules which may not be obvious from simply examining the punching hand. In this way, the total system’s properties are as inherent in the punch as they are in any other action undertaken by the system. The whole system here is a superclass and the punch a subclass which inherits many properties and routines from its parents. In ignoring the system, and focusing on the punch, we ignore all properties of the system not explicitly defined in the ‘punch’ class; a mistake with potentially grave consequences. Rote training of blocks has the unfortunate side effect of emphasizing the subclass, and not the superclass.
Now how does this relate to fuzzing? When responding to Adam’s remark, I fell into this trap. My rote training in fuzzing had inappropriately emphasized its specifics. It had become more important in my mind than the system to which it belonged: software testing. Though I was in the right arena, I had thrown up a block designed to make contact with the hand, not one aware of the hand’s superclass. Had my training in fuzzing emphasized its place within its system, I’d still know the same literal information, but might also have been able to successfully block. I might have been able to identify the NYU system as non-random, and therefore obviously not fuzzing. Most importantly, I may even have been able to do all of this out of the blue, in the loud, stimulating, bar environment.