Know Your Tooles

In the audio field, the best known practitioner of such testing is Floyd Toole, who did a great deal of such work for loudspeaker manufacturers at the National Research Council of Canada facilities in Ottawa, Canada. Toole’s work is published in the AES Journal between about 1982 and 1992. If you are curious about this sort of stuff, check it out. It is definitely worth the effort.

Anyway, I tested some loudspeakers in Ottawa under Floyd’s direction back in 1991, and was really impressed by several things. The first was that the tests were far more convincing and conclusive to us than the objective test measurements that we obtained from the anechoic chamber. The second was that the tests were fast. The third was that the three of us doing the listening (all of us were involved in the loudspeaker design project that led to the tests) achieved easy consensus, even though we differed in backgrounds, critical listening experience and musical tastes. And, finally, we obtained these results reliably with an array of recordings that we all knew quite well. In short, we could all hear the same differences. As a result, I learned more about the quality of our design efforts in four hours than I had in the previous two years of listening to various prototypes that we had built, saving me a bundle of money and causing the designer to build greatly improved speakers.

As a spinoff from this experience, I instituted ABCD tests as an exercise in a course on technical problem-solving at Berklee. One of the first times we ran it, we tested a batch of bookshelf speakers, including one Famous Brand, one Awful Brand, and two No Name speakers. We kept all of the speakers behind a screen and let all the classes listen to them as many ways as they wanted, using lots of different recordings. Two interesting things happened. The first was that at the end of one of the class sessions, when we removed the sheet and let the students identify what they had been listening to, one of them saw that the Famous Brand was D, which he had scored low. He said, “Boy, do I feel stupid! How could I have rated Famous Brand so poorly?” This is a classic example of “label bias.” Had he known the brands as he listened and evaluated, he would have changed his scoring, because of his belief that Famous Brand had superior performance. It did not occur to him to trust his ears’ preference, and that perhaps the Famous Brand speakers were not quite all they were cracked up to be. We all do this, incidentally.

The second interesting thing was that there was general consensus, across about 100 students, about the relative merits of the speakers. Our instructions were pretty vague: “Rate each speaker between 0 and 10, where 10 is perfect reproduction and 0 is wretched reproduction.” When students asked, “What do you mean by perfect reproduction?,” we just told them to use their instincts. There was some variation in results when different music was used, but the general consensus was strong. Even people who can’t tell you why they like a speaker will nevertheless generally be able to rate its quality with about the same precision as someone who can clearly articulate perceived differences.

This non-verbal “ear-brain” intelligence transcends objective physical measurements in many cases, and is often far more sensitive than the objective measurements, which means that devices with quite similar measured physical qualities may be perceived as sounding significantly different. This has led to the twin phenomena of Audiophiles and Golden Ears. The former are individuals who quite seriously believe that minute differences in physical composition of components of an audio system (such as the cables, f’rinstance) yield significant differences in sound quality. Golden Ears are engineers or producers who are noted for their ability to hear audio details that confound most of us, as in: “I can hear that the transformer on the third mic preamp has a few extra turns in the secondary winding. Please use mic preamp seven on the bass zither” or “That take must have happened after they moved the sofa out of the hall next to the studio door – the sound is a little less whuffy.”

Far be it from me to debunk such stuff. Just because I can’t hear it doesn’t mean that nobody else can. I’ve had my Audiophiled and Golden-Eared moments, and heard things that others couldn’t . We all have to have real, honest-to-God obsessive faith in our own abilities to proceed in this business. It is this faith, incidentally, that speaks to the relationship between science and art. Science is about knowing, based on what you’ve measured. Art is about feeling, about believing. Music (as opposed to sound) is not amenable to measurement, and it is not really very knowable in a scientific sense, except for the technical elements of the craft used to make it.

When we ask people, or ourselves, to make these subjective judgments of sound quality, what we are really doing is saying to them, “Don’t be objective. Using your musical intuition and feeling, let your ears just respond to the sound. Tell us what you think sounds ‘bestest’. Make up a number to rank it on a scale of ‘bestestness’.”

What we then do with our statistical constructions of their collective intuitions is to build a bridge across the gulf between knowledge and faith, trying to convert the expressions of faith (“I think Microphone 22 sounds bestest, maybe an eight, don’t ask me why!”) into quantified objective knowledge (“241 producers ranked the MediaDyne Whispersnatcher SX69 at an average score of 7.83 on a scale ranging from 0 to 10.”). This latter statement can be used to predict, in a meaningful way, how producers in general will perceive this microphone. This is one of the essences of scientific method. And it is useful information for us when we choose a microphone to buy without going to the trouble and expense of testing it ourselves.

Obviously, this bridge has limitations, but it does exist and we can use it for fun and profit if we (a) don’t take it totally seriously and (b) keep clearly in mind what we are trying to do with it, which is to shorten up an otherwise exhausting decision-making process.

• The first thing to remember is that there is no bestest.
• The second thing to remember is we can never prove issues of faith.
• The third thing to remember is that making decisions can involve an element of error that is unavoidable.