Bench pressing a few hundred pounds develops monstrous quads. Or it eventually blows out knees and ligaments depending on what kind of balance you bring to the exercise. Test bench measurements of amplifiers and speakers seem similarly predestined to grotesque outcomes and pain. The latter is endured by the reviewer if his subjective assessment doesn't seem to correlate with the graphs; or the manufacturer if his creation gets buried by the measured results.
Recently, a reader hipped me to this month's Stereophile issue in which Art Dudley takes the measure on Shigeki Yamamoto's Soundcraft A-08 SET. Before reading the review, I fired off a brief e-mail to this reader. "Lemme guess. Art loved it, JA slammed it on the bench." His reply was equally terse. "Yes to both." (Having seen the measurements since, the Yamamoto is a ruthless design abomination pure and simple.)
|What else is new? Certainly not that Michael Fremer, in the same issue's review of the DeVore Fidelity Silverback speaker, closed with "Now, let's hope the measurements back me up."
It's nearly become a joke how very experienced listeners find themselves contradicted by measurements. In the end, it's not the listening impressions that ought to correlate with the measurements. It's the measurements which must correlate with the listening impressions - if the measurements are to be meaningful at all.
|That's the key. Meaningful. It's too convenient for the pro measurement guys to brush critics of measurements like myself off with a "too cheap to conduct and publish his own measurements" or "too chicken to let the world see that his hearing is faulty" slander. It's the lack of correlation that bothers me. To my way of thinking, it completely invalidates measurements as a predictive tool for end user satisfaction. Measurements add an aura of scientific rigor to the review process. Really? It's become nearly silly how often Stereophile reviewers look over their shoulders in anticipation of the de rigeur contradictions JA's results will post to theirs. What's scientific about that?
The newer S version of the Yamamoto is one of my reference amps. If JA's measurements truly meant what they seem to mean, I've just been publicly invalidated as a reviewer. As has Art Dudley. As has Sean Casey of Zu as a speaker designer who uses the Yamamoto as one of his reference amps. As have any of the folks who listen to this amp to clearly not deserve any audiophile props for their choice. Needless to say, Yamamoto-San will continue selling his amp and I will continue treasuring mine.
To my peers in the press who do publish measurements, I say continue what you're doing. I'll continue not to bother with measurements. After having spent two days with a test tone generator, hi-tech software programs and a microphone -- not in an anechoic chamber but my own listening room -- I've concluded, once again, that the insane variability of the graphs due to very small microphone placement changes bears no relation to what I'm hearing in the listening seat. If it were just me, I'd write myself off as an idiot on the bench. But seeing that JA with all his experience and seriousness of purpose does no better with his measurements (i.e. having them routinely [not!] correlate with what his writers describe), I feel vindicated that our present state of popular measurement taking simply isn't up to the standards of the free test bench we were all born with: our two ears bracketing the most powerful computer on earth, our brain.
I do believe that certain cracker jack designers like Vladimir Lamm and Roy Johnson do know exactly what to measure and how their measurements correlate to perception. Like a musician with a cracked instrument, they've learned how to compensate so that the end result matches the input data. That's the key. Compensation. In measurement terms, this means overlaying families of measurements. In isolation, a specific measurement means very little. Only when combined and evaluated against a multi-dimensional construct of "different ways of looking at the same things" does everything come together in a specific abstract model that can be encompassingly interpreted. Even then, the precise effects of the end user's room -- on loudspeakers -- can at best only be approximated but never be accurately predicted.
I invite anyone serious about this topic to exchange e-mails with Roy Johnson of Green Mountain Audio. I predict you'll come away realizing that there's far more to it than popular perception acknowledges. I'm equally certain that most designers of popular audio components will tell you that while measurements are helpful to trouble shoot and get the design process started, the concluding efforts of parts selection and circuit tweaking are all done by ear. If the folks making this stuff rely on their ears, shouldn't the folks buying their stuff -- you and I -- feel equally confident in our own ears? For me, that's a rhetorical question. I don't buy cars based on specs but test drives. I don't buy audio components based on graphs but listening. And I don't review audio components based on graphs but listening. That doesn't mean I begrudge those publications that use measurements their approach. I simply doubt that their considerable efforts and associated expenses in that regard truly serve the vast majority of their intended audience.
Granted, their audience might disagree and feel they very much benefit from the various graphs and accompanying explanations. In which case I have to say, those readers are a big bunch smarter than I. Does that mean our readers -- who suffer the absence of any measurements that would support or undermine our own reviews -- should be considered similarly - um, daft and unsophisticated as this Editor? That too is a rhetorical question of course. I rather believe that those who trust their own hearing exhibit a higher grasp and connoisseurship of what's important about this hobby than those who rely on machines to validate their own experience. But that's just me. Call me old-fashioned and not with them fast-moving times. Different tweaks for different geeks. To the measurement guys, I say drop the silly subjective review commentary altogether and just publish the graphs. Let the readers extrapolate the scientific meaning directly and undiluted by accompanying commentary of what it all means. Let them base their purchase decisions and general impressions of the product point blank on the test bench. Wouldn't that be truly scientific? (With the three-strikes-and-you're-out-law in effect, I've just exceeded my rhetorical question limit. I'm out now. Damn. I had a few more smart aleck comments in my back pocket. Next time...)