This is the first essay in a series on reviewing and the review process. The goal is to invite dialogue on aspects of the process. In this installment, I discuss various metaphors for the review process, reject the conventional view of the reviewer as judge and suggest instead that it may be more illuminating to think of reviewers as expert witnesses. In subsequent essays, I'll explore what audio expertise consists of, how it can be cultivated and how it should be demonstrated in reviews. It is undeniable that there is a growing cynicism about the integrity of the review process. Elsewhere, I have addressed some grounds of mistrust. Srajan has recently weighed in as well. At the heart of his short essay is a distinction between neutrality and objectivity on which he rightly insists. Srajan suggests that a reliable and credible review process requires objectivity, not neutrality. Professors are not neutral about the well-being of the students they mentor, but mentoring requires occasional brutal acts of objective judgment - for the good of students as well as potential employers. The same distinctions are at work, even more so, in parenting. A lack of neutrality can compromise objectivity, but need not; and the presence of neutrality or impartiality -- an indifference to outcome -- hardly guarantees objectivity.


The notions of neutrality, impartiality and objectivity are vague of course and the differences among them controversial and subtle. For the purposes of this essay, I treat neutrality and impartiality interchangeably and mean by either that the reviewer who is neutral or imparital has no partial interest in the outcome of a review. The attribute of impartiality is therefore a lack of partisan interest.


In reflecting on the distinction between neutrality and objectivity (the absence of even a weak precommitment to the outcome of a review), it occurred to me that the value and reliability of reviews does not always depend on the reviewer's objectivity either. It is a commonplace that novice reviewers are asked to review equipment they already own. Since he already owns what he is reviewing, the novice reviewer is fully invested in the outcome. There is no pretense of neutrality, impartiality or the kind of objectivity that we would demand of a judge. Nevertheless, such reviews can be informative and the reviewer develops his craft in part by working with material that is already familiar to him.
Additionally, editors often assign products to reviewers who are already very familiar with a particular manufacturer's products. The fellow who has owned every version of the Watt/Puppy system from the 3s to the 5s and 6s is likely to make one hell of an expert witness when it comes to a review of the new 7s. The energy in some of the most exciting reviews comes from a reviewer having discovered a product or design he loves and his wanting to share his excitement about it with others. There is no neutrality in such reviews. And that's a good thing, too.


The emphasis on neutrality is distracting at best. This is a very small industry or community. There are friendships all around. There is no reason to suppose that someone cannot write meaningful, fair and informative reviews of equipment designed by friends. There may be very good reasons for trying to avoid having products reviewed by antagonists. But it hardly follows that valuable and credible reviews cannot be penned by life-long friends. The academic world could not survive otherwise. I often review grant and publication submissions submitted to granting agencies and journals by friends of long standing. Even when I sincerely hope the grant will be worthy of funding or the essay worthy of publication, I don't hesitate to act responsibly.


In the academic journal I edit, we adopt a double-blind review process but if a reviewer receives a paper that she recognizes on other grounds, we encourage the reviewer not to disqualify herself on those grounds alone. Provided the reviewer believes that she can write an informative and helpful review, we want to have it.


A range of factors can cloud judgment. A good reviewer must be vigilant to insure as best he can that his judgment can be defended on the evidence. In my experience in both the academy and audio, reviews are more often let down by the reviewer's lack of modesty than by his failure to adopt a stance of impartiality or indifference to outcome. The emphasis on neutrality, impartiality and a legalistic (as opposed to scientific) conception of objectivity comes from adopting the misleading twin metaphors of reviewing as judging and reviewers as judges. The better juridical analogy is that of a reviewer as expert witness.
In a courtroom, witnesses are retained by lawyers who are advocates of the interests of their clients. They are not themselves advocates but the evidence they offer is often marshaled in support of the client's interests. Typically, expertise is invoked when ordinary understandings fail us in the courtroom: How to understand the results of DNA testing in the OJ case or how to design and locate gas tanks in the Volkswagen case. Audio reviewers are not judges or jurors; nor are they lawyers making a case or submitting a brief. They are experts. They have the capacity to put systems together and to observe audio performance and evaluate the musical virtues, if any, exhibited by those components and systems. They may be expert at more, but we have a right to demand that they not be expert at less.
Granted, expert witnesses are witnesses for one of the parties in an adversarial setting. Nevertheless, various legal and ethical rules as well as the adversarial process itself work to keep the experts reasonably honest, to purify the evidence they present, the observations they report, and the inferences they draw. According to the U.S. Federal Rules of Evidence, expert witnesses
do not owe a duty of loyalty to the client; they are required to remain independent and prohibited from becoming the client's advocate. Their duty is to analyze, explain and offer an accurate opinion of the relevant issue at hand, They do have a duty to testify truthfully. The opinions they offer must fall within the scope of their specialized knowledge.


Though expert witnesses are not required to disclose the basis of their findings, they may be required to do so under cross examination. Not only are expert witnesses subject to cross examination, their claims are likely to be challenged by the testimony of experts on the other side. An expert's long-term credibility is
invariably at stake. If she is an expert within a particular field -- e.g. medical malpractice -- there are invariably ethical rules within the field that are designed to enable experts to perform their functions in a professional manner, to avoid conflict and to comply with the law's demands.

At the end of the day, not every expert witness will be principled, independent or truthful. If she is exposed as no more than a partisan shill -- whether for corporate clients or the plaintiff bar -- her reliability and integrity suffers accordingly. She may continue to reap financial rewards for a time as she is a very predictable witness, but her character will be impugned, and in due course she will be of little value - sadly even to those to whom she has sold her soul.


I cannot tell you how many expert witnesses, wittingly or otherwise, have done more harm than good to the side that hired them. Those who fail worst are those who misconceive their role as advocacy. By overlooking plain facts, covering up less obvious ones and overemphasizing those that lend credence to their side, they increase their vulnerability while weakening rather than strengthening their client's case.
By thinking of the reviewer within the model of an expert witness rather than a judge, we are in effect abandoning the thought that the reviewer must be neutral or objective - virtues we associate with fairness in adjudication.
There is no mention of neutrality or objectivity in the responsibilities of expert witnesses; rather, experts are to be truthful and independent and must limit themselves to areas of their expertise.


Without neutrality or objectivity, how might we insure integrity in audio reviews? It is one thing to suggest correctly that neither neutrality nor objectivity is required in order for reviews to be fair, meaningful and for the process to have integrity. It is another to explain how a review process unconstrained by such requirements might nevertheless prove trustworthy. Even if we accept -- as I suggest we do -- that reviewers are under similar duties to be truthful, remain independent and offer opinions only within the scope of their expertise, isn't it likely that without the requirements of neutrality and objectivity, reviewers will sooner or later ultimately end up in someone's pocket?


I don't think so. Of course, there are reviewers who are little more than shills for this or that manufacturer; but they are few and far between. There is simply no reason to suppose that those drawn to audio reviewing are in it for anything other than the love of music and the forms of its reproduction. This is not a high-paying job. Some no doubt seek fame and power if not fortune. Good luck.