Luke Wright, Webmaster
LTW Site Sensation Awards (site closed)
Most web site award programs are owned, run, and judged by a single person. However, an increasing number of programs are starting to use panels of reviewers, for varying reasons. Both styles of reviewing have their advantages and disadvantages, and among programs of a similar standing the quality of both types of programs are usually about equal.
You will find that many top-rated programs use multiple reviewers, often between two and four, and there are many possible reasons for this. Often a single reviewer may miss a disqualifying point or an obvious mistake, which may be picked up by other members of the review panel.
Opinions of a site by individual reviewers often average themselves out, providing a more solid opinion of the site from the program's overall point of view. Some add evaluators that are fluent in a different language to the program's "home" language, opening their program up to parts of the world that previously could not access the program's main sets of awards.
One downside to this approach is that individual judges' scores have the potential to vary greatly, especially if they interpret the criteria differently to the others (due to a lack of familiarity with the criteria or other reasons).
However that does not mean that the review of a sole judge — especially one with plenty of experience — cannot be at least as authoritative as that of a panel. Experienced reviewers, more often than not, are more consistent with their interpretations of the criteria and therefore their scores are more consistent. This is often reflected in the quality of their winners' list, which is usually of a similar quality to similarly rated programs with multiple evaluators.
A single reviewer with much experience is often better than a panel of less experienced judges. One problem with a single-reviewer system is that if he or she takes an opinionated view of a site for some reason or another, either positive or negative, there is no ability to balance that bias out.
Many industry-backed awards, such as the Webby Awards in the United States and the Australian Internet Awards, use a large number of expert judges to decide their best sites each year. The panels used by other web awards are usually run on a smaller scale, and this often results in a more authoritative decision compared to sole-judge programs. However, compare some of the World's Top Awards or AS! 5.0 rated awards, where you will find examples of both multiple and sole judge programs, and I would dare you to challenge the authority of their review, no matter which category they fall into.
Karen Pimtzner, Webmistress
petalperfect Digital Photography Gallery (site closed)
This is certainly one for the debating tables, as I've seen excellent and poor examples of both (and one other!). I think it's more a question of which one is right for which award program and for which applicants.
When award givers create their award programs, they usually base their criteria on the areas of web site design that they enjoy the most. These specific areas may not be areas in which they feel they are completely qualified to evaluate others. They may feel that these weaker areas will hinder their ability to perform a fair evaluation of a web site. So they seek out other evaluators who have greater experience in these specific areas. One could assume that this would give an awards program a well-rounded and balanced review team. In the majority of cases, it does.
Award programs with multiple evaluators can evaluate sites from all different perspectives, both objective and subjective. Sites can be evaluated for design by design experts, content by literary experts, coding by technical experts. They can even be evaluated in a myriad of different languages by expert linguists.
Multiple-evaluator programs require that 3-4 individuals (on average) must make the time in their own busy schedules to visit and review a web site. They evaluate the site according to the established criteria and scoring method and come up with a score. These scores are usually averaged and a final score and award level is decided, based on this average. This can take time ... sometimes, a lot of time.
Programs with multiple evaluators can sometimes take up to a month to complete an evaluation. This timeframe can be even longer if the criteria requires multiple visits to a site by each individual evaluator. With a multi-evaluator program, an applicant can get a variety of opinions on their web site from a variety of evaluators with individual personalities, expertise, tastes and, yes, even moods. Of course, with multiple-evaluator programs, you only have to impress a "majority" of evaluators with your web site.
Dual-evaluator programs are usually formed by award-givers who have experience reviewing sites in all areas of their criteria, but they need a second or occasional unbiased opinion about a web site. If the primary and secondary evaluators are experienced evaluators, it can mean that an applicant's site will be evaluated efficiently and in a shorter timeframe. To earn these awards, you have to impress at least one of the evaluators with your site.
Single-evaluator programs are the hardest to second-guess. Evaluators in this category are, for the most part, highly skilled webmasters with expertise in all areas of their specified criteria. They are single evaluators by choice. They feel that no one knows their program as well as they do, and they choose to go it alone. For obvious reasons, these programs can have longer review times, as well. Some single-evaluator programs are the hardest to earn Gold from. You only get one chance to impress a highly seasoned award-giver.
There are also single-evaluator programs where the award-giver is a novice. They are just starting out and they haven't decided what type of award program is best for them. It takes time, experience and talent to become a proficient single evaluator. This doesn't happen overnight.
So you see, there's no iron-clad rule about which method of evaluation is the best. It depends on the individual award-givers, the criteria of the program and a lot of other elements too numerous to mention here. What's the best way to decide if an award program will give you a fair, honest and professional assessment of your website?
| 
