Ask the Experts, Question 4:

Is an award program with an evaluation staff better than those without one?

by Míc Miller, Webmaster, The Beeline
21 December 2001, update 7 March 2003

Print Version

Luke Wright, Webmaster LTW Site Sensation Awards (site closed)

Editor's Choice

Most web site award programs are owned, run, and judged by a single person. However, an increasing number of programs are starting to use panels of reviewers, for varying reasons. Both styles of reviewing have their advantages and disadvantages, and among programs of a similar standing the quality of both types of programs are usually about equal.

You will find that many top-rated programs use multiple reviewers, often between two and four, and there are many possible reasons for this. Often a single reviewer may miss a disqualifying point or an obvious mistake, which may be picked up by other members of the review panel.

Opinions of a site by individual reviewers often average themselves out, providing a more solid opinion of the site from the program's overall point of view. Some add evaluators that are fluent in a different language to the program's "home" language, opening their program up to parts of the world that previously could not access the program's main sets of awards.

One downside to this approach is that individual judges' scores have the potential to vary greatly, especially if they interpret the criteria differently to the others (due to a lack of familiarity with the criteria or other reasons).

However that does not mean that the review of a sole judge — especially one with plenty of experience — cannot be at least as authoritative as that of a panel. Experienced reviewers, more often than not, are more consistent with their interpretations of the criteria and therefore their scores are more consistent. This is often reflected in the quality of their winners' list, which is usually of a similar quality to similarly rated programs with multiple evaluators.

A single reviewer with much experience is often better than a panel of less experienced judges. One problem with a single-reviewer system is that if he or she takes an opinionated view of a site for some reason or another, either positive or negative, there is no ability to balance that bias out.

Many industry-backed awards, such as the Webby Awards in the United States and the Australian Internet Awards, use a large number of expert judges to decide their best sites each year. The panels used by other web awards are usually run on a smaller scale, and this often results in a more authoritative decision compared to sole-judge programs. However, compare some of the World's Top Awards or AS! 5.0 rated awards, where you will find examples of both multiple and sole judge programs, and I would dare you to challenge the authority of their review, no matter which category they fall into.

Karen Pimtzner, Webmistress
petalperfect Digital Photography Gallery (site closed)

This is certainly one for the debating tables, as I've seen excellent and poor examples of both (and one other!). I think it's more a question of which one is right for which award program and for which applicants.

When award givers create their award programs, they usually base their criteria on the areas of web site design that they enjoy the most. These specific areas may not be areas in which they feel they are completely qualified to evaluate others. They may feel that these weaker areas will hinder their ability to perform a fair evaluation of a web site. So they seek out other evaluators who have greater experience in these specific areas. One could assume that this would give an awards program a well-rounded and balanced review team. In the majority of cases, it does.

Award programs with multiple evaluators can evaluate sites from all different perspectives, both objective and subjective. Sites can be evaluated for design by design experts, content by literary experts, coding by technical experts. They can even be evaluated in a myriad of different languages by expert linguists.

Multiple-evaluator programs require that 3-4 individuals (on average) must make the time in their own busy schedules to visit and review a web site. They evaluate the site according to the established criteria and scoring method and come up with a score. These scores are usually averaged and a final score and award level is decided, based on this average. This can take time ... sometimes, a lot of time.

Programs with multiple evaluators can sometimes take up to a month to complete an evaluation. This timeframe can be even longer if the criteria requires multiple visits to a site by each individual evaluator. With a multi-evaluator program, an applicant can get a variety of opinions on their web site from a variety of evaluators with individual personalities, expertise, tastes and, yes, even moods. Of course, with multiple-evaluator programs, you only have to impress a "majority" of evaluators with your web site.

Dual-evaluator programs are usually formed by award-givers who have experience reviewing sites in all areas of their criteria, but they need a second or occasional unbiased opinion about a web site. If the primary and secondary evaluators are experienced evaluators, it can mean that an applicant's site will be evaluated efficiently and in a shorter timeframe. To earn these awards, you have to impress at least one of the evaluators with your site.

Single-evaluator programs are the hardest to second-guess. Evaluators in this category are, for the most part, highly skilled webmasters with expertise in all areas of their specified criteria. They are single evaluators by choice. They feel that no one knows their program as well as they do, and they choose to go it alone. For obvious reasons, these programs can have longer review times, as well. Some single-evaluator programs are the hardest to earn Gold from. You only get one chance to impress a highly seasoned award-giver.

There are also single-evaluator programs where the award-giver is a novice. They are just starting out and they haven't decided what type of award program is best for them. It takes time, experience and talent to become a proficient single evaluator. This doesn't happen overnight.

So you see, there's no iron-clad rule about which method of evaluation is the best. It depends on the individual award-givers, the criteria of the program and a lot of other elements too numerous to mention here. What's the best way to decide if an award program will give you a fair, honest and professional assessment of your website?

If you follow the above, you can be assured that whatever type of evaluation staff an awards program may possess, it is the "best" type for that particular awards program.

Jenny Goellnitz, Webmistress
An Arete Wave of a Site (program closed)

Whether an award program uses a panel or a single judge, the real factor that comes into play in making an award program a quality one is the experience, dedication, and level of expertise the judge (or judges) has in the area they are being asked to evaluate.

Experience can be indicated by general qualifications to evaluate web sites (such as HTML knowledge, or knowledge in Flash) and how long the person has been judging web sites for awards. Dedication can often be seen in membership in an ethical type of organization or by an award rating, both good indicators that the person takes their program and the judging of applicants seriously.

Yet another important aspect is judges should be well qualified to make subjective decisions in the area they are being asked to evaluate. Evidence of this can most often be found on the judge's web site; if they are being asked to evaluate for a "cutting edge" award, then his or her web site should be "cutting edge" to show he or she has knowledge and is an expert in that area.

A panel of judges offers the advantages of what you'd get at a good pot luck dinner if everyone brings their best dish — a diverse meal that showcases everyone's talents. Most award programs have some subjectivity to them; what I consider to be "a good color scheme" may differ from what you consider a good color scheme, and having a group of judges may help average things out and perhaps make the final score more objective.

A panel made up of carefully selected judges who work well together can be a real asset to an award program.

But, there is really no reason why well qualified individuals cannot be good judges on their own. Usually the site's creator, for no one knows the criteria as well as the evaluator who wrote it. The individual award givers shoulder all the responsibility for award decisions, and if they are serious about their program, their desire to make a mark as serious award program evaluators will show with a winners' list that really fits their criteria. In short, they'll work hard to make their winners' list shine too.

In the end, that's probably the best way to tell how well an award evaluator or panel of evaluators are doing — study the criteria, and then have a look at the winners' list. Looking at the winners' list gives you an idea of the program's standards and how good a job the judge or judges are doing in holding sites to the criteria set forth by the award program.

Lynn Pilewski, Webmistress
Hidden Pearls, Gems of the Internet Awards (award program closed)

Not necessarily. The value of awards should not be judged by whether a site is evaluated by an individual or a panel of judges. Some single judge awards are incredibly difficult to achieve, while some awards that are judged by a panel are almost give away awards. So, the number of people evaluating your site for an award is almost a non-factor.

How then should you go about judging the value of an award? In three ways:

First, look at the award site itself. Is the design of the site attractive and functional? Would you be proud to link to this award site? Is the actual graphic attractive enough to make you want to display this award on your pages? The design of an awards program is the first impression that a visitor receives. If the program designer has paid meticulous attention to detail in his or her own site, then you can expect that the same attention will be paid to your review.

Second, look at the criteria for winning the award. Criteria should clearly outline what the award program looks for in its winners. It should completely explain all rules and regulations pertinent to the award. The best and most valuable award programs make their requirements crystal clear to anyone who takes the time to look. The program will also list the qualifications of their evaluators, whether an individual or a panel. They take the time to let the applicant know what the application process entails. In short, they treat their visitors with courtesy. As an applicant, you can expect to be treated with the same respect.

Finally, look at the winners' list. An award program can best be judged by its winners' list. All of the winners should be 100% compliant with the site's stated criteria. The winners' sites should be presented in a gracious manner. If you would love to see your site featured on this list along with the current winners, you can bet that this program is a "good one".

An award program should not be judged by the number of evaluators, but by a combination of factors that, together, add up to its rating of excellence.

Werner Blankenagel, Webmaster
Blanki's Home (program closed)

A clear question — a clear answer? No.

Why?

An award program should always have a personality. This is only achieved when the subjective opinion of a single award master is behind the program, and the opinion is clearly expressed in the laudatios that are sent to the winners.

It seems to have become a fashion to improve an award program by having more award masters on the judging panel, especially high-profile ones. But does the quality of the award program improve as a result of this?

I don't believe it does for several evaluators, who often are evaluators at other award programs too, dilute the quality of the laudatios. In the end they all begin to sound alike. It should be the efforts and experience of a single award master that improves the quality of an award program, and fulfills its purpose: Personal recognition for the done work by a webmaster.

Debbie Sharp, Webmistress
Mesweet's Awards

What it boils down to, when you apply for an award, is that you are asking for an opinion. When you apply for an award that has a single judge, you are asking for a single opinion. When you apply for an award that has several judges, you are asking for several opinions.

I have heard that some people think award programs with several judges are better, because they think the scoring is fairer. In more cases then not, the judges who evaluate for other sites already have their own award program too. Chances are you're also going to apply for their awards.

What if you have already been turned down for an award by a judge at a certain site? Well, off you go to apply for another award at a different site and lo and behold there on the panel is the same judge that has just turned you down for an award. Think about it. Do you think that judge will give you a high score? If a site has several judges, just one really bad score can mean you're not getting an award even if you got a good score from some of the other judges.

When I created my awards program, I knew that if I was going to have the winners display my award at their site, then I wanted it to be my single opinion that put it there. It's my award that I created. Why would I want other opinions on who to give the award to?

Ahhh, but there can be flaws with a single judge. What if there is a conflict of interest and you cannot evaluate a site with impartiality. You want your program to be honorable and fair, and this is why I decided that I did need other opinions from time to time. Although I am a single judge and usually evaluate sites by myself, there are times when I have to step down and let my evaluating team take over. Since I have a foot in both worlds, how could I possibly say one is better then the other.

A single judge with one opinion or several judges with several opinions, it really comes out the same in the end. You either will or will not win the award.

Will Harbeson, Webmaster
The Medals Of Excellence

The question of "lots of judges" has always been a difficult one for me. I am a single judge program with the occasional "gut check" added by my lovely wife. She probably has the better eye in the family but I've not been able to convince her to join me in looking at each entry. So, If I could get Helen to join in my judging efforts I would be a multi-judge award program in a heartbeat.

I also serve as a judge for TMC Designs, an AS! 5.0 rated program, so I understand the need for serving as a judge outside my own program. Any WTA or highly rated AS! program has hundreds of applicants each month. My current volume of entries is low in comparison so, in my opinion, higher rated programs have little choice but to bring in some help in the form of "jurors." At this point my program has only a few entries a month and I can get to them myself. So, which is better?

The great single factor in deciding the question is the quality of the individual judge. Are their qualifications listed on a judges bio on-site? Is their experience listed there as well? How many programs might this judge work for? If it's numerous then really how much time does the judge have to make a quality decision on the entry or are the judges just a jury panel with no more expertise than simply sitting there?

I've seen a few award programs with an expert for content and another for Flash or an accomplished artist for graphics, etc. I like this judging format very much. It allows for a detailed review by a panel of experts in their own field and gives each judge valuable additional time to spend on a site. However, I told you this is a difficult issue for me. The dedication and expertise of many single judge programs allows for a complete review without assistance.

So, my bottom line is this: As always, look over a program before you apply there. Do you like the qualifications of the judges and/or the jury? Have you visited their winners list and do you like the sites you see there? Are they awarded in a consistent fashion? Have you read the qualifications of the single judge or the list of qualifications of each judge?

Personally I feel a great responsibility as an applicant for any award. I must do my own homework before I apply. It's like buying a car. I don't just pick the pretty red one. I want to know who made it, who's selling it, where I can get it fixed and how long it might run well.

We should ask the same of an award giver: Does the award giver (and their mechanics) show expertise in the making of the program? Are they members of an ethical organization (repair shop) if you need something fixed? How long have they been doing this? Will it run for a while? The questions are the same for a single judge, a multi-judge or a jury based program. Choose wisely and you will be a happy consumer.

Jim Docherty, Webmaster
Red Stag Awards

In a word no, but let's qualify that. What is most important here is the quality of judging and not the quantity. A program sporting a half-dozen judges with no experience, poor taste in design or little attention to detail will never be as meaningful as one that has a single judge of impeccable credentials. Needless to say, the judge (or judges) plays a direct role in the overall quality of any program and ultimately determines the quality of the winners list.

Equally important is the degree to which the judges will uphold the standards of a program. If the only judge is also the owner, chances are good that the purpose is well understood and the criteria will be followed. However multiple judges carry their own brand of what is important ... and that's not always in concert with the program owner's vision. Once again, the quality of these judges (particularly when averaged) helps insulate against a particular bias or inconsistency.

That said, a system with multiple (quality) judges can serve a number of important roles for an award program:

Remember that we're still fighting an image that our awards are simply giveaways by individuals. A list of qualified judges can establish a program as something more than the opinion of an amateur.