The Role of Evaluators
by Richard Berends, Webmaster
The Maestro Awards of Excellence
15 November 2000
Print Version
The Internet has become one of the most popular hobbies in the world, and exactly the same is true about creating websites! Yes, creating your own website is easier than you think! It is a pastime already practiced by millions across the globe. With the help of the many services freely available on the Web, almost anyone can create their own website!
Ever since the beginning of the Internet, the quest for improving it has been a reality. New features and gadgets are added every day. As a result, the entire Internet is constantly under construction and on the move! It's a dream come true for website developers and designers, especially when it generates revenue, but a nightmare for site evaluators who have to deal with the new developments. However, the basics will NEVER change. This will become clearer as you read this article.
The Human Factor
You need to understand that your website gets evaluated by human beings — not machines — when you apply for awards. There are some automatic evaluation programs available, but they are far from perfect. They just check the basics, such as HTML coding and spelling (and the correctness of these is even in doubt!). You will soon realize that only human beings can conduct the perfect evaluation!
In a survey conducted earlier this year, I asked the owners of award programs to list 5 of the best and 5 of the worst experiences they have had as website evaluators. Almost all of them listed the same negative experiences, but their positive answers were uniquely different. (See the Grief and Sigh of Relief Survey later in this article). This underlines the human factor in award programs.
Types of Evaluators
You could say that an award program is an extension of the Webmaster's personality! In most cases, the Webmaster is the sole evaluator for his or her program. Although some award programs have review panels, members of such panels usually fit well into the profile of the awarding website.
Let's take a look at who you can expect to review your website. The first type of evaluator is not only the most common, but also the most popular. They represent almost 80% of all the award programs on the Web. Many of these programs started out as an extension of an existing homepage or as a part-time hobby, once the Webmaster gained some skills in web design. The evaluator judges websites mostly on its looks and content, but nevertheless is still able to make a fairly reasonable decision.
The second type of evaluator looks at sites more aggressively. Many of these award programs developed from the Webmaster's experience in web or graphics design. Additional elements are evaluated, such as the design, content, coding and use of technology. The Webmaster evaluates the majority of the sites, but in some cases you may find a small panel of evaluators, or colleagues (in case of a small design bureau), who assist in doing the job. These evaluators are interested in furthering their own knowledge of design, and a new approach to web design will definitely get their attention. The evaluations are done in a more professional way and the results are fairly accurate.
Finally, you find the professional evaluators who make it their job to ensure that website excellence persists. These evaluators look at all aspects of a website, pinpointing small problems within the core structure. Content and design are just two of countless elements being evaluated when they judge a website.
Experience
When you search the Web for website awards, you are most likely to find some award indexes, a few award programs, and some awards on display by the Webmasters who won them. If you haven't already found them, you might want to drop into Award Sites!, the premier awards rating program on the Web, and Website Awards, home of the popular Website Awards Worksheet.
You will find some very experienced award programs on the Web. The quality of these awards are determined by the quality, content and age of their programs. If rated, they will usually have a high rating.
Winning high rated awards is very difficult. One of the reasons for this is that the evaluators have built up a lot of experience in reviewing websites. Their trained eyes can do the task quickly and efficiently, and they can smell a problem even before they've seen it!
Evaluation Panel
The other day I saw a posting in the Award Sites! Forum which asked for an article about evaluation panels. I will not go into any depth on this subject as there may already be an article in the making on this issue by another award master. However, the importance of an evaluation panel cannot be ignored.
An evaluation panel is a relatively new phenomenon. Many are based on friendships inside the awards community, with people sharing the same interests and exchanging knowledge and experience, and others may be a small business partnership in web or graphic design. Whatever the case, a good and experienced evaluation panel can greatly improve the consistency of an award program.
Some advantages of an evaluation panel are:
 |
Consistency in scoring is almost guaranteed. The applicant can expect a more accurate score. |
| |
|
Socially better. You get to know your co-evaluator and make closer friends. |
| |
|
Relieves the pressure and workload of the award master. You can pay more attention to critical issues within your award program. |
| |
|
The capacity of your program increases, which means you can handle more applications. |
| |
|
Your program appears more professional to the public and applicant. |
Some disadvantages of an evaluation panel are:
|
Disagreements are more likely among your evaluators on certain issues. |
| |
|
Time to evaluate websites will be much longer. You are now depending on scores from co-evaluators. |
| |
|
Maintaining a high level of professionalism is very difficult and requires dedication at all times. People will expect that from you. Your winner's list can make or break it. |
I can think of more reasons for an evaluation panel, but I want to keep it brief. I may expand on these reasons in a sequel to this article. For now I would like to briefly explain how evaluators feel when they review websites. In particular, what makes them feel good and what drives them nuts.
Grief and Sigh of Relief Survey
A while ago I asked various award masters to send me a list of their best and worst experiences as evaluators. You'll be surprised to see that almost all the negatives are violations of the common criteria you find in standard award programs.
In no particular order, the top 5 negative experiences that cause grief are:
|
No "thank you" after giving an award. It's important to remember the evaluator spends his or her free time to review your website — free of charge — so the least you can do is say "thank you." |
| |
|
The pop-up's and advertisement horror. It really looks like everyone wants to make a quick buck and they forgot about the Internet's real intention and purpose, which is to store information. |
| |
|
Not reading the criteria. Some applicants believe that all award programs use the same criteria, and this drives evaluators completely nuts. "They would have won the award if they had paid attention to the criteria." So beware dear readers, the dogs are unleashed.... |
| |
|
People seem to forget there is more than just one type of computer or browser. Using excessive Java, technology or rare plug-ins in combination with jiving gif animations is a recipe for problems on some computers and browsers. Add in a little extra load-time, and we have a winner! |
| |
|
People who evaluate the evaluator's website after being awarded a colour or level of award they don't like. Also, comparing award programs that have the same rating at Award Sites! (i.e. one program awarded gold and the other bronze). Who is the evaluator here? This is extremely unfair to say the least. |
In no particular order, the top 5 positive experiences that make evaluators happy doing their jobs are:
|
We've met many good ol' friends who share the same interest or grounds. I met my best friend this way, and I believe others will do so in the future. |
| |
|
A "thank you" which makes you cry instantly, no matter how humble the result was. Making someone happy is just one reason why we are doing this! |
| |
|
Effort being made on already awarded sites. It makes evaluators happy when applicants follow their advice. It boosts their efforts in trying to improve the Web! (One of the negative points was that some applicants blatantly applied again without changing a word in their sites) |
| |
|
Being in the hot seat. "I love being asked for an opinion" also appeared on various occasions. For some reason evaluators love to be "needed" or "required." |
| |
|
Evaluators get to watch websites become better and better because someone wants an award. They tell a friend. That friend tells another friend. Soon all these websites are trying hard to make their homes on the Internet the best possible so they can win an award. |
There were many more reasons and my apologies for not listing them all. The above reasons were given the most emphasis by the evaluators who participated in the survey. I thank them for their participation and for taking the time to send me their best and worst experiences as website evaluators.
Conclusion
I hope this article has shown you that there is more to being an evaluator than just handing out awards. We are human with feelings, but we can't let them cloud our judgement. Our role is to help to improve the quality of the Web. The human eye still plays an important role in something as technologically advanced as the Internet, which makes us impartial partners in Web excellence! If computers can't do it, we can!
www.website-awards.net
www.website-awards.net/articles/article31.htm