Ask the Experts, Question 8:

How long does an evaluation take and do you look at every page?

by Debra Sharp, Webmistress, Mesweet's Haven & Awards
15 April 2002

Print Version

Bitten Jønsson, Webmaster
DWD Award Program (site closed)

This question actually contains two parts, and I feel it necessary to treat them separately. My answers are based on the experience I have gained from working as an evaluator for various award programs.

How long does an evaluation take?

Do I look at every page?

To make a proper evaluation, you have to look at every page in the site. If we only look at a few pages, what have we seen? What are we going to base the evaluation on? Remember, it is not just the technical and visual part of a web site that is being evaluated. It is also the content. Think of all the knowledge we gain by evaluating web sites.

It's great to be entertained — to have a good laugh — but to gain new knowledge makes a web site even more interesting. If the content is presented in a visually pleasing manner and the site is technical sound, then I think it's safe to say, "We have a winner!"

Geetha Thirunakkarasu, Webmistress
Geetha's Web Design Awards (program closed)

The evaluation period varies for each award program, and it takes about a maximum of 6 weeks, depending on the number of evaluators involved and the number of visits made by each evaluator.

It also depends on the quality of the evaluations. New or inexperienced evaluators might base their evaluations strictly on the look of a website or on their likes and dislikes. If they can't make their own decision, sometimes they'll look at the awards previously won by the website and make a decision based on that.

Good quality award programs are judged by highly experienced evaluators. The evaluations are done in a professional way, and the evaluators can identify even small problems. They use a score sheet to make sure all websites are evaluated using the stated procedure.

The evaluation starts with the list of disqualifiers. This is the fastest way to determine if the site has all the required elements. The evaluator then continues to apply standardized testing procedures, and sometimes multiple visits are required.

Professional evaluators will do a fair and THOROUGH evaluation to get accurate and consistent results. Evaluators help to improve the quality of the web by dedicating time and effort to reviewing websites and encouraging webmasters to strive for excellence.

Kim Cole, Webmistress
World of Knowledge Awards (program closed)

The length of time an evaluation takes and whether every page is examined is really dependent on the site being evaluated. I will cease an evaluation if I come to a disqualification point, and so will many award givers I know. If I see that all of the text on the first few pages of a site has been centered for no apparent reason, I do not proceed any further. This review is very short.

Sites that pass my disqualification points take longer to evaluate. I click through them to visit most of the pages, but I don't spend a great deal of time on any one page or section. At this point, I am just getting a feel for the site. Perhaps ten minutes for a large site.

The second time I visit the site I go through it much more slowly. I try to visit every page, although it is possible that a linked page gets missed here and there. This second review can take an hour, an hour and a half, or it may be spread out over two days. It all depends on the size and complexity of the site in question.

Although I use very detailed criteria, each evaluation is really a brand new start. The structure of the site, in large part, will determine how I view it. So will the design, as far as ease of navigation and reading are concerned. Even the topic may lead me to view the site in a particular order.

Rest assured, however, that look at the site I do. All of it. After all, I cannot put my seal of approval on a site unless I am sure the entire site has earned it.

Debra Sharp, Webmistress
Mesweet's Awards

The time it takes to evaluate a site depends on the size of the site and how many times you visit it. My first visit usually takes between 2 and 15 minutes. Just long enough to get a first impression and make sure the site passes all my disqualification points.

Sites that pass the first visit are put in my "yes" folder, and they get a second visit, which takes between 15 minutes and an hour. Every site left in my "yes" folder after the second visit gets a third one, which is probably the final visit. The third visit also ranges from 15 minutes to an hour.

I look at most of the pages in a site, but I can't say that I look at every one. I don't always look at the "awards won" pages.

Will Harbeson, Webmaster
The Medals Of Excellence

How long processing an application takes is a direct reflection of the criteria on the award site you applied to.

I take nearly 30 days because I require sites to show that they have updated their content during the review period. I look at each site right away to make sure there is nothing that will keep them from winning my award. If the site passes that initial screening, I set it aside for 2 weeks and then check every page again for updates and continued compliance.

If I find no changes to content on the second review, I place the site aside for another week and repeat the process until I can assure myself that I have given the applicant every possible chance to win.

This does not mean that awards are not valuable if the process is quicker than mine. Most experienced reviewers can tell in a very short visit if the site will win their award. The award master is completely familiar with his own criteria, and applying it to applicants' sites is not a lengthy process. Experienced evaluators can conduct a thorough review in a very short period of time.

I would be wary of awards that come immediately after you apply. In my opinion, they are nothing but clutter on your page and have little or no value. I think it takes at least a visit to the applicants site to see each and every page. The award master should click on all the links, look at the navigation for ease of use, and apply all the criteria points carefully.

Ulrich Pokorra, Webmaster
Xmas Stars Awards

The Xmas Stars Award program is owned, run and judged by a single person (that's me) who knows all the criteria and requirements by heart. As a result, my evaluations and scoring are relatively consistent.

I'm quite aware that my standards are high and only a few sites are nominated for a more intensive evaluation. However, I do not hurry through a site once it is nominated. I take the time to enjoy each and every page. I enjoy the graphics, the words, the features, and some of them even inspire me to improve my own websites.

It's probably more than an hour, but I don't keep track of the time it takes to evaluate a site. I make notes about my impressions and what I like and don't like, if there are any dislikes. Most outstanding websites don't have any weak points, do they?

When cruising around a site, I often know right from the beginning if it's worth my top award. If I think it is, I tend to forget about the time factor and simply enjoy the experience. Hopefully there will be many more of these really great experiences in the future.

Sometimes I get a content rich site that is difficult to read and evaluate because of the topic. I'm not an expert in every topic, and it can be hard to understand what the author is saying. I have to read some pages two or three times. I enjoy them, though, for I get valuable information in return for my efforts. It's the principle of giving and taking.

All my applicants deserve a thorough and detailed review, regardless of how much time it takes. They have given their best in the creation of their websites, and I have a responsibility to give my best in the evaluation.

I personally emphasize the quality of judging more than the quantity. I am very proud of my winners' list. I spend a lot of time reviewing sites to help webmasters improve the quality of their work. If they're happy with my efforts, then I am happy, too, and I will continue to disregard the time factor. If you enjoy what you're doing, time really doesn't matter.

Fabian Lorenzen, Webmaster
El Fabi Awards

In my opinion, there's no specific time frame in which an evaluation of a website can be completed. Since each website is different, the amount of time required to evaluate them will differ. Regardless of the differences, however, they all deserve a conscientious and fair evaluation.

Almost every award program puts their applicants through a preselection process. If it meets the basic criteria, the award-master will invest further time to develop his own personal impression and opinion of the applicant's site.

There is also the question of familiarity. An experienced award-master has visited a large number of websites, and he may remember an applicant's site from a previous visit. This will reduce the time it takes to evaluate the site.

Most evaluation processes have two or three stages. The first stage is the initial impression the award-master gets during the first visit, which usually takes less than five minutes. This includes a quick look around the site to make sure it meets all the basic criteria for the award.

Sites which pass the first stage are visited a second, third or even fourth time to complete the evaluation process. During these visits the award-master forms an impression and opinion of the visual and content quality of the site and decides if it is worthy of an award. The evaluation time will vary from site to site, depending on the number of visits and the complexity and size of the site.

Dave Selig, Webmaster
Ultraweb Awards

My evaluations can take anywhere from a few seconds to two weeks, depending on a number of factors. Often, when I visit an applicant's site, the homepage contains an obvious criteria violation which results in a disqualification. In those cases, the evaluation ends within seconds or minutes. In other cases, I'll review the introductory page of each major section of a site, find no violations, then begin an in-depth review.

If the size of the site is manageable (7 to 200 or 300 pages), I review each page. If I discover a disqualifying element along the way, the evaluation ends there. Otherwise, I review the entire site. The review time is different for each individual site that contains no disqualifying elements.

If the site contains hundreds or thousands of pages, I usually limit my review to 300 or 400 pages. If a webmaster creates 300 or 400 superb pages that would qualify for any level of the Ultraweb Award, I am satisfied the webmaster has created a high quality site.

For example, if a site has 80 pages of true content and 900 pages of links to other sites, I will review each of the 80 pages in depth and examine over 220 of the link pages. I find it becomes counter productive to review more of the link pages. My time can be better spent reviewing a different site.

Of course, if I later learn that a site which earned the Ultraweb Award contains a major violation of my disqualification criteria (for example, a page contains pornographic or racist material), I will remove the site from the winner's list.