A Winning Evaluation Team
by Jef Peace, Webmaster
PeaceWork Certified Sites (site closed)
2 January 2001
Print Version
There's no magical formula which will instantly produce a quality evaluation team, but there are some ideas that can be shared. In this article we'll take a look at motivations for assembling a team, methods of finding evaluators, tips on how to keep the team together and advice on how to make sure your team remains effective.
When I first sat down to write this article, I found it difficult to come up with the proper words. After several false starts, I realized what the problem was. There are actually several beginnings. Rather than bore you with the details, I'll simply start, not at the beginning, but in the middle where it really began.
In the ... Middle
At one point PeaceWork Certified Sites (PWCS) was in danger of closing down because of the loss of one evaluator and the imminent loss of three more. Fortunately, the very day I began to think about closing the program, I received an email from Debbie Frey indicating an interest in joining our PWCS evaluation team. To say I nearly broke the keyboard with my quick reply would be putting it mildly. I "hired" her on the spot!
That was the actual beginning (eight months after I started the program) for it was my first experience of working with an evaluator in cyber-space. My original team consisted of my wife, four of my employees and myself. This allowed me to have meetings and get input on an almost daily basis while looking them in the eyes. Email does not allow for a lot of the nuances we take for granted in our spoken communications, so the first week with Debbie was a comedy of miscommunications.
Once we acquired a "feel" for each other, it smoothed out considerably. Debbie's now proven to be PWCS's most valuable resource. I came away from that initial week with many valuable lessons. Chief among them was the realization that it's a good idea to read your outgoing email at least twice before you click on the "send" button.
Do You Need a Team?
A flippant "yes" would suffice here, but it wouldn't make for a very interesting or informative article. In my opinion an evaluation team is absolutely indispensable to a quality awards program. Let's take a look at the reasons why an evaluation team is so crucial to a program's ability to offer quality evaluations.
| 1. | Objectivity Having a team of people looking at the same site allows less room for subjective skewing of the points awarded for specific criteria. Differing opinions are much less destructive to an evaluation because they can be averaged in some manner. |
| |
| 2. | Thoroughness It goes without saying that each of us has our own idea about what's important. Having several people evaluating the same site will ensure a much closer scrutiny of all pertinent elements. |
| |
| 3. | Consistency Having a team allows individual members to have "days off." An evaluation team means less likelihood of an individual's mood on a given day dictating the outcome of a site's evaluation, for good or bad. |
| |
| 4. | Speed As stated before, each of us has our own ideas of what's important. Individual members of the team tend to scrutinize more closely those elements they deem most important and spend less time on elements which they do not. This should be encouraged for it allows a much quicker turn-around on individual submissions. |
| |
| 5. | Respectability It stands to reason that a team of evaluators demands more respect than an individual. I have a hypothesis concerning this. It will never be relegated to a theory because it will remain untested. My hypothesis is this: If I were to "fire" my evaluators tomorrow and announce that PWCS was now Jef Peace's Awards, I would see a significant drop in the number of submissions received. |
Wanted – Quality Evaluators
Okay, we've determined a panel is important, so how do we acquire one? As Shakespeare wrote: "Ah, there's the rub!" I was extremely fortunate because the five people who applied for the position were all quite competent and needed very little direction to get started. It doesn't take a Math Major to figure out that 5 applications in 6 months is not a sign of success, so I won't tell you how I went about assembling my team. Instead, I will quote some sound advice which came from reputable sources.
| Solitaire, NetMagick.Net (site closed) | |
| |
| "Most of my judges were personally invited after researching their potential value to the program. To assemble a team of reviewers for an award program you try and find a variety of skills that are inherent in each person on the team and then encourage them to develop other skills to become a well rounded reviewer. You need, as a minimum, someone who's strength lies in their artistic ability and someone who's strength is technical." | |
| |
| "What most people don't know is that you need a new person on the staff occasionally to keep your feet firmly planted in reality. That person's initial job is to ask you why you do things the way you do them. Explaining things to that person often leads you to re-examine your objectives and your criteria." | |
| |
| "Last but not least, you need one person you can trust implicitly to take over for you should you ever end up incapacitated for any length of time. That person is your right hand. They listen to you, they encourage you, they criticize you, and they send you a hug when you feel bad. That person is the most important member of your staff, because without them you can lose your perspective." | |
| Bob Taylor, Millennium Design Services (site closed) | |
| |
| "The Internet is a vast faceless forum. Countless thousands of web sites exist merely to promote themselves at the cost of others. When I first set out to develop an awards program I wanted to make sure that recognition was given to those sites which I felt served a true purpose on the Internet." | |
| |
| "The struggle I had was how to go about getting our award recognized enough that it would mean something to the recipient. Award Sites! presented me with the forum I needed to at least get the word out. Although our rating was a 3.0 in the beginning, I knew then that Millennium Design had a voice." | |
| |
| "Being recognized within the awards community comes with a great amount of commitment to those you are awarding. When I realized that Millennium Design had a viable program I searched the awards community for help. Judges were not selected at random and were not given free rein over our program. Millennium Design closely monitored and still monitors all applicant scoring." | |
| |
| "When selecting judges make sure that they have the same values as yourself when it comes to awards. Make sure that any one of your judges is worthy and capable of taking complete charge of your program should the need arise. I experienced a personal crisis over the past several months. During that period of time Rhonda of Otakou Peninsula was given all passwords and pages from Millennium's website. I was prepared to turn over the full reins to her in order that the program continued." | |
| |
| "Things have since been cleared and I am back at the helm ... but this is what I mean when I say make sure you have faith and trust in those you have by your side. With that type of commitment from your judges, your program will flourish. Also, do not be afraid of help. Keep an open ear and mind." | |
| |
| "Each of the judges that Millennium Design has selected have in their own way gone well above the call of duty and far exceeded any of my expectations. So rather than repeating any more, I will let the above advice speak for itself." | |
| Maggi Norris, Nem5 Awards Program | |
| |
| "Most of my judges were found among the applicants for my award. Some of them applied. I invited others to work with me. The people I asked had to impress me not only with their design and/or artistic skills, but with their ability to communicate." | |
| |
| "I needed people who could understand all of my criteria, which has sent some people running in confusion. I needed people to tell me when I was being too strict or too lenient or whether I was just doing it wrong. In order to set up criteria that required multiple scores for certain awards, I had to have judges who could keep up the pace and stay motivated over a period of time." | |
| |
| "I try to give every judge a job they are comfortable doing. If Flash is what they do best, then Flash is what they should judge. This has kept most of them pretty happy working as a group. My staff has a very democratic basis. We vote on all major issues and changes to the program. I believe OUR program gets better every day because we all work together on it. I consider myself very lucky to have found such a wonderful, talented, friendly and opinionated group to work with." | |
Keep 'em Happy!
Once you have a team assembled, the most crucial element becomes keeping it intact and viable. My grandmother often said "The body seldom wanders far from where the mind is happiest." This is true of an evaluation team as well. If your evaluators are not happy with what they're doing for you, they will invariably move on.
It's very easy to lose evaluators. All you have to do is ignore them, criticize them, or in a myriad of small ways offend them. I treat my evaluators as though they are good friends simply because they are. It is crucial to an effective evaluation team that all members are tolerant of each other. They must be able to agree to disagree when an agreement is not possible. A good team is comprised of respectable people who are open and honest with you and each other. But it doesn't end there.
Unless you are actually paying your evaluators in the coin of the realm, it's important to pay them with respect and kindness. You must let them know they are important to you and do so with sincerity. If it becomes necessary to point out errors, do so with empathy and understanding and above all, be gentle and diplomatic. In short, all your dealings with your evaluators should be conducted with uncompromised honesty, respect and empathy. They are performing a valuable service for you. This service should be rewarded with sincere gratitude.
Well-oiled Machinery
A good evaluation panel can be compared to fine machinery. Even if properly assembled and built with superb craftsmanship, a machine requires regular maintenance to assure a high level of productivity. Once your team is assembled and trained, it is a good idea to "oil your machine" at regular intervals. Direct contact, such as email exchange, meetings held in chat rooms and/or forum boards, is crucial to the effectiveness of your program.
It's easy to make the mistake of assuming your evaluators will be offended by specific instructions. A word of advice — don't assume anything. In my experience the more professional and proficient the person, the more that person appreciates specific instructions. I know that I, personally, prefer to have all the information available in as minute detail as possible before beginning a new task.
Keeping your evaluators informed and up-to-date is the best way to assure they will adapt quickly to any changes you may implement. The best way to accomplish this is to include them in the decision making. Ask them what they think of a proposed change rather than inform them of a change that is already made. Even if you decide to go against their advice, they will know you value them enough to ask.
It's In The Bag
Assembling an evaluation team is probably one of the most difficult tasks you will face. It's also one of the most rewarding experiences you can have. Keeping the team working as a unit will guarantee the success of your awards program. It will bring you credibility more effectively than any other single element.
I strongly urge you to assemble a team and to do it in a patient manner. It won't happen overnight. Once your team is assembled, however, you will find yourself wondering how you ever managed without them.
www.website-awards.net
www.website-awards.net/articles/article33.htm