Evaluations Guide for Assigners

Sections:
Best Practices and Key Terms
Getting Started
Sections and Questions
Completing Reports
Monitor Evaluations





What is an Evaluation? 
An Evaluation is a survey that can be created by an Assigner to better understand key factors taking place at games. Evaluations equip Assigners with information to better lead their association.


How does it work? 
Create an Evaluation that spans a time range, then configure it to be answered by the roles of your choosing by asking the respondents to score the Subject on the Criteria that you define.
1. First, set up an Evaluation by following the instructions below.
2. Next, train and set expectations or policies within your organization, or with the roles tasked with completing the Evaluation. 
3. Then, as time passes, games are played, and Evaluations are completed, the data will be available to you for analysis.  


Who are the Respondents? 
Once an Assigner creates an Evaluation and sets the parameters, any mix of Assigners, Officials, and Contacts can respond to the survey with crucial information that will help you better understand what is taking place and act where needed.


What if I have specific needs for my conference? 
Evaluations can be tailored for broad groups for comprehensive data collection, or to target a small group for specialized feedback or performance review, Evaluations can scale to the data collection needs of your organization.






Best Practices

Here are some tips before you get started.
*Note: Groups must be licensed for Evaluations to access the Evaluations feature.

1. Start simple! Evaluations can be fine-tuned to match many needs. By starting simple, you can see what works and doesn't work and adjust it the following season. This ensures you get value out of evaluations from the start.
2. Create separate evaluations for contacts. It takes more work to create separate evaluations, but creating a separate evaluation for contacts allows you to compare an official's performance as viewed by the schools and from their peers.
3. Copy evaluations for following seasons. Instead of having an evaluation run forever, put a start and end date on an evaluation.  When the next season rolls around, copy that evaluation and adjust the dates. This allows you to ask the same questions but gather data from different time periods.  After a few seasons, you will have a great perspective on how the group and individuals are progressing.
4. Use sections (categories) - Sections allow you to group criteria and define the subject of the Evaluation. If you have related items, you can see how the group and individuals are doing in a general category. Are they amazing on the field but don't handle the coach, players, and crowd well? Do they know the rules like the back of their hand but aren't always positioned to make the right calls?

 

Key Terms

Evaluation - A survey or questionnaire that can be created by an Assigner to better understand key factors taking place at games.  
Subject (previously known as categories) - A target that is being evaluated. The standard subjects are Game, Site, Official, & Teams (Home & Away).
Criteria - A concise term, concept, question, or metric that is being evaluated on a range defined by the Assigner.  
Standard - An explanation, definition, or example of the Criteria. It can be viewed by respondents by clicking on the Criteria term. 
Stars - An Assigner-defined upper threshold that requires a comment from respondents. It has extra reporting visibility and can be configured to send email notifications.  It is usually indicative of an exceptionally good evaluation score. 
Red Flag - An Assigner-defined lower threshold that requires a comment from respondents. It has extra reporting visibility and can be configured to send email notifications.  It is usually indicative of an exceptionally poor evaluation score. 



 

Getting Started

To set up the evaluations:

1. Click on the Reports tab.
2. Click the "Evaluations" sub-tab.
3. You will be directed to the Evaluations dashboard. To setup a new evaluation report, click on "New Evaluation."  This will bring up the screen with the evaluation report settings, reporting permissions, and criteria to build a custom report.

4. Enter a name for the evaluation (example: Peer Review, Coaches' Evaluation).
5. Select if you want to use an ArbiterSports template, copy one of your existing evaluations, or start from scratch.
   1. For first-time Evaluations users, it's recommended that you utilize an ArbiterSports standard template. You can skip a few steps and always modify the Evaluation to your needs.  
6. Click Next.

 

7. Fill out the Basic Settings: Score Range, Require Comment If Below, Reminder to Complete, Publish Responses Min, and Require Comment If Above.

a. Score Range: The score range you want your report to use. Use the format 1-5 with any number scale you want (As an example, 1-10. Score from 1 to 10, 1 being the lowest, 10 being the highest).

b. Require Comment If Below: If the evaluator gives a score lower than the number you define here (example: 2), they will be required to comment and give a reason for that score. If you want to require comments all the time, the number should be whatever the lowest number in your score range is (i.e., 1-10, 1 would be the lowest).
c. Reminder To Complete: Sends an email reminder after however many days you select. For example, select 3 and a reminder will be sent 3 days after the game.

d. Publish Responses Min: Once this number of responses has been met, the evaluation report will be shown to those who are allowed to view it. At this point, a “responses” link will appear to the officials and sub-administrators that have been given permission to view them. All responses will be shown to viewers anonymously.

e. Require Comment If Above: If the evaluator gives a high score (example: 5), they will be required to comment and give a reason for that score. If you want to require comments all the time, the number should be whatever the highest number in your score range is (i.e., 1-10, 10 would be the highest).

8. Select the Sports & Levels for the evaluation to be completed. Next, select the Positions for the sports and levels.

9. Add email recipients to receive a notification for all completed evaluations, only completed evaluations with Red Flags, only completed evaluations with Stars, or only completed evaluations with Stars and Red Flags.

10. Enter the Start Date and End Date for the evaluation to be active.
11. Select who the evaluation report can be completed by:  Admins, Evaluators, Lead Officials, All Officials, Team Contacts.

1. Admins - Allows admins in the group to complete evaluations. Because admins complete the evaluations from the games and assignments screen, they can only complete evaluations on games they can see (scope, global filter, etc. will limit those games).  Admins can evaluate multiple games that happen at the same time, as well.
2. Evaluators - An evaluator can be anyone that is put on the game for the purpose of evaluating only. Evaluators are created from the sports/levels screen. Edit the sport that you want an evaluator for. Create a new position or edit one of the existing ones and mark it “observer.” This allows you to rank certain officials to be evaluators. 
3. Lead Officials - A lead official is defined by the preference you have set under Settings.  It can be one of two defaults:  The official in the first listed position or the official who is highest ranked in the first listed position.  Admins can choose the lead official on a game by game basis by editing the slot and marking it as the lead official.  Each game is limited to a single lead official.
4. All Officials - This allows officials to complete evaluations on each other.  Since officials can't evaluate themselves if an evaluation only has the plate umpire selected to evaluate the plate umpire won't see an evaluation for that game (unless there are multiple evaluations). There is no permission or preference for which officials can complete evaluations.  The permission to 'Select games to evaluate' is to allow evaluators to find officials to evaluate.
5. Team Contacts - You can allow your team contacts to complete evaluations on officials.  If you don't want all team contacts to evaluate officials, go to Resources and then the Teams sub-tab.  Open the contacts for a team and only check the Eval checkbox for contacts that should complete evaluations.  By default, contacts can complete evaluations.
6. Note:  No one can evaluate an evaluator.  If you have an evaluator listed on a game, they will not appear when completing evaluations.  If you receive a notice that an official can't be evaluated, check to make sure that the position didn't get marked as an evaluator (observer) by accident.  For example, occasionally we'll see a referee marked as an evaluator. As a result, they don't appear.

12. Select who the responses can be viewed by: Officials, SubAdmins.
    1. Officials can be given permission to view responses that were made regarding them. You can also allow SubAdmins to view the responses.
13. Click Save. If you are creating a new evaluation report from scratch, click Build Criteria.

14. To set up the Criteria, click on the Questions tab towards the top and refer to the instructions below.

 

Sections and Questions

1. Enter a Scoring Description. Describe what you want each rating to count as. This will explain to the evaluator what would be expected of the official to receive a five, four, etc. (For example, 5=Excellent, 4=Above Average, 3=Average, 2=Poor, 1=Unacceptable). Below is a screenshot as an example:

2. Click on the green plus sign under the Sections area to set up the subjects.

a. Sections- When completing an evaluation report, the criteria will appear grouped in the categories that they are attached to.

b. Subject (previously known as categories) - A target that is being evaluated. The standard subjects are: Game, Site, Official, & Teams (Home & Away).



c. Any criteria that you tie to the Officials section will fall under the positions on the game (such as in the image below, Line Ref 1, Line Ref 2, etc..).
d. Game, Home Team, Away Team, and Site will have their own sections that follow the positions/officials to evaluate.

Below is an example of what the evaluation report will look like to the evaluator. The sections, based on what you set up for the subjects (mentioned above), will show in list formatting on the left side for the evaluator to complete.

3. Click on the green plus sign under the Questions area to build the criteria.

a. Criteria - A concise term, concept, question, or metric that is being evaluated on a range defined by the Assigner.
b. Questions - The questions are the points of officiating that you want the officials to be evaluated on (for example: punctuality, professionalism, knowledge, communication with coaches and partners, etc.).

4. First enter the criteria name (for example, the topic of what will be evaluated, which could be Professionalism, Judgement, Player Management, etc.).
5. Enter the standard. For example, the name of the criteria may be “Appearance,” and the standard would be a description, such as: “Has proper, clean, and well-fitted uniform, black shoes, etc.”

a. Standard - An explanation, definition, or example of the Criteria. It can be viewed by respondents by clicking on the Criteria term.

6. Enter the weight for that criteria. Provide a percentage of how much that criteria affect the score of the complete evaluation. (For example 10%)

a. The weight determines how much each question will affect the overall score. Make sure that the weights of all of your criteria add up to 100%. The total of all weights entered cannot exceed 100% but may fall below 100% for the purpose of finalizing and drafting your criteria. This allows assigners to add all questions, then determine the weight of each criterion.

7. Check the "Required" box if the evaluator is required to score that criterion.
8. Click the blue floppy disk icon to the left to save.
9. You can use the Preview button at the top to see what the evaluation will look like to the evaluator.
10. When you are done building the questions, click Save at the bottom.

 Completing Reports

Where to find Evaluation Reports:

1. Go to the Schedule tab.
2. Click on the slots of the game that needs to be evaluated.
3. Click on the red A+ icon located on the right side of the slot information. Note that the evaluation report (A+) will not show up on future games.

4. Select the evaluation report you need to fill out from the dop-down menu and click go.

How To Complete Evaluation Reports:

1. Verify that all of the game information is correct.
2. Carefully read the Scoring Description before continuing.
3. Using the drop-down, select a score next to each criterion. To view the criteria’s standard, click on the criteria name.
4. Enter comments as needed in the indicated text box on the right side. When each of the score areas is completed, write any additional comments using the Summary Comment text box at the bottom.

5. To send the report, click the Submit button at the bottom. If you would like to return later to finish the report, click Save.

 


 

Monitor Evaluations

1. Go to the Reports tab.
2. Click on the Evaluations sub-tab.
3. Click on the name of the evaluation you wish to monitor.

 

4. Click on the Monitor button towards the top.

 

5. There are six tabs that can be used to monitor evaluations: All Evaluations, Game, Site, Official, Team, and Evaluator.
6. On the left side under Utilities, you can use the Mass Delete option to clear out all Evaluation responses.
7. If at any point you need to return to the Evaluation or Evaluation dashboard, click Exit in the bottom right corner.

All Evaluations
Under All Evaluations, you can read the individual evaluations that have been submitted for each official.

1. Click on the All Evaluations tab.
2. To view the average scores and comments for each official, click on the number in the Average column next to the official.
   1. On this screen, you will see the average score given to the official for each criterion.
   2. Click on the average score to view an accumulation of comments related to that criteria.
3. To view the report for each official, click on the number in the Completed column.
   1. On the left side, click on the book icon to access the evaluation. The book will be 'open' to  signify you have already viewed the evaluation report. The book will be 'closed'  to signify you have not viewed the evaluation report.
   2. You can also access the evaluation by clicking on the Score on the far right side.

If there is an issue, the assigner may unsubmit the report, which sends it back to the person who submitted the report for revision and resubmission. The official then will have the opportunity to either completely redo the report or just modify it. The assigner can change the scores or comments from the report view page as well.

Monitor by Game
Allows you to view scores and comments related to games and the criteria grouped under that section.

1. Click on the Game tab.
2. In the first column, you will find the game number if you want to view the responses for a specific game.
3. To view the average scores and comments, click on the number in the Completed column.


Monitor by Site
Allows you to view scores and comments related to the criteria you grouped under the Site section.

1. Click on the Site tab.
2. In the first column, you will find the Site for the games that have been evaluated.
3. Click on the number in the Completed column to view scores and comments.


Monitor by Official
Allows you to view scores and comments related to the criteria you grouped under the Officials section.

1. Click on the Official tab.
2. In the first column, you will find the name of the officials that have been evaluated.
3. Click on the number in the Completed column to view scores and comments.


Monitor by Team (Home and Away)
Allows you to view scores and comments related to the criteria you grouped under the Home Team and Away Team sections.

1. Click on the Team tab.
2. In the first column, you will find the Team Name with the sport and level for the games that have been evaluated.
3. Click on the number in the Completed column to view scores and comments.


Monitor by Evaluator
Under Monitor by Evaluator, you can read the individual evaluations that have been submitted for each official.

1. Click on the Monitor by Evaluator tab.
2. To view the average scores and comments from the evaluator, click on the number in the Average column.
   1. On this screen, you will see the average score that the evaluator has given for each criterion.
   2. Click on the average score to view an accumulation of comments related to that criterion.
3. Click on the number in the Completed column to view each report from the evaluator. You can also access the evaluation by clicking on the Score on the far right side.
   1. On the left side, click on the book icon to access the evaluation. The book will be 'open' to signify you have already viewed the evaluation report. The book will be 'closed'  to signify you have not viewed the evaluation report

 

 

Also, see our Officials Evaluations Guide and Contacts Evaluations Guide.




 

End of article content.