Views:

 

 

 

GOAPPLY REVIEWER ADMIN GUIDE

 

INTRODUCTION 

 

GOapply applications can be reviewed by both internal and external reviewers using the GOapply review features. These features allow the user to create and manage specific review groups for particular phases of their submitted applications. In addition, GOapply Users can be made reviewers and can be added to as many review groups, as necessary.  

 

The GOapply review features are divided into several components, which can be found on the left-hand navigation pane in the GOapply workspace.

 

 

 

Tip: We offer a Suggested Learning Course for GOapply Reviewer, which includes several instructional videos. GOapply Reviewer Course

OVERVIEW OF GOAPPLY REVIEWER COMPONENTS 

 

GOapply Reviewer is made up of many of the same entities as GOapply. Users, Applications, and Status Tracking are all employed in Reviewer in addition to several Reviewer-specific entities. These include Reviewers, Review Groups, Review Group Applications, and Reviewer Request Score. Below is a high-level overview of Reviewer entities; we will dive deeper into these topics later in this document. 

 

USERS 

Your reviewers will register for a GOapply account independently. This process is the same as applicant registration, which we will walk through step-by-step later in this document.  

 

APPLICATIONS 

From the Application, you can see which Review Groups it is attached to.  

 

STATUS TRACKING IN REVIEWER 

Status Tracking records are what informs Reviewer which phases/applications are ready for review.  

 

REVIEWERS 

Reviewers are all GOapply Users that have the Reviewer or Applicant and Reviewer role assigned. 

 

REVIEW GROUPS 

Reviewers will not have access to any applications to review until they have been added to a review group.  

 

REVIEW GROUP APPLICATIONS 

Review Group Applications use the GOapply Status Tracking record to display the application for the reviewer. If the GOapply Status Tracking record does not exist or does not contain a PDF application, the application will not display correctly in the reviewer dashboard.  

 

GOapply Admins can upload an alternative file to be displayed for the reviewer. In the Status Tracking Record, there is an upload field called Reviewer File. Here is where you would upload what you want displayed to the reviewer if it is different from the submitted PDF. This file will override the submitted PDF.

 

image

 

REVIEWER REQUEST SCORES 

Reviewer Request Score records allow you to see the score that a reviewer gave. These records also contain the Reviewer Form Response json file. This json file is a string of code; it is similar to the Form Response in GOapply Status Tracking records.  

 

REVIEWER DASHBOARD 

In all areas of akoyaGO CRM, you have a system dashboard called GOapply Application Review that tracks applications that need to be assigned to a review group, applications with a conflict of interest, applications with scores from review groups, and active reviewer request scores. 

 

GETTING STARTED 

 

Now that you know how the GOapply Reviewer entities work together, let’s get started creating Review Groups and adding Reviewers. 

 

CREATE A REVIEW GROUP 

Each review group consists of general info (including the application and phase), settings to determine what reviewers can see, the reviewers themselves, and review group applications. All of these components are necessary for reviewers to have everything they need.  

 

The following steps detail the process to create a completed review group.  

 

1. From the Review Group entity, click “New Review Group.”  

 

The New Review Group form will load.  

 

2. Name your review group.  

 

3. Select the application and phase that the review group will be assigned to.  

 

4. Set a review group deadline in the Review By field. Once the Review By date has passed, the review group will no longer appear in the reviewer’s list in GOapply. 

 

5. Lead Reviewer Recommends: This settings allows your lead reviewer to recommend which application gets awarded and if awarded, the amount awarded. This feature will ONLY apply to the reviewer where Lead Reviewer = Yes. Choose from the drop down

  • No (or null): This feature is turned off 

  • Recommend to Request: If chosen, this setting will make it so the lead reviewer's choices will map to the associated Request record. When this option is chosen, an additional question will be added to your reviewer form called Recommended Award Status.

The lead reviewer can choose Award or Deny. If Award, an additional Recommended Amount field will be added where they will input the amount they recommend to award. The amount field will be required.

 

Once submitted, their choices will be mapped to the Request. A field called Review Recommender will populate with the GOapply User (Process tab > Decision section).

 

 

The Recommended Grant amount will also fill in with the amount they recommended. Their choices will also map to the Review Group Application. 

 

  • Recommend to Review Group: If selected, this option will also add the Recommended Award Status (and Recommended Amount field if Award) questions to the review form. This option will also map the lead reviewers choices to the Review Group Application. This option will NOT fill in any field on the request. 

 

6. Click Save. Upon save, the Review Form Definition field will populate with its unique string of code and lock to prevent accidental deletion. To download the Review Form Definition, click the link in the field. 

 

Click the Settings tab of the newly created Review Group. 

 

REVIEW GROUP SETTINGS 

 

The Settings tab contains various customization options for each review group. These settings involve the information you wish to share with the reviewers. All but Anonymous Review will default to Yes. Anonymous Review will default to No. Scoring Type will default to Sum (total score).

 

A screenshot of a computerDescription automatically generated

 

Show Other Reviewers Comments: If Yes, the Reviewer Comments button will be visible on the application review form. If set to No, the Reviewer Comments button will not be visible on the application review form. Please note that if Show Other Reviewers Comments is set to No, the Show Reviewer Weighted Score setting is rendered null, and the Public setting in Form Builder (page 10 of this guide) will also be null.  

 

If Show Other Reviewers Comments is Yes:  

 

image

 

If No:  

 

image

 

Show Mark for Conflict of Interest: If set to Yes, the Conflict of Interest checkbox will appear on the Reviewer’s review form. If set to No, the Conflict of Interest checkbox will not be visible on that form. 

 

image 

 

Anonymous Review: Select Yes if you want applicants to appear anonymous. If Yes is selected, the applicant’s name will be removed from the PDF that the reviewer sees. The reviewer PDF will be stored in a field called Reviewer File in the GOapply Status Tracking record. 

 

Show Reviewer Weighted Score: If Yes, reviewers will be able to see how other reviewers scored the application. It renders as the far right column in the Reviewer Comments. If No, the weighted score column is not visible.  

 

image

 

Scoring Type: Choose Sum, Average or Sum by Reviewer then Average by Application.  

 

  • If Sum (total score), each question will be equally weighted, and the reviewer score will be the total of the reviewer’s answers. With sum selected, you can still set the question’s value in the Choices section of properties in Form Editor. See Set Scoring section of this document for more information. 
  • If Average, you will be able to set weight of each question in that question’s properties, akoyaGO section. More information can be found in the Question Properties section of this document. 
  • If Sum by Reviewer then Average by Application, the scores will be averaged by adding all the scores from each reviewer, then dividing by the number of reviewers. The average score will be shown in the Review Group Application record, in the Reviewer Scores tab. 

 

 

REVIEW GROUP FLOWS 

 

Two flows exist to duplicate a review group. To run the flow, select the review group, click Flows, and choose either Duplicate Review Group or Duplicate Review Group and Reviewers.

 

image

 

Duplicate Review Group: Duplicates application, phase, review by date and time, review form definition, review form and all properties, all settings, show staff comments and attachments setting, and staff notes 

 

Duplicate Review Group and Reviewers: Duplicates application, phase, review by date and time, settings, show staff comments and attachments setting, staff notes, reviewers 

 

Please note the staff notes and staff attachments are NOT duplicated with these flows. 

 

ADD REVIEWERS 

 

Reviewers are the GOapply users that will be evaluating the applications in their review groups. Reviewers must be added to each review group they are reviewing applications for. In order to be added to a Review Group, the user(s) must have the Reviewer or Applicant and Reviewer role.

 

From the Reviewers tab of the review group, click New Reviewer in the All Reviewers subgrid. 

image

 

The New Reviewer form will populate with the review group information. Review Complete and Lead Reviewer will default to No. 

 

image

 

Select a GOapply User in the Reviewer field. If the user is the lead reviewer, make that designation in their reviewer record. Lead Reviewers will be able to see all reviewers' comments regardless of the Show Reviewers Comments setting in the review group settings. Please note that the intended functionality is to have one lead reviewer per review group. If you try to set another reviewer as lead reviewer, it will override the previous lead reviewer. 

 

Review Complete: this field will automatically flip to Yes when the Reviewer has completed reviewing all the applications in all the review groups that they are assigned to. It will default to No upon implementation, so if you are in the middle of a review cycle on 8/4/2022, you will need to manually flip Review Complete to Yes for that cycle.  

 

The Reviewers tab of the Review Group record are three subgrids: All Reviewers, In Progress, and Complete.

 

  1. All Reviewers shows the Reviewers assigned to the group and includes a Review Complete column that will read Yes or No. 

 

  1. In Progress shows the Reviewers who have started the review process but have not submitted all their assigned reviews. 

 

  1. Complete shows the Reviewers who have completed all their assigned reviews.  

 

image

 

 

BULK IMPORT REVIEWERS

 

Rather than adding reviewers one by one, GOapply Administrators also have the option to bulk import reviewers.

 

Step 1: Create an Excel document and save it as a .csv. 

 

CRM import relies on an Excel spreadsheet where columns = fields and rows = individual records. For bulk importing reviewers, your spreadsheet should contain, at minimum, columns for the First Name, Last Name, and Email. Best practice is to include as many columns as you can define. Potentially, each field on the GOapply User form can be a column. Below is an example of a reviewer import file.

 

image

 

Step 2: In the GOapply workspace, navigate to the Reviewer Bulk Import entity.

 

 

 

Step 3: Select Import from Excel > Import CSV

 

From the toolbar at the top of the screen, select Import from Excel (you may need to click the elipses depending on your screen width), then click the carrot to select Import from CSV.

 

 

Step 4: Choose File and follow the wizard. 

 

An Import wizard will pop out from the right side of the screen. Choose your .csv file and click Next.

 

 

Make sure ‘First row contains column headings’ is checked and click Review Mapping. Allow Duplicates should be set to No.

 

image

 

If your mapping looks correct, select Finish Import. You will see a message that says, “Your data has been submitted for import.” You can close the wizard.

 

Step 5: Run the GOapply – bulk import users workflow

 

Depending on the size of your import, it may take a few minutes, but after you refresh the page, you should see all the records you imported in the Bulk Reviewer Import view. Once you see all your records, select all using the check mark at the top of the screen and click Flow in the toolbar at the top of the screen and choose GOapply – bulk import users. This workflow will create the records as GOapply User records.

 

 

image

 

When the workflow is finished running, all records should disappear from the Reviewer Bulk Import table and move into the GOapply Users table.

 

Step 6: Reviewers get an email and set up their account.

 

When the workflow runs, it includes a step to send an email to each reviewer created. Depending on whether the user is brand new or already had a GOapply account, they will see an email alerting them to the fact that they have been added as a reviewer. If they are brand new to GOapply, they will also be asked to create a password.

 

The emails are using the email templates GOapply – Invite New User or GOapply – User Already Exists. The workflow will check and apply the proper template. If you wish to edit the templates, you can do so from the Tools workspace.  

 

image

 

Step 7: Add reviewers to review groups.

 

Use the steps described above to add your new reviewers to their review group(s).

 

 

STAFF ATTACHMENTS AND COMMENTS 

 

The Staff Attachments and Comments tab in the Review Group record allows for staff notes that apply to ALL applications for that review group. Attachments can also be added here as public documents for the review group.  

 

These notes and attachments are displayed above the review group applications. 

 

image

 

REVIEW FORMS 

 

Review forms are displayed for the reviewer side by side with the application. Question types include a text field, a radio button, a drop down, a long text comment, and a rich text editor. For example, below you’ll see the submitted application (or whatever is in the Reviewer file tab of the Status Tracking record) on the left, and the reviewer’s questionnaire on the right. At the bottom of the page they can save a draft or submit.

 

image

 

CREATING A NEW REVIEW FORM 

 

From the review group, navigate to the Form Builder tab. Select Launch Form Editor.

 image

 

The Form Editor will load in a separate tab.  

 

 

Unlike the form editor in GOapply, the form editor in Reviewer is only able to support one page.

 

Please ensure you design your form to account for this. You will be familiar with this Form Editor; it is the same one you use to create your GOapply Applications. On the right-hand side, you will see the 6 question types. Drag and drop the question types from the toolbox into the form to add.  

 

Keep in mind that if the Lead Reviewer Recommends setting in the Review Group settings = Recommend to Request or Recommend to Review Group, potentially two additional questions will appear on your form.

 

REVIEW QUESTION TYPES 

 

There are five types of question: a text field, a radio button, a drop down, a long text comment, and a rich text editor. Please note that only Radiogroup and Drop-Down field types can have score values assigned. 

 

Single Input: a text or number response field. These fields cannot be used to calculate scores.  

 

Radiogroup: a single selection radio button style multiple choice. These responses can have score values assigned.  

 

Drop Down: a drop-down list of responses. These responses can have score values assigned.  

 

Comment: a long text or number response field. These fields cannot be used to calculate scores.  

 

Editor: a long text or number response field, with rich text formatting. These fields cannot be used to calculate scores. 

 

File: This is a file upload field. Add this field if you want reviewers to be able to upload files. Supported file types include 'txt', 'rtf', 'doc', 'docx', 'xls', 'xlsx', 'png', 'jpg', 'jpeg', 'gif', 'csv', 'ppt', 'pptx', 'pdf', 'mp4', and 'heic'. Max size for all uploads total is 20MB. When a reviewer uploads a file with their review form, it will be stored in the Review Response record for that question. For example:

 

 

If the question type is a type used to calculate reviewer scores, the scoring must be included in the choices for that question.  

 

QUESTION PROPERTIES: 

 

Each question field has a properties menu. Open properties by clicking on the question and selecting Properties from the far right.

 

image

 

Common settings for each criteria include:  

 

Required: Required setting can be found in the General dropdown in the form of a toggle called “Is Required?” If on, these criteria must be completed for the reviewer to mark the review as “complete.” Required criteria will be denoted with an asterisk in the review form. Required will default to no.  

 

Weight: Weight can be found under akoyaGO. This functionality is only available if the Scoring Type in the Review Group’s setting is set to Average. If the criteria are averaged, enter the weight of the score. Note: if averaging is used, the weights of all criteria must total 100.00.  

 

Public: This setting can be found in the akoyaGO dropdown. If checked, the criteria responses will be shared with the review group in the Reviewer Comments tab. Note: the review group setting Show Other Reviewers Comments must also be turned on for this to appear. Lead Reviewer will be able to see all reviewer responses. 

 

SET SCORING  

 

Scoring can be set on the dropdown or radiogroup question types only.

 

1. In the Properties for the scoring field, drop down Choices 

 

image

 

2. For Value, enter the option, then two colons, then the numerical value of that choice. The proper format for this formula is no spaces. For example, Agree::80. In the Text field, simply enter the option that you want the reviewer to see. In this example, you’d just put Agree. Repeat this process for each option in the drop down.

 

If your Review Group settings designate the scoring type as Average, you will see a property called Weight under the akoyaGO section. Here is where you will set the weight of each scored question. Weight is only available for drop down and radiogroup question types.

 

image

 

REVIEW GROUP APPLICATIONS 

 

Now that you have created your review group, designated reviewers, and created the review form, it is time to add the submissions for the reviewers to review.  

 

This is done through the entity called Review Group Applications. In that entity is a view called “GOapply Reviewer”. Through this view, you can manipulate the applications that a reviewer sees. For example, deactivating a record in that view would remove it from the list of applications that the reviewer sees. 

 

ADD REVIEW GROUP APPLICATIONS 

 

Next, you’ll assign the submitted applications to the review group.

 

Review Group Applications is an entity in akoyaGO CRM and uses Status Tracking records to determine which submissions get shown to the reviewers.

 

Bulk add submissions from Requests entity:

 

You will only want to add applications this way if you have more than 1 review group reviewing the set of applications. This will allow you to select a set of Request from a view and add them to a review group.

 

Step 1: Copy the name of the Review Group because you will need to enter it correctly in a workflow dialog.

 

Step 2: You can go to an individual Request record or a view to choose multiple to add

 

Step 3: Go to Flow > GOapply Add Request to Review Group

 

A screenshot of a computerDescription automatically generated

 

Step 4: Click Continue in the first box that pops up

 

image

Step 5: In the next window paste in the name of the Review Group and click Continue.

 

A screenshot of a computerDescription automatically generated

 

Manual add:

 

1. Starting from the submitted Status Tracking record, click on the Related tab, and choose Review Group Applications. 

 

 

2. Click +New Review Group Applications 

 

 

3. The New Review Group Application form will open and autofill the Request and Status Tracking field. All you need to do is assign the Review Group. The Recommended Amount and Recommended Award Status fields will only populate if the Lead Reviewer Recommends setting in the Review Group = Recommend to Request or Recommend to Review Group. If this setting is not being utilized for the review group, those fields will not auto populate. Save.

 

 

 

Now, the submission(s) you just added will appear in the review group’s dashboard in GOapply. Repeat this process for each submission that the review group needs to review.  

 

REVIEWER RESPONSES 

 

Reviewer responses are recorded in the Review Responses entity, Reviewer Request Score entity, in the Reviewer Scores tab of the Review Group Application entity, and in the Responses tab of the Reviewer record.  

 

REVIEW RESPONSES 

 

Review Responses is an entity that stores every response from each reviewer. For example, if your reviewer form has five questions, every time a reviewer submits the form, five reviewer response records will be created.

 

Inside a Review Response record, you will see:

 

 

  • Question Type: the type of question from the Form Builder toolbox, i.e., radiogroup, dropdown, text, file.
  1. Question: The name of the question 

 

  1. Response: The choice selected or the comment entered by the reviewer 

 

  1. Reviewer: A lookup to the Reviewer record 

 

  1. Review Group: A lookup to the Review Group record

 

  1. Request: A lookup to the associated Request record 

 

  1. Review Group Application: A lookup to the Review Group Application Record associated with this response 

 

  1. Status Tracking: A lookup to the associated Status Tracking record 

 

  1. Public: Whether this question was Public or not (a property in Form Editor) 

 

  1. Score: What the associated score was for the choice made by the reviewer 

 

  1. Weight: The weight that this question carries 

 

  1. Conflict of Interest: Whether or not this submission was a conflict of interest  

 

REVIEWER REQUEST SCORE 

 

Reviewer Request Score records contain: 

image

 

  1. Reviewer: A lookup to the Reviewer record 

 

  1. Review Group Request: A lookup to the Review Group Application record 

 

  1. Reviewer Form Response: A reviewer form response json file 

 

  1. Score: The score attributed based on the reviewer’s answer to the question 

 

REVIEWER RECORD         

 

Reviewer responses and scores can also be seen from within the Reviewer record. 

 image

 

REVIEWER REPORT 

 

In the Review Groups entity, you have a report called GOapply Review Responses that you can run to see each reviewer’s responses for each application assigned to their group.

 

 

 

 

Comments (0)