Overview of Open Response Assessments

Key Elements of ORA

1. Prompts: In ORA, prompts serve as stimuli for learners' responses. These prompts can include analytical or descriptive questions and may also allow for the uploading of image or text files. There may be one or more Prompts or questions, learners are asked to answer. Each prompt can contain useful information for learners, such as guidance on the expected word or sentence count for their responses, the types of files they are permitted to upload, or what they can anticipate after submitting their answers. From the learner's perspective, each prompt is displayed above the area where they enter their responses.

2. Rubric: A rubric is used as a criterion for assessing responses. This tool consists of criteria that describe the expected features of a response and provides options for evaluating each criterion. Rubrics are required for all assignments. Across all types of assessments in an ORA assignment, whether it be a self, peer, or staff assessment, each response is graded based on the same Rubric. Regardless of how many Prompts there are in a problem, you must add a Rubric to it. As part of the assessment process, the assessor observes the corresponding rubric for each Prompt and compares the submission with it in order to grade the ORA. An assessment rubric consists of several criteria as well as additional options for each criterion:

1. Criteria:
   All students’ responses should meet the characteristics described in each criterion. An example would be what concepts to cover in a response or how much supporting information should be provided. It is worth noting that a prompt and a name are assigned to each of the Criteria.
1. Generally, the name of a criterion corresponds to a one or two-word summary of the criterion, such as "Content" or "Organization". It is important for the Criterion Name to be unique within the assignment. Also, once the assignment has been released this name cannot be altered.
   
2. The Criterion Prompt states how the response should be evaluated according to the criterion.
   
2. Options: The criterion typically has a range of Options, usually ratings, to indicate how well each response fits the criterion. For instance, options include but not be limited to “Fair,' “Good," or “Excellent." In each criterion option, a name, an explanation as well as a point value are provided.
   

1. The Option Name summarizes the rating in one or two words.
   
2. An Option Explanation provides information that helps the assessor determine whether a response matches the rating or not. Therefore, you must ensure that each option is explained in a way that is as specific and detailed as possible.
   
3. The Option Point Value represents how many grade points this option is worth.
   

Note: In a given assignment, there can be a variety of options for different criteria. Additionally, it is possible to include a criterion without options, but with an entry field for learners or staff to provide feedback.

1. A prompt based on a criterion
   
2. Option names for the criterion
   
3. Detailed descriptions of each option
   
4. Options and their grade values
   

3. Assessment Stages: It is possible for assignments to include some or all of the following steps and phases.

1. Peer Assessment Only: Peer assessment is the step in which learners evaluate the responses of other students within the course. Each response is then scored by selecting an option for each criterion in your rubric based on the response provided. Not to mention, the learners are able to evaluate each response, include textual feedback, or add comments.
2. Self-Assessment Only: During the self-assessment process, the learner can see their own response as well as the corresponding rubric. Also, within the self-assessments just as with the peer assessments, a rubric is provided that outlines the criteria which will be evaluated by the learner, and they then choose an option for each criterion.
3. Self-Assessment to Peer Assessment: During this assessment, the learner should first complete the self-assessment process and then evaluate their peers.
4. Self-Assessment to Staff Assessment: The learner should first complete the self-assessment process and then be evaluated by an individual member of the course team.
5. Staff Assessment Only
   ‏: The staff assessments include evaluating a learner's response via the use of an individual member of the course team. The course team members will evaluate the responses according to the rubric provided for the prompt.

Creating an Open Response Assessment Assignment

Steps to Create an ORA Assignment

Step 1- Add the Component: To begin creating an open response assessment component, follow these steps:

Important: Only one ORA component should be added per course unit, as having multiple ORA assignments can lead to submission errors for learners.

1. Open the unit in Studio where you intend to set up the ORA.
   
2. Under the Add New Component section, select Open Response.
3. Choose an appropriate template from the list of Open Response Assessment options.
   
4. Click on the Edit button in the problem component that appears.
5. Use the component editor to incorporate prompts, the rubric, and to configure additional settings for the ORA component.
   
6. Remember to click on Save after each editing session. You can continue to make adjustments until you publish the unit.
   

Note: Once an ORA assignment is published, you cannot modify the structure of the rubric or change the point values assigned to each criterion. Corrections to typographical errors in the rubric will only be visible to learners who have not yet started the assignment. Nevertheless, it is possible to adjust the due dates and the weight of the ORA assignment after publication.

Step 2- Add Prompts: You can format the text and include images in an open response assessment prompt in the same way you would for a Text component. To add prompts or questions to your ORA assignment, follow these steps:

1. In the open response assessment component editor, click on Prompt.
   
2. Replace the example prompt with your desired prompt.
   
3. To add another prompt, click on Add a Prompt and repeat this step for each additional prompt.
   

Step 3- Add the Rubric: Develop a clear and well-structured rubric that outlines the scoring criteria for the learner responses, ensuring it covers the expected characteristics of the responses.

1. In the ORA component editor, click on the Rubric tab.
2. In the section for the first Criterion, input the name and prompt text (limited to 100 characters) for your initial criterion.
3. In the Options sections related to this criterion, provide a name, explanation, and point value for each option you wish to include.
4. To remove any options, click Remove located at the top right corner of the Options section.
5. To add more options, click on Add Option.
6. Next to Feedback for This Criterion, choose a value from the dropdown menu:

1. Select None if you do not want to allow feedback for this specific criterion.
2. Choose Required to make feedback mandatory for this criterion.
3. Select Optional to permit feedback without making it compulsory.
   

1. Repeat the previous steps to create additional criteria. To add more criteria than those provided in the template, click Add Criterion at the end of the criteria list.
2. Under Feedback for This Response, provide instructions for learners on how to give overall written feedback on the responses they are evaluating. You may keep the default text in the Feedback Instructions and Default Feedback Text
   fields or replace it with your own wording.

Step 4- Assignment Schedule: Establish a timeline for your assignment that includes deadlines for submissions, peer reviews, and grading. In the Schedule tab of the ORA component editor, you have the option to configure the schedule and due dates for your ORA assignment using one of three different modes:

1. Configure deadlines manually: By selecting this option, you have the flexibility to establish individual deadlines for each step of the Open Response Assessment. The following options are available:

1. Response Start Date and Response Start Time: These settings determine when learners can begin working on their responses. Until this date and time, learners are unable to enter text or upload files. Afterward, they can start their work and submit files.
2. Response Due Date and Response Due Time: These settings specify the deadline for when learners must complete and submit their responses. Once this date and time passes, submissions will no longer be accepted for the assignment.
3. Peer Assessment Start Date and Peer Assessment Start Time: These settings indicate when learners can start assessing their peers' responses. Before this time, learners will receive a notification that peer assessment has not yet commenced.
4. Peer Assessment Due Date and Peer Assessment Due Time: These settings specify the deadline for completing peer assessments. After this date and time has elapsed, learners will be unable to assess their peers. It is crucial that learners assess the required number of peers by this deadline to receive a grade.
5. Self Assessment Start Date and Self Assessment Start Time: These settings determine when learners can begin self-assessing their responses. Prior to this time, learners will receive a message indicating that self-assessment has not yet started.
6. Self Assessment Due Date and Self Assessment Due Time: These settings specify the deadline for completing self-assessments. After this date and time, learners will no longer have the ability to complete their self-assessment.
   

2. Match deadlines to the subsection due date: When you select this option, all due dates for the Open Response Assessments (ORA) will automatically align with the due date of the subsection they belong to. Instead of specifying individual dates and times for submission, peer assessment, and self-assessment deadlines, all of them will be set to the same due date as the subsection. This approach offers several advantages: 

1. Alignment with Other Assignment Dates: ORAs will share the same due date as other problems within the same subsection, simplifying the course timeline and reducing complexity for students.
2. Utilization of Grace Period and Individual Extensions: By configuring the deadlines this way, ORA assignments can take advantage of the grace period and individual learner extension features.

Note: Since submission and assessment deadlines are all set to the same date with this option, there will be no buffer time between the response due date and the peer assessment due date. If you choose this setting for a peer assessment ORA, it is essential to clearly communicate to learners that they must submit their responses early enough to allow their peers adequate time to complete their reviews.

3. Match deadlines to the course end date: When this option is selected, all due dates for the Open Response Assessments (ORA) will be automatically set to the end date of the course. Instead of assigning individual dates and times for submission, peer assessment, and self-assessment, all deadlines will align with the course's end date. This setting is particularly beneficial for self-paced courses.

Note: Since submission and assessment deadlines are all set to the same date under this option, there will be no buffer time between the response due date and the peer assessment due date. If you choose this setting for a peer assessment ORA, it is crucial to inform learners that they must submit their responses early enough to allow their peers sufficient time to review them.

Step 5- Select Assignment Steps: Open Response Assessment (ORA) assignments can incorporate various assessment steps, including learner training, peer assessment, self-assessment, and staff assessment. To add the steps to your open response assignment, follow these actions:

1. In the ORA component editor, select the Assessment Steps tab, look for the following headings:

1. Step: Learner Training
   
2. Step: Peer Assessment
   
3. Step: Self Assessment
   
4. Step: Staff Assessment
   

2. Check the boxes for the steps that you want to include in your assignment.

3. (Optional)To rearrange the order of the steps, simply drag them into your desired sequence using the bar on the left side of the steps list.

Step 6- Specify Additional Settings: After adding your prompt and rubric, the next step is to specify additional settings for the assignment. These settings encompass the type of response learners are required to submit, assignment dates, and whether learners will have access to a list of the top-scoring responses.

1. Display Name: Specify the title of your assessment to be displayed to learners.
2. Specify the Response Type: Learners can submit written responses, files, or both as part of their assignment submissions. If you require learners to upload files, ensure that your prompt includes clear instructions regarding the necessary files and acceptable file types. To define the types of responses required, follow these steps:
   

1. In the ORA component editor, select Settings

2. For Text Response, choose one of the following options:

1. None
   
2. Required
   
3. Optional
   

3. In the Response Editor field, choose the editor type that students will use to format their responses:

1. Simple text editor: A basic text field without formatting options.
   
2. WYSIWYG Editor: A visual text editor that supports text formatting.
   

4. For File Upload Response, select one of the following options:

1. None
2. Required
   
3. Optional
   
4. If you select Required or Optional, two further options will appear: Allow Multiple Files and File Upload Types.
5. For Allow Multiple Files, choose True or False. Setting it to True enables learners to upload multiple files. If you want to restrict submissions to a single file, set it to False.
6. For File Upload Types, choose one of the following options:
   

- PDF or Image Files

- Image Files

- Custom File Types: If selected, a File Types field will appear where you can enter the desired file name extensions, separated by commas.

5. For Allow LaTeX Responses, select either True or False.

6. Top Responses: You can decide if learners should see a section displaying the highest-scoring responses submitted for each question in the assignment. If enabled, this section will only appear after a learner has completed all steps of the assignment, and you can specify the number of top responses to display. To configure the Top Responses section:

1. In the ORA component editor, select Settings.

2. In the Top Responses field, specify the number of responses you want to display in the Top Responses section, which will appear below the learner's final score.

If you prefer not to include this section, set the number to 0. The maximum number allowable is 100.

7. Teams Enabled:  To configure a Team ORA, enable the option Teams Enabled by setting it to True. This action will present a dropdown menu labeled Selected Team-set, which defines which group of teams is eligible to submit a response for this assignment. Students can be part of only one team within each team-set. Therefore, by configuring multiple team-sets, you can create groups tailored for different assignments. For example, you might establish separate team-sets for homework, midterm exams, and the final project, allowing students to collaborate with different classmates for each type of assignment.

1. Show Rubric During Response: By default, learners will not see the rubric while working on their responses. However, you can enable this feature to allow learners to view the rubric during their work, so they understand how they will be evaluated. To enable this feature:

1. In the ORA component editor, select Settings.

2. Set Show Rubric During Response to True.

When this setting is activated, a collapsible section will appear during the response step, positioned above the first prompt, providing learners with a detailed breakdown of the grading criteria.

Note: This rubric is the one you established in the Add Rubric step. Learners will be able to see all option names, descriptions, and point values for each criterion.

1. Enable Flexible Peer Grade Averaging: To minimize the likelihood of learners becoming “stuck” in the waiting phase, you can enable the Flexible Peer Grade Averaging feature, which reduces the requirements for receiving a grade during the peer assessment step. When this feature is activated, seven days after a learner submits their work, the number of peer reviews required to receive a grade will decrease to 30% of the original requirement, rounded down to the nearest whole number.
   

Note:  Flexible Peer Grading
will never decrease the required number of peer reviews below one. If a learner has not received any peer reviews after seven days, the very first peer review they obtain will constitute their entire peer grade.

Step 7- Specify Settings:  Once you have selected the steps you want to include in your assignment, you can specify settings for each of those steps.

1. Learner Training: For the learner training step, you will enter one or more example responses that you have created and specify the expected option for each criterion in your rubric. To add and score learner training responses, complete the following steps:

1. Under Step Learner Training, click on View / Add Sample Responses. This will expand the section to show the sample responses you have already set up.

2. Select Add Sample Response.

3. In the Response field, input the text of your example response.

4. Under Response Score, select the option you want for each criterion.

1. Peer Assessment: For the peer assessment step, you will define the number of responses each learner must grade, the number of learners required to grade each response, as well as the start and due dates. All fields in this step are mandatory. To specify peer assessment settings, follow these steps:
   

1.  Locate the Step: Peer Assessment heading.

2. Click on View Options & Configuration to display the step settings.

3. Next to Must Grade, enter the number of responses that each learner is required to grade.

4. Next to Graded By, enter the number of learners that must evaluate each response.

5. Next to Enable Flexible Peer Grade Averaging, select True if you want to enable this feature.

1. Self Assessment: For the self-assessment step, you will specify the start and end times for the assessment phase.
   

1. Locate the Step: Self Assessment heading and enable it.

2.  Switch to the Schedule tab.

3. Under the Self Assessment Deadlines heading:

1. Next to Start Date and Start Time, enter the date and time when learners can begin their self-assessment.
2. Next to Due Date and Due Time, input the date and time by which all self-assessments must be completed.
   

1. Staff Assessment: For the staff assessment step, there are no additional settings required after you have selected the step to be included in the assignment.

Step 8-: Test the Assignment: To effectively test your Open Response Assessment (ORA) assignment, you can follow these steps:

1. Set Up the Assignment: Begin by setting up the assignment in your course.

2. Schedule Future Dates: Configure the section or subsection date to a future date. This allows you to control when the assignment will be accessible.

3. Publish the Unit: Once the assignment is set up and the dates are configured, publish the unit.

4. Recruit Beta Testers: Invite one or more beta testers to participate. These testers will submit their responses and grade each other's work.

5. Gather Feedback: After the testing phase, ask the beta testers to provide feedback on various aspects of the assignment. Specifically, inquire whether they found the questions and rubric clear and easy to understand, as well as if they encountered any issues during the assignment process.