In an ORA or an Open-response Assessment learners must primarily hand in their essay responses; hence, to end the assignment, they must also go through a number of assessment steps such as peer evaluation or self-assessment.

Please note that cohorts are not respected when open-response assessments are visible to all learners. That is to say, students from one cohort may be assigned the task of grading responses that were submitted by another cohort’s students. If you are interested in creating open-response assessments that can be divided between cohorts, then in order to use this assessment you must do so within a course component that has been defined as cohort-specific. Check out Making Use of Cohorts in Your Courses and How to Create Cohort-Specific Course Content for more information on cohorts.

The following topics will explain and discuss various concepts related to open-response assessments.

• Open-response Assessment Essentials
• Means of Calculating Open-response Assessments Scores
• Overview of Open-response Assessment Best Practices

Several elements are included in an ORA assignment.

• There may be one or more Prompts or questions, learners are asked to answer.
• There is a Rubric, according to which all prompts in the assessment are graded.
• There may be one or more Assessment Steps. In other words, throughout an assignment, it is possible to include a learner training step, a peer-assessment step, a self-assessment step, and a staff-assessment step.

In a given assignment, there can be a variety of options for different criteria.
Additionally, it is possible to include a criterion without options, but with an entry field for learners or staff to provide feedback. Check out Supplying Criterion with Comment Exclusive Fields to learn more in this regard.

1. A prompt based on a criterion
2. Option names for the criterion
3. Detailed descriptions of each option
4. Options and their grade values

During the assessment process, learners do not see the criteria names, but they do appear on the page where the learner's final ORA grade is displayed.

See Adding Rubrics for more information on creating a rubric.

The following demonstrates what an example of a criterion and its options might look like within a given rubric.

The Sample Criterion

Name: Evidence

Prompt: Does this response provide evidence for the theory of evolution? (5 marks)

Options

Points	Name	Explanation
0	Not at all	This response does not provide any proof to validate the theory of evolution.
1	Direct Observation	This response addresses the similarities that exist amongst the living organisms alive today and similarities between embryos that can be witnessed via direct observation, providing anatomical evidence.
3	Fossils & Remains	This response also compares and contrasts ancient organism remains found within fossil layers, providing fossil evidence.
5	Molecular Biology	This response not only points towards the other available evidence but also discusses DNA and the genetic code that reflect shared ancestry and the relations between species, which is one of the strongest proofs for evolution. Any hints towards artificial selection and breeding may also be accepted.

Steps Involved in the Assessment Process

Additionally, you can also specify how your assignment will be assessed by specifying the Assessment Steps. It is possible for assignments to include some or all of the following steps and phases.

• The Learner Training Step
• The Peer Assessment Step
• The Self-Assessment Step
• The Staff Assessment Step‏

If you choose to include a Learner Training step, it is required to also include a Peer Assessment step. Moreover, it is necessary to perform Learner Training before Peer Assessment or Self-assessment can be undergone. Additionally, edX recommends placing Peer assessments before Self-assessments given both are utilized within the assignment. Lastly, a Staff assessment should be the last step in any assignment that includes one.

Upon reviewing the assignment, you can see the types and orders of the assessments. In the example above, upon submitting their responses, learners must first complete a learner training step (“Learn to Assess Responses”), then they must perform peer assessments on other learners’ responses (“Assess Peers”), and finally, carry out a self-assessment (“Assess Your Response”).  

An Overview of the Learner Training Steps

Throughout the Learner Training stages, learners are taught to perform their assessments as part of their learning process. Each learner training assessment contains one or more sample responses written by you, along with the scores you would assign. These responses are then reviewed by learners who attempt to grade them similar to yours.

As previously mentioned if a Learner training phase is included within the assignment, it must be followed by a peer assessment; learner training must precede both peer and self-assessment.

During a learner training assessment, the Learn to Assess Responses stage launches right after a student submits a response. In addition to the rubric, the learner also sees one of the sample responses you created; however, the scores you assigned to the response aren't visible. There is also a preview of how many sample responses the learner will assess.

After selecting an option for each assignment criterion, the learner selects Compare Your Selections with the Instructor's Selections. In the event that all of the learner's selections match those defined by the course team, the next sample response will automatically be presented.

Learners see the response again if their answers differ from those specified by the course team, with a message informing them that their assessments differ from the instructor's.

Learners continue to try scoring the sample response until their scores meet the course team's scoring standards.

Read the Complete Guide to Learner Training for more information in this regard.

An Overview of the Peer Assessment Steps

Peer assessment is the step in which learners evaluate the responses of other students within the course. Each response is then scored by selecting an option for each criterion in your rubric based on the response provided. Not to mention, the learners are able to evaluate each response, include textual feedback, or add comments.

According to edX, if you include both peer and self-assessments, you should place the peer assessment first.

Check out Calculating ORA Scores to can find out how the results of peer assessments influence a student's assignment grade here.

Number of Responses and Assessments

It is possible for some students to submit their responses without performing any peer assessments, as a result, some responses might not receive the required number of assessments. Therefore, to ensure learners receive sufficient scores on all responses, it is essential to assign a greater value for the number of responses they are required to assess in comparison to the number of assessments each response must receive. For instance, suppose you mandate three assessments per response, that would mean the learner could be required to assess five responses in return.

Under the circumstances that some students have not managed to achieve the required number of peer assessments while all the responses have already been assessed, those participants can then carry out peer assessment on responses that have been previously assessed by other learners. Subsequently, the learner to whom the response belongs can then review the extra peer assessment whilst viewing their scores. Although, it is important to note that the additional peer assessments do not contribute to the response's score.

Each criterion has a comment field, which appears below the options that are available for that criterion.

You can learn more about Adding Rubrics and Supplying Criterion with Comment Exclusive Fields.

In addition, participants can choose to carry out further peer assessments by clicking on the Assess Peers heading to expand the step once again; hence, click on Continue Assessing Peers.

Moreover, edX recommends that if you plan to incorporate both peer and self-assessments into your course, you include the peer assessment before the self-assessment.

The final grade for an assignment will not be determined until the staff assessment step has been completed. During staff assessment steps, learners receive scores that override scores from other assessment types, such as the peer assessments that follow staff assessment steps.

ORA assignments with staff assessments are best suited to courses with smaller groups of students.  Suppose you decide to create an ORA assignment with peer assessment and staff assessment for a course that has cohorts, and you choose to make it available only to specific cohort members. For those in the remaining cohorts, you will create a single ORA assignment that consists only of peer assessments. Learn how to create courses that are tailored to learners in different cohorts at How to Create Cohort-Specific Course Content.

Plus, don’t forget to check out How to Perform Staff Assessments in ORA Assignments to learn all about grading staff assessments in an ORA.