All Collections
FAQ
For teachers
How feedback scores were calculated in Scoring 2.1
How feedback scores were calculated in Scoring 2.1
David Kofoed Wind avatar
Written by David Kofoed Wind
Updated over a week ago

The components of the feedback score

The feedback score is a combination of 5 different things:

  1. The constructive score

  2. The submission evaluation accuracy score

  3. The feedback evaluation accuracy score

  4. The feedback completeness score

  5. The feedback evaluation completeness score

The motivation for the feedback score is to give students a meaningful incentive to provide helpful feedback to each other.

The 5 components of the feedback score

We will here describe each of the 5 above scores, and then explain how they are combined into a single feedback score.

The constructive score

This score measures how constructive a student is when giving feedback.

When a student receives feedback from another student, they are asked to rate how constructive the feedback is on a scale, each step corresponds to a score:

  1. Not at all useful (0%)

  2. Not very useful (25%)

  3. Somewhat useful, although it could have been more elaborate (50%)

  4. Very useful, although minor things could have been better (75%)

  5. Extremely useful, constructive and justified (100%)

For an assignment, the received scores on this scale is averaged to obtain a total constructive score. If a student does not answer this question, it is not counted in the average.

As an example, say a student has given feedback to three peers, where the first peer has chosen "Not very useful" for their received feedback, the second has chosen "Extremely useful, constructive and justified" and the third rater has not answered, then the total constructive score is (100% + 25%) / 2 = 62,5%.

If a student has not received any evaluations of their constructiveness from their peers, then they do not get a constructive score.

The submission evaluation accuracy score

This score measures how accurate a student is when evaluating a submission.

When multiple peers have evaluated the same submission, it is then estimated how accurate each evaluation is. The submission evaluation accuracy score is calculated for each rubric criteria, and then averaged together across the criteria.

To get the submission evaluation accuracy score for a student on a single criteria, the following steps are performed:

  1. The average of the answers from all the peers that evaluated the submission on this criteria is found.

  2. The difference between the average answer and the students answer is found. This is called the "inaccuracy" and is a percentage between 0% and 100% where 0% means that the student answered the same as the average answer, and an inaccuracy close to 100% means that the average answer was very far away from the students answer.

  3. The inaccuracy is averaged over the different evaluation criteria to obtain an average inaccuracy for the student.

  4. Finally the accuracy is obtained by taking 100% - inaccuracy.

Now the submission evaluation accuracy is a score between 0% and 100% where an accuracy of 100% means that the student always agreed with their peers. The lower the submission evaluation accuracy score, the more the student disagrees with their peers.

If the teacher has evaluated the submission, then the accuracy is determined as the deviation from the teachers scores instead of the average of the peers scores.

The feedback evaluation accuracy score

This score measures how accurate a student is when evaluating the constructiveness of the feedback they have received.

By default it is assumed that students are accurate when they evaluate the constructiveness of the feedback they have received. The only way to receive less than 100% on the feedback evaluation accuracy score is when the teacher decides to overwrite the feedback evaluation a student has given.

The feedback evaluation accuracy score is best explained with an example. Say that a student has received feedback from a peer. They decide that the feedback is "Not very useful" (2/5 on the scale). The teacher disagrees, and overwrites with the choice "Extremely useful, constructive and justified" (5/5 on the constructivity scale). In this case, the difference is 5 - 2 = 3 with a maximum possible difference of 4. This gives a feedback evaluation accuracy score of 25%.

If the student and teacher agrees, the score is 100%
If the student and teacher disagrees with a difference of 1, the score is 75%
If the student and teacher disagrees with a difference of 2, the score is 50%
If the student and teacher disagrees with a difference of 3, the score is 25%
If the student and teacher disagrees with a difference of 4, the score is 0%

If the student has not evaluated the constructiveness of their received feedback, they do not receive a feedback evaluation accuracy score.

The feedback completeness score

This score measures if a student has given the feedback they were supposed to.

In each assignment the teacher decides how many peers each student is supposed to evaluate. The feedback completeness score is the percentage of peer evaluations that the student has completed compared to how many they were supposed to.

As an example, say each student is supposed to give feedback to 3 other students and a student only completes 2 of these. Then a feedback completeness score of 2/3 = 66,6% is obtained.

The feedback evaluation completeness score

This score measures if a student has evaluated the constructiveness of the feedback they have received.

The feedback evaluation completeness score is the percentage of constructiveness evaluations that the student has completed compared to how many they were able to give.

As an example, say that a student receives feedback from 3 peers, and rates the constructiveness of just the first peer's feedback. Then a feedback evaluation completeness score of 1/3 = 33,3% is obtained.

Combining the 5 scores into a total feedback score

To get the final feedback score, the above 5 scores are combined in the following way:

The Constructive score, the submission evaluation accuracy score and feedback completeness score are averaged into one number A.

The feedback evaluation completeness score" and the feedback evaluation accuracy score are averaged into another score B.

The final feedback score is the weighted average of A and B where A has 80% weight and B has 20% weight (meaning that A counts four times as much as B).

Did this answer your question?