The external examiner process is a common process where degree-awarding bodies engage external examiners to provide impartial and independent advice, as well as informative comment on the degree-awarding body's standards and on student achievement in relation to those standard Curriculum mapping.

Using the combination of Curriculum and Evaluation a full integrated, digital External Examiner process can be integrated in the curriculum development and maintenance process.

The examiners are part of the defined process and will be involved at the right moment in time. The external examiner is supported using a Curriculum workflow in providing relevant information and directed to the evaluation (in Evaluation) to evaluate the specification, study, module and deliver feedback. The feedback will immediately become available in the Curriculum process.

The process starts in Curriculum, using the default configuration options of Curriculum.

An example implementation is used to show the actions to be performed.

The configuration will not be handled in detail, since that is available in the other manuals on the respective configuration subjects. Only some specifics for the EE process will be pointed out.

The example below shows the sample process configuration with the different statuses, workflows and roles. In this case the External Examiner process is a separate modelled process that is automatically started (propagation) by the Specification creation and review process.

The difference with standard processes is the usage of a number of Evaluation related hooks:

• Create and publish evaluation: sent a request to Evaluation to generate a new evaluation with the external examiner as participant and the other relevant relations to manage and provide feedback on the evaluation.
• Publish evaluation: once the evaluation is finished and feedback is given by the internal organisation, the evaluation report(s) are published
• Request feedback: sent a request to Evaluation to enable feedback mode, so the relevant stakeholders can provide feedback on the external examiner results
• Close evaluation: after finishing the process, the evaluation can be closed for all users.

The same hooks are also available from the process management overview to manage and intervene in the process when required.

Another new addition is the option to sent the the evaluation out to multiple examiners, but wait with proceeding to the next status only after all examiners have finished. In case this options is set, it is shown as a checkbox under Wait for last participant. This configuration (Proceed after last) will cause the transition to only proceed when ALL participants have finished their task.

In the process manager, the finished participants will have a green checkbox that will show the completion date when hovering.

‍

The process will sent the relevant notification to the external examiner and provide an overview of the outstanding evaluations to the external examiner.

Clicking an outstanding task will open the workflow for the external examiner.

The workflow is a standard workflow with in this example configuration two specific pages related to the external examination.
In the context of the Evaluation integration three new standard page templates are made available

• evaluation-feedback: page providing direct access in Curriculum to the feedback sections in the external examiner evaluation report.
• evaluation-participate: page providing direct acces to the evaluation in Evaluation. Used to lead the external examiner to the evaluation.
• evaluation-report: page providing direct to the evaluation report, used for stakeholders providing feedback, the final report for the examiner, ...

Not all information is provided via the evaluation. There is also information provided that is related to the Curriculum or managing the process.
Since the external examiner starts and runs through the process in Curriculum, based on authorisation specific information can be amended or added by the examiner.

A special case, added for the EE process, is the option to provide 'examiner related' information to the object (e.g. Specification) being examined. Since differente examiners can work on the same object / Specification it should be possible to add person-related data that is not visible to other examiners.

To solve this request, the standard object extension in Curriculum is used, in combination with the Item page template.

The example below shows the personal information provided by the External Examiner, using an item page with the flag 'Show mine' set.

The example below shows the same information as above seen from another stakeholders perspective. This stakeholder is allowed to see the information provided by the different examiners and uses an Item page with 'Show mine' not set and the field 'Created by' selected as to be shown on the page.

The external examiner evaluation is using the basic configuration options provided by Evaluation. Some specific additions and configuration options are added to support the different requirements posed by an external examiner evaluation.

The main two difference are

• examiner evaluations are personal, where student evaluations are multiple respondent for a specific object, e.g. a module.
• standard student evaluations will lead the students through the evaluation in a specific order and restrict change after submitting an answer.

Both differences are implemented as configuration option, so it's not strictly examiner bound, but just a new type of evaluations that is supported.

The example below shows an example of an external examiner evaluation.

New added is the left hand navigation option that will support the examiner in getting direct insight in the different sections to be covered, navigate freely between the different sections and have immediate insight in the status (answers) per section.

The information shown behind the section is: [total questions answered] / [total questions] [icon] for each block:

• Green Counter & Checkmark: Indicates all questions answered, including a tooltip.
  To change the tooltip text use the label: Page.Participate.Progress.Done.Tooltip.
• Orange Counter & Half Icon: Indicates unanswered non-required questions (initial state).
• Red Counter & Asterisk: Indicates at least one required question is unanswered.

If there are unanswered questions a tooltip is shown, e.g. "Unanswered: 2 required, 1 left".
To change the tooltip text use the label: Page.Participate.Progress.NotDone.Tooltip

The example below shows the situations:

• General questions: All questions answered (green checkmark)
• Flexible education: No questions answered, all 5 questions are optional (orange checkmark)

A special case is the final section (submit the results).

This will provide an overview of the status and the option to view all given answers in one overview by clicking 'Show my answers'.

Not the submit button will only be enabled if all mandatory questions in the evaluation have been answered.

As said not the external examiner report is added, but a configurable way to support another type of evaluations with only two new configuration options on Evaluation type:

• Sidebar navigation: Select the checkbox to enable the sidebar navigation. Selecting will cause only the sidebar navigation to be enabled, the tabular / bread-crumb navigation will not be shown.
• Show answers after completion: Select the checkbox "Show answers after completion". This feature, which is disabled by default, allows users to review all questions and answers in an expandable section. All users can view their answers before they submit the survey.

‍

‍

Based on one of the earlier mentioned big differences a new report type has been introduced. Where the student evaluations combine and report on the results from the different students, the examiner report is a report based on a single user.

Anew report type PARTICIPANT_REPORT is introduced enabling the definition of reports based on individual participant responses. This feature requires a new template for the evaluation type and the REPORT_ALL access level.

As part of the implementation of the different template additional security is added to prevent users from accidentally report on individual students. Only using a specific template AND having the correct authorization will allow reporting on individual persons.

‍

Next to that a new visualisation option is added to the reports to better cover the larger reports with different sections and amount of questions. The report improves the readability and navigation of reports by using the visualisation option Cards view.

The new report has the following visualisation additions:

• Sticky, Collapsible, and Interactive Sidebar
  A configuration options is added to display a sidebar that offers direct navigation to each block, feedback, summary, tags, filters, etc. This sidebar is user-friendly, automatically updates on scroll, and can be used to navigate to different sections of the report. The sidebar can be hidden using the button in the top-left corner and is only visible on desktop.
• Support for text and bullet type questions
  The cards support all the visualisations, but only text and text bullet visualised questions will be displayed in a separate sub-card.
• Show/hide blocks
  A configuration is added to only show the questions in cards view and don't show the block they are part of.

The image below shows an example of the new reporting option.
The default options like preview (to beatify for printing) and export are still supported.

A new option is made available to enable providing feedback directly to individual participants directly in the report.

Using the configuration options Allow writing feedback to participants on the evaluation type and Feedback editable on the report, it's possible to give feedback to individuals directly from the report. And of course the user should have FEEDBACK rights.

Once configured the individual feedback option, each of the individual reports will show an Edit-button underneath the configured 'feedback areas', enabling to define participant specific feedback.

‍

‍

‍

‍

REPORT, USABILITY - New reporting visualisation option, including navigation - EV-147

‍

‍

‍