Print bookPrint book

Outcomes Reports

Outcome Manager Block

Site: Remote-Learner.net
Course: Remote-Learner.net
Book: Outcomes Reports
Printed by:
Date: Friday, 23 June 2017, 2:55 PM

Outcome Manager Block

The Outcome Manager block is available for Moodle 1.9 only, it is not available for Moodle 2.0.
  • This tool enables Schools to load State Standards from Academic Benchmarks into their Moodle site and tag quiz and/or lesson questions with the standards.
  • When standards change, the school can update their site with the new standards by reloading the file from AB (when AB provides the new file).
  • When the standards are loaded and questions are tagged, Schools can generate detailed reports on how their students are doing against state standards.

Standard outcomes must be added to the site before they can be added to courses and linked to questions. This is done by adding the Outcome Manager block to the site home page and uploading an outcomes XML file.

The Outcome Manager block at the site level is intended to import outcomes and assessments only. The reports are generated from the Outcome Manager block at the course level. That is covered in the Outcomes Manager Block for Courses section of this page.

Outcome Manager block on a site home page:
Outcome Manager bock on site home page

Outcomes


Select Import Outcomes from XML to import outcomes to the site. Any outcomes can be imported as long as the XML file is formatted correctly. The following page has more information about how to setup the outcomes XML files for import.

Importing Outcomes from XML:
Importing outcomes interface

The XML import process converts the XML into a tree structure of outcomes. This tree structure is what users will see when linking outcomes to questions in courses. Also, the outcomes will be referred to as objectives once uploaded. The next image shows an example of the National (Stem) Math Standards objective when editing a quiz question.

Objectives

Assessments


Assessments represent outcomes that have been flagged as assessed. Each outcome can be flagged as assessed by importing assessments via XML. The assessed items are used for the Objective Mastery report. Only assessed outcomes will be shown in the Objective Mastery report.

The assessments are uploaded using the Import Assessments from XML link in the Outcome Manager block. The upload screen here is a little different than the outcomes upload screen. The assessment XML will need to reference the content file (outcomes file) because assessments only mark outcomes as assessed. The content file will be used to establish the location of the outcomes being assessed in the system. The content file can be chosen from the location it was uploaded to (site files > blocks > outcome_manager folder - make sure to select the Choose link located to the right of the file name) or it can be uploaded again along with the assessment file.

Importing Assessments from XML:
Uploading assessment page
The following page has more information about formatting the outcomes and assessment XML files for import.


Outcome Manager Block for Courses


The Outcome Manager block for courses will allow you to add objectives to a course and generate outcomes reports. To add this block, go to the course home page, turn editing on, and then add the Outcome Manager block as shown in the next image.

The Blocks block

The Outcome Manager block on a course home page:
Outcome block at the course level

For the reports to work we will need to assign objectives to the course and then link the objective's expectations to questions in quizzes or lessons. The objectives are represented by the site level outcomes added to the site. To add objectives to the course select the Assign Objectives to Courses link in the Outcome Manager block.

The next image is showing the Add Objectives to Course screen. Follow the steps in the image to add objectives to a course. The box in step 1 is listing all the objectives imported to the site. The National Math Standards is the objective I just uploaded. Keep selecting the title in the next section until you have selected the standards in the Objective section, then select the Add Selected Objectives button.

Add objectives to course:
Adding objectives to a course

Objectives can be deleted from the course by selecting the X next to the objective only if no activities are using those objectives.

Deleting an objective in a course

Once the objective has been added to the course, its expectations can be linked to questions in quizzes/lessons. The example in the next image is assigning expectations to a quiz question.

The objectives need to be linked to questions in order to generate data for reports. For quizzes, this can be done from the question bank or directly from the quiz. The objectives can be linked to questions when adding or editing quiz questions. For lessons, the objectives are linked to questions in the same way, when adding or editing questions in the lesson.

The following image shows the objective added from the image above. The objective listed will be available whenever adding/editing questions in quizzes and lessons for this course. This example has only one objective, shown by the first level of the tree structure. The objective has three sub-levels, or the objective consist of 4 levels total. The last level is referred to as Expectations, the objective is linked to questions via the checkboxes in this level. The outcomes reports will be using the Objective or Expectation levels for report data.

Objectives

In the example shown above I have used the Expand All button to show all the levels of the objective. The objectives can also be expanded on an individual basis. To link an objective to a question select the checkboxes next to the objective's expectations. Multiple expectations can be selected.

Note: When linking outcomes/objectives to questions, be sure to hit the 'Expand All' button before saving changes. The top of the previous image has a red arrow pointing to the button. If checkboxes are selected and then the branch is minimized before saving changes, then the link to the question will be broken. This is scheduled to be fixed in the ELIS 1.9 versions.

The following pages have an example of an outcomes XML file and examples of the outcomes reports.

XML Formatting for Outcomes/Assessments

Outcomes


The XML file used to import the outcomes needs to be properly formatted to successfully upload the outcomes. The following XML images showing the beginning of the National (STEM) Math Standards will help explain this.

There are entries in the XML file that are critical to the upload working correctly. The most important one is the version number assigned to the outcomes, this number has to be unique. Do not upload outcomes with the same version number to the site.

In the following image the version number is inside of the <grade> tags. The version number we assigned the National (STEM) Math Standards for grades 9-12 is "200.1 Grade 9". The ".1" in the number is representing this version of these standards, this is the first version uploaded to the site. If these standards change next year, we would change the version number on the new standards to "200.2".

XML example of outcomes

The following image shows the next section of this XML file. In this file the outcomes will be structured in 4 levels. Level 1 is represented the by <grade> tag. The other 3 levels use the <topic> tags.

Level 1 is represented by the <grade> tag in the image below. There is only one level 1 entry, the objective.

Level 2 is represented by the first <topic> tag that follows the <grade> tag. The id for this topic is Math-Standard-1. This <topic> does not close until the fourth to last line of the image. Anytime a new <topic> is opened without closing the previous <topic>, a new level is started.

Go to the first <topic> in the image again. If you follow the file for a few lines you will come across another opening <topic>. This <topic> represents level 3. The id for this topic is Math-Standard-1-A.

The next <topic> is opening level 4. Level 4 has the expectations that will be linked to questions and used in reports. There is no level 5, so every level 4 <topic> has a closing </topic> tag. The id for this topic is Math-Standard-1-A-1. The objectives listing image, shown below, shows the Math-Standard-1-A-1 expectation.

All the <topic> tags for Math-Standard-1 will have to be closed before starting Math-Standard-2, the second level 2 entry in the XML image. So at the end of Standard 1 in the XML file (third to last line in image below) we have </topic>, </topic>, </topic>. The first </topic> is closing Math-Standard-1-C-2; the second is closing Math-Standard-1-C; and the last one is closing Math-Standard-1.

XML example of outcomes

There are a quite a few entries that are made for each <topic> in the XML file example. Here's a list of the items in the example above with brief descriptions:
  • id - Each entry/branch of the outcome tree is assigned an id. The image of the XML file above provides examples. The id of the first <topic> is Math-Standard-1.
  • parent id - For level 3, the previous level 2 entry is the parent id. In the second <topic> of the example, the parent id is Math-Standard-1. The id is Math-Standard-1-A.
  • state_num - This is the last character of the topic id. For the first <topic>, Math-Standard-1, the state_num is 1. For Math-Standard-1-A, the state_num is A.
  • label - This is only used for the first level of topics in the example.
  • seq - Each <topic> is assigned a sequence number. The first topic has seq=1, the second topic has seq=2, etc.
  • level - This is referring to the level of the outcomes/objectives; just levels 2, 3, and 4 have this entry. This is a somewhat misleading in the XML example, because the first level <grade> does not have a level number. Only the topics are being assigned a level number. The first topic tag in the XML is labeled level 1, but it is actually level 2 in the outcome tree structure.
  • CDATA - This contains the objectives content.
Objectives Listing:
Objectives

Select this link to view the objective listing and xml images side by side. The objectives listing image is showing an example of the outcomes in a quiz once the xml file has been uploaded to the site and the outcomes have been added to a course.

Assessments


The assessment XML files are formatted slightly different than the outcomes XML files. First, the version number needs to be changed in the assessment XML. Remember that the version numbers need to be unique - Do not import different outcome or assessment files with the same version numbers.

Second, assessments are assigned at the objective level not the expectation level. To flag the entire objective as assessed we must include all the elements of the objective. In the example outcome XML file above there is only one objective. So the entire outcome file will be included in the assessment file, the only change is the version string.
XML example of assessment file

Assessment Notes:

Assessment files only flag outcomes as assessed. That's why a content file is uploaded/chosen with the assessment file, to establish the location of the outcomes being flagged as assessed in the system.

The Objective Mastery Report is designed to work with the assessed objectives only. It's the only report that uses assessed objectives.

Student Response Report

The Student Response report will show user scores for a quiz/lesson and will also show a user's correct and incorrect answers for multiple choice questions.

The report example below is showing each user's username that took the quiz, the user's score and pass/fail mark for the quiz, and the last column is showing the user's answers on the 3 quiz questions. The + symbol means their answer was correct. Incorrect answers have the incorrect choice the user selected.

This report is intended to be used with multiple choice question types. The report will not be fully functional unless multiple choice question types are used.

To generate a Student Response Report select the Student Responses Report link located in the Outcome Manager block at the course level. Next select a quiz or lesson you want to generate a report for and select Run Report. All quizzes and lessons from this site with questions linked to objectives will be available from the drop down menu here, not just the ones from this course.

Student Response Report

To export the report, select one of the file format icons located in the upper left hand corner of the report. The report will download locally to your computer. If Flash format is selected a new window will open and display the Flash version.

Item Analysis Report

The Item Analysis report shows the objective's expectations that are linked to a question in the quiz/lesson along with answers for the question. The following is a description of each column of the report from left to right:
  • Item Number - The question number from the quiz/lesson.
  • Description - The description of each objective linked to that question.
  • Correct Answer - The correct answer for the question shown next to each objective assigned to that question.
  • Answers selected by users - The last columns show the amount of times each answer was selected by users.
This report is designed to use multiple choice question types like the Student Response report.

The report options are just about the same for each outcomes report. First select the type of activity, quiz or lesson. Then enter the standard and select the quiz or lesson the report will be generated for. When the options are set correctly, select the Run Report button. From this screen Item Analysis reports can also be generated for all the quizzes/lessons, that have questions linked to objectives, on the site.

Item Analysis report

The last page of the report shows a bar chart, each bar represents the percentage of correct answers for that quiz question. The report options of the previous image has a field to enter the standard for this report, each bar from the chart in the following image has a red line representing that number.

Last page of Item Analysis report

To export the report, select one of the file format icons located in the upper left hand corner of the report. The report will download locally to your computer. If Flash format is selected a new window will open and display the Flash version.

Objective Mastery Report

The Objective Mastery report only shows data for objectives that have been flagged as assessed. Objectives in the system can be flagged as assessed by importing assessment items via XML. This is done from the Outcome Manager block on the site home page.

This report shows data for this flagged objective, the report is calculating the amount of expectations from this objective that are linked to questions in a quiz/lesson. This data is displayed as a total for the entire objective, shown as a single bar in the chart.

In the report example shown in the image below there is only one objective linked to this question, though any number of expectations from the objective could be selected. If there were multiple assessed objectives here, there would be multiple bars in the chart.

The following is a description of the data points in the report from left to right:
  • Objective - The objective number flagged as assessed and linked to a question in quiz/lesson. The objective is linked to questions via checkboxes in its expectations sub-level.
  • Description - The description of each assessed objective linked to a question in the quiz/lesson.
  • Correct/Wrong - The amount of correct/wrong responses for an assessed objective. Multiple expectations from each assessed objective can be linked to questions. The correct/wrong responses will be counting the selected expectations from each objective.
  • Total - Total responses linked to the assessed objective.
  • Weight - Weight of each objective, this is decided by the total number of responses linked to the assessed objective or the "Total" column. In the report example below, the weight is 100% because there is only one objective.
  • Mastery - The percentage of correct responses for the assessed objective.
This quiz used in the report example below only had one of the three questions linked to a objective flagged as assessed. Two expectations from the objective were selected for the question. The report will be counting responses to this question twice for each user, because there were two checkboxes selected within the flagged objective for this question.

Currently (ELIS 1.8.2) this report is counting user data twice for the quiz totals, the individual totals are correct. This error has been corrected and the fix will be included in the ELIS 1.8.3 update.

The report options are just about the same for each outcomes report. First select the type of activity, quiz or lesson. Then enter the standard. Select the quiz or lesson the report will be generated for. When the options are set correctly select the Run Report button. From this screen reports can be run for all the quizzes/lessons on the site with questions linked to objectives.

Objective Mastery report

The report is showing the data for the five users who have taken this quiz. Their names are listed to the right side of the report, the names link to individual reports for each user.

Individual Object mastery report

To export the report, select one of the file format icons located in the upper left hand corner of the report. The report will download locally to your computer. If Flash format is selected a new window will open and display the Flash version.

To return to the previous report screen, select the icon with the backward pointing arrow in the upper left hand corner of the report.

Expectation Mastery Report

The Expectation Mastery report shows the expectations added to questions along with users results for those questions. The expectations in this report represent the last level in the outcomes tree structure, the section where checkboxes are selected to assign the objectives to questions.

The following is a description of each column of the report from left to right:
  • Expectation - The objective number linked to the question in quiz/lesson.
  • Description - The description of each objective linked to a question in the quiz/lesson.
  • Correct/Wrong - The amount of correct/wrong responses for an expectation.
  • Total - Total answers linked to the expectation.
  • Weight - Weight of each expectation, this is decided by the total number of responses linked to the expectation or the "Total" column.
  • Mastery - The percentage of correct responses for the expectation.


The report options are just about the same for each outcomes report. First select the type of activity, quiz or lesson. Then enter the standard. Select the quiz or lesson the report will be generated for. When the options are set correctly select the Run Report button. From this screen reports can be run for all the quizzes/lessons on the site with questions linked to objectives.

Expectation Mastery Report

The last page of the report is showing the Expectation # and the Master % in a bar chart.
Expectation Mastery Report bar chart
Notice the user names to the right of the bar chart in the image above. The user's names link to individual Expectation Mastery reports.

Individual Expectation Mastery Report:
Individual Expectation Mastery report
The expectations with two responses in this report were linked to two of the questions in the quiz.

To export the report, select one of the file format icons located in the upper left hand corner of the report. The report will download locally to your computer. If Flash format is selected a new window will open and display the Flash version.

To return to the previous report screen, select the icon with the backward pointing arrow in the upper left hand corner of the report.

Outcome Manager Block for Students

Students of each course will be able to access a student version of the Outcome Manager block when viewing the course home page. There are 3 outcomes reports available to students. The following images show examples of the reports.

Two of the student reports are the same as the administrator reports, the Expectation Mastery Report and the Objective Mastery Report. The student version of these reports only includes that students data. The Objective Analysis Report is a report that is only available in the Outcome Manager block for students.

Outcome Manager block:
Students outcome manager block

Objective Analysis Report


The Objective Analysis Report uses a different report options screen than the other outcomes reports, users select a class here instead of a lesson or quiz. Users can generate a report showing scoring totals for all objectives linked to questions in quizzes and lessons in a course.

Objective Analysis report

Objective Analysis Report


Expectation Mastery Report


Expectation Mastery Report


Objective Mastery Report


Objective Mastery Report