Print bookPrint book

Data Hub Basic

A manual for Data Hub Basic

Site: Remote-Learner.net
Course: Remote-Learner.net
Book: Data Hub Basic
Printed by:
Date: Sunday, 20 August 2017, 7:30 AM

1 Data Hub Overview

Data Hub Basic:

Data Hub Basic overview:
Moodle is a powerful system for delivering courses, however there are often times when users need to integrate with information from other systems such as SIS, EPR, HR, and Financial Record keeping systems. To facilitate this, Remote-Learner has built the Data Hub, a tool for 2 way communication of information between Moodle and other systems - users, courses, enrollments in and grades out. DH also provides a way to quickly setup Moodle courses by uploading a formatted CSV file and using template courses.

Update Frequency:
Inbound information (to Moodle):
Currently incoming information (users, user information, enrollments, courses, etc.) is updated every 5 minutes by default. This can be changed in the Data Hub Block settings.
Outbound information (from Moodle):
Grade and course information is exported every 24 hours by default. This can be changed in the Data Hub Block settings.


Data format(s):
The current version of Data Hub provides for import of CSV (comma separated values) files - a common format that can be easily created and/or edited in most data management tools (including Excel, OpenOffice, Access, etc.). Subsequent versions of DH will support XML files and WebServices.

Data Categories:

User data- this is data about the user, that includes some or all of the information that goes in the user profiles in Moodle. See below for a detailed description of the user data handling in DH Basic.

Course data- this is data about course properties that may be set by the data import. See below for a detailed description of these properties.

Enrollment properties- this is data about the enrollment status of a user - which courses the user is enrolled in, what their status is, what their role in the class is, the completion status for their courses, etc. See below for a detailed description of these properties.

Automating Data Import/Export:

Data Hub automatically imports files placed in it’s import folder every 30 mins by default and loads any new files that are placed there into Moodle. If your source data system (SIS/ERP/HRMS, etc.) can be setup to automatically export files, it can send them to the DH target folder in various ways – for instance via SCP, shell scripts, etc.

Since source systems are all different, we can’t guarantee that a particular client system can be fully automated, but it if it can be set to automatically export CSV files, then it is generally a simple matter for the system's administrator to set it to automatically export those files to the Data Hub Basic target folder. Once that is done, Data Hub will load the files every 30 minutes by default.

We work with clients to help their system administrators setup their systems on an hourly basis, if necessary (cost will depend on the specific system, how well documented it’s export functions are, etc).

2 Data Hub Basic Block Configuration

To view the Data Hub Basic block configuration go to the Site Administration menu and select Modules > Blocks > Data Hub. The DH file locations are entered and described on this screen.

It is important to let Remote-Learner staff set up these locations for you so that your user data is put in a secure location that will enable DH Basic to operate correctly. If these locations are blank, please contact your Sales Representative to purchase the DH Basic setup.

Data Hub block configuration
  1. Import file location: This is where files will be placed to be processed by Data Hub. The files placed here will be automatically processed every 30 minutes by default. Files can also be processed immediately by selecting the Save and Process button on the User, Course, and Enrollment tabs. Selecting the Import all button on the General tab will import all three files at once.
  2. Export file location: Export files are processed every 24 hours by default, this time can be changed by the "Export period" setting, see #8 below. Export files can be processed immediately by selecting the 'Export now' button on the General tab. If you don't want to have DH export any files, leave the Export file location field empty.
  3. Log file location: Location of all import and export log files - every time the batch file runs, a log will be written, describing in detail what actions were taken by the import script.
  4. Email notification: Enter the id numbers of users that should be emailed the log files. The system will then email log files to the email associated with the id number. Note that a user's id number must be the same as their username for the notifications to work. This will be fixed in Data Hub 2.0.
Data Hub block configuration screen
  1. Import period: Set the import frequency. An example of the time format is 3d, 4h, 50m. This would import files every 3 days, 4 hours , and 50 minutes.
  2. Create groups and groupings: Automatically create new groups in Moodle courses.
  3. Import date format: The are 3 import date formats, whichever format is selected here must be used in the import files.
    • M/D/Y - MM/DD/YY
    • D-M-Y - DD-MM-YY
    • Y.M.D. - YY.MM.DD
Data Hub block configuration screen
  1. Export period: Set the export frequency. An example of the time format is 3d, 4h, 50m. This would create an export file every 3 days, 4 hours , and 50 minutes.
  2. Unique export file names: This gives each export file a unique time stamp. If this is selected then each export will have a separate file name. If this isn't selected then the previous export file will be replaced when a new export file is processed.
  3. Include all historical data: If this is selected all the completion data for the site is included in the export report each time it is run. If this isn't selected, then only completion data since the last report is included.
  4. Disable cron: This will turn off the standard cron process for Data Hub - this will stop DH from processing files, so please makes sure Remote-Learner support confirms that the DH Cron is operational before this is activated.
Moodle relies on a timed procedure, know as the Moodle cron, to run many recurring tasks, such as sending notifications, forum posts, and messages, running reports such as ELIS reports and completion checking, and integration related batch processes such as LDAP synchronization. By default, Data Hub also uses the Moodle cron to process enrollment files. When these files exceed 40,000 lines of data, it may take too long to run, causing other operations to fail to run.

To address this issue for very large enrollment/user creation files, we implemented a separate system-level cron process to call the Data Hub processes by itself which will not impact the Moodle cron from running regularly.

If you need to run very large enrollment files, please open a support ticket to have the optional large file processing cron task setup for you - checking the box in the example above will not setup the DH cron for you, it will just disable the standard cron, meaning Data Hub will no longer run.

Remote-Learner support will have to configure your server to run the actual cron to process the Data Hub files. This also requires that your hosting level be at our Hosting Level 3, or above, or on our Red Hat Enterprise Virtual server system.

Data Hub block configuration screen
  1. Using DH Basic: This setting lets you switch between using Data Hub Basic and ELIS Data Hub. When this is selected, Data Hub Basic will work - when this is not selected, ELIS Data Hub will work.

3 Accessing the Import, Export, and Log Folders Via SFTP

This page will demonstrate how to access the Data Hub import, export, and log folders via SFTP. There are two examples, one for Windows and one for MacOS.

Windows Access


WinSCP will be used to access the folders via SFTP in this example. The following link has a free WinSCP download, http://winscp.net/eng/index.php.

The WinSCP login screen is shown in the next image. To login and access the Data Hub file locations you will need to:
  1. Enter the host name or server address.
  2. Enter a username and password for accessing the Data Hub folders via SFTP. Remote-Learner will setup the username and password.
  3. Make sure SFTP is selected from the file protocol drop down menu.
WinSCP login

Once logged in the Data Hub folders must be located. Select the folder icon as shown in the following image to enter the file path in the Open Directory window.

WinSCP

The file paths are located on the Data Hub block configuration screens. Go to Settings block > Site Administration > Plugins > Blocks > Data Hub plugins > Version 1 import and Version 1 export. The file paths are setup by Remote-Learner.

Enter the file path in the open directory window.

WinSCP open directory pop-up window

The next screen is showing the import folder on the right hand side of the image. From this screen you can drag and drop files into the import folder. When files are dragged and dropped into the folder or out of the folder they will be copied to the new location.

WinSCP drag and drop


MacOS Access


Cyberduck will be used to access the folders via SFTP in this example. The following link has a free Cyberduck download, http://cyberduck.ch/.

The Cyberduck login screen is shown in the next image. To login and access the Data Hub file locations you will need to:
  1. Select the open connection plus (+) symbol.
  2. Make sure SFTP is selected from the file protocol drop down menu. This can be configured as the default setting.
  3. Enter the server address, e.g., elisdemo.remote-learner.net.
  4. Enter the username and password for accessing the Data Hub folders via SFTP. Remote-Learner staff will setup the username and password.
  5. Enter the file path. The file paths are located on the Data Hub block configuration screen. Go to Settings block > Site Administration > Plugins > Blocks > Data Hub plugins > Version 1 import and Version 1 export. These file paths are setup by Remote-Learner staff.
Cyberduck login

Once the correct folder is opened files can be copied to the folder location by dragging and dropping them into the folder. When files are dragged and dropped into the folder or out of the folder they will be copied to the new location. Files in the folders can also be copied to your local system by double clicking on the files.

Cyberduck with import folder open

4 Accessing Data Hub Basic

To access Data Hub Basic, add the Data Hub block to the site. Go to the Site Administration block > Modules > Blocks > Data Hub and make sure the "Use DH Basic" setting is selected at the bottom of the page.

Data Hub block

Select the Data Hub page link to open DH Basic and display the General tab as shown in the next image.

The User, Course, and Enrollment tabs and files are explained on the following pages. Please review those pages before getting started with the 'import all' or 'export now' functions on the General tab. You will need to understand how to setup the different file types before successfully importing files.

IP Basic general tab


Import All


The 'Import all' button will import and process all three file types in the order they are displayed on the General tab - user, course, and then enrollments. For example, by importing all three file types at once you can:
  1. add users to the site,
  2. create courses, and
  3. enroll the users in the courses.
The documentation on the following pages demonstrates how to setup the import files.

Import files can be set to run automatically as frequently as every 5 minutes. Go to the Site Administration block > Modules > Blocks > Data Hub > Import period to edit these settings.

Import period settings

Note: When using the Import All button do not upload empty files to be processed, this will cause the subsequent files to be deleted without processing and won't produce an error message. To avoid this don't include empty files in the import folder. When Import All is selected Data Hub will check for all three file types, if only one file is present then DH will process one file. This issue will be fixed in the Data Hub re-write, due to be released with ELIS 1.9.3.


Export Now


The 'Export now' button exports user data for courses in a CSV format. Export files can be set to run automatically as frequently as every 5 minutes. Go to the Site Administration block > Modules > Blocks > Data Hub > Export period to edit these settings.

Export period settings

All the CSV export files are stored in the Export file location, the file location is shown on the Data Hub block configuration screen.

5 Importing User Information

Users can be loaded into Moodle via a CSV file. The name of this file is indicated in the File name field. This is configurable, for example if your record system's user export process produces a file called users.csv, or people.csv, you can type that name in the File name (1) field and save changes. The file name entered here is the file name that Data Hub will import and process from the 'Import file location'.

Currently DH only supports csv files, however subsequent versions will support XML, webservices, etc. At present, you can only choose csv.

2. The user properties mapping table enables you to map the names of user properties from your export file to properties in DH.

Data Hub Basic user property map with tabs

Data Hub Basic user property map

Any of these parameters can be set or updated via the Data Hub import file. If you make changes to any property values, be sure to save your changes with one of the save buttons on the bottom of the screen.

Moodle custom user profile fields will be on the users property map. In DH Basic, the custom user profile fields all start with "profile_field_" then the shortname of the custom field, but this can be configured in the user property map.The following page has additional information about using custom profile fields.

The properties that can be set via the user csv files are listed below with brief descriptions:

idnumber: User's id number.

auth: Enter the user authentication type here. The example below this chart has more information.

username: Username is a required field.

password: Password is a required field.

email: Email is a required field.

firstname: Firstname is a required field.

lastname: Lastname is a required field.

city: City is a required field.

country: The country name should be entered as it would in Moodle, e.g., United States. This is a required field.

maildigest: Enter no digest, complete, or subjects.

autosubscribe: Enter yes or no.

trackforums: Enter yes or no.

timezone:

language: Check the Moodle site for the appropriate entries, use the shortname shown in parentheses. For example, English (en_us).

theme: Enter the shortname of the theme.

screenreader: Enter yes or no.

description: Enter text.

institution: This is an optional field, located in the Optional section of the Moodle user profile.

department: This is an optional field, located in the Optional section of the Moodle user profile.

action: The actions for user files are add, update, disable.

profile_field_coursecode: This is a custom profile field for the site. For this entry, enter "profile_field_coursecode" in the header, then add a value for this profile field in the column for the user.


Authentication (auth): Follow the steps below to find the value that should be entered for different authentications. This is not a required field, if a value is not entered then the default authentication will be used.
  1. Go to the Site Administration block > Users > Authentication > Manage Authentication.
  2. Mouse over (do not click) the Settings link of the authentication type, this will display the url shown in step 3.
  3. The end of the url displays the name that should be entered as a value in the csv file, the example is showing that "alfrescosso" should be entered for the Alfresco SSO authentication.
Manage user authentication setings

The following list is showing the names of standard authentications and the entries that should be made for each authentication in the csv file.

Authentication Names CSV Entries
Manual accounts manual
No login nologin
Email-based self-registration email
Dummy authentication plugin for ELIS elis
Dummy authentication plugin for Mahara mahara
Moodle Network authentication mnet
Alfresco SSO alfrescosso
OpenID openid
CAS server cas
Curriculum Management System Authorization crlm
External database db
FirstClass server fc
IMAPserver imap
LDAP server ldap
NNTP server nntp
No authentication none
PAM (Pluggable Authentication Modules) pam
POP3 server pop3
Radius server radius
Shibboleth shibboleth


Required Fields


The required fields for importing user files are:
  1. username
  2. password
  3. firstname
  4. lastname
  5. email
  6. city
  7. country
  8. execute/action

Data Hub Actions


The 'execute' field from the image above is represented by the 'action' column in the CSV files. DH tells Moodle what needs to be done with imported data via the Actions column. The following actions can be performed for users:
  1. add - add a user
  2. update - update a user
  3. disable - disable a user - this deletes the user and all of their data.
The sample file below illustrates this function, there is also a link to download this csv file below the image:
Data Hub Basic user file

Select this link to download the sample CSV file used in the example above for adding/updating/disabling users.

The first column of the file contains the action field, which tells DH to add, update, or disable the user. In the file the first two users are being added to the site. The third user is being updated. The fourth user is being disabled, which will delete the user and their records. The actions column enables DH to provide incremental updates, as well as enabling the updating or deleting of users via the automated process.


Password entries:
  • changeme - if changeme is entered for the password, users will be forced to change their passwords when they access the site.
  • * - if * is used for the password, the user will be able to login with * regardless of password requirements.
  • no entry - users with no entry for their password will be added/updated with a blank password, the users will be able to login with only a username.

User.csv file FAQ:

  • What if we send an empty file (no actions in that time period)?
    You will get an error saying that the file was empty, no users will be updated, changed, or deleted.

  • What does it mean to “disable” a user?
    Disabled users are deleted from the system.

  • How is the “add or update user” feature different than doing our own LDAP authentication?

    Data Hub supports user fields (such as custom profile fields) that are not supported by Moodle's LDAP plug-in.

Importing/Processing Data Hub Files

The 'Import file location', specified in the Data Hub block settings, is where the files should be uploaded for processing by Data Hub. To complete the import process for the example above I will follow these steps:

  1. Name the CSV file, for this example it's named 'user.csv'. Make sure the file name shown on the User tab screen is the same, user.csv.

  2. Load the file to the 'Import file location'. The files are automatically processed every 24 hours.

  3. To process the file immediately, select the Save and Process button from the bottom of the screen on the User tab.

Once the file is processed a screen appears describing whether or not the file was processed successfully. Below that there will be a link to the log file.

The log file displays each action taken in the file, with a brief description about the success of the action. For example, if a file of 30 users was processed, then there would be a list describing the action taken for each user's record.

A complete log of all actions is kept in the logs file, as well as (optionally) emailed to an address or addresses each time the DH script runs. The log file location is shown on the settings screen for the Data Hub block. And the option to email the log file to a user is available on the same screen.


* Requires that User themes are enabled in the Admin/Appearance settings of Moodle

5.1 Using Custom Profile Fields

Custom profile fields can be set via Data Hub user files.

The Data Hub User tab lists the custom profile fields added to the site. The property that should be entered in the user csv file is shown on the user property map, it can also be edited on this page. Data Hub Basic will always add "profile_field_" before the profile field shortname by default.

Users tab

The property shown on the user tab is entered in the header of the csv file and the value to be assigned for this field will go in the user row.

Example file creating user while setting custom profile field value

The value entered will depend on the profile field:
  • menu of choices: For this profile field assign one of the menu options to the user
  • boolean: For the checkbox profile fields enter "0" to keep the checkbox unchecked, and enter "1" to select the checkbox
  • text input: For text input, just add text

6 Importing Course Information

The Course tab is where you can set the import parameters for course information.

We can set the name of the csv file, as well as the file type (though currently only csv is supported). The file name entered here is the file name that Data Hub will process from the 'Import file location'.

Data Hub Basic course tab picture 1

The course properties mapping table enables you to map the names of course properties from your export file to properties in DH. Any of these parameters can be set or updated via the DH import file.

Data Hub Basic course tab picture 2

If you make changes to the property values, be sure to save your changes with one of the save buttons on the bottom of the screen.

The properties that can be set via the course csv file are listed below with brief descriptions:

category: The course category, e.g., miscellaneous.

format: The course format, e.g., topics or weeks.

fullname: The fullname of the course.

guest: Enter yes or no for guest access to the course.

idnumber: The course id number.

lang: Set the course language. Enter the language as it is shown in the Moodle course, e.g., English (en_us). Currently the site default will be used for this setting, not the value entered in the file for this field.

maxbytes: The maximum upload size for files in the course. Currently the site default will be used for this setting, not the value entered in the file for this field.
metacourse: Enter yes or no.

newsitems: Enter how many news items will be shown, 0-10.

notifystudents: Enter yes or no.

numsections: The number of topics or weeks. If this optional field is used in the csv file, a value must be put in for each course.

password: Enter password if one is used.

shortname: The course shortname.

showgrades: Enter yes or no.

showreports: Enter yes or no.

sortorder: Enter the order of the courses, e.g., if there are five courses assigned a value between 1 and 5, the course with value 1 will be the first course in the list. Currently the sort order doesn't work in sub-categories.

startdate: Enter the start date, e.g., 09/05/2010.

summary: Enter the summary.

timecreated: Time created.

visible: Enter yes or no in the visible column to set the course visibility or availability.

link: Enter the shortname of the template course in the link column of the csv file, if a template course is being used. The next page has more information about creating courses using templates.

action: The actions for courses are create, delete, or update.


The required fields for courses are:
  1. category
  2. fullname
  3. shortname
  4. execute/action
Additionally, you need to specify the topic (format) and number of sections (numsections) for the course(s).

Data Hub Actions


DH tells Moodle what needs to be done with imported data via the Actions column. The following actions can be performed for courses:
  1. create - create a course
  2. delete - delete a course
  3. update - update a course.
In the following example row 2 is updating a course. Rows 3 and 4 are creating courses using the course from row 2 as the template. Row 4 is deleting a course.

In the Category column, /, designates a sub-category. For example, in row 3, Test 2863 is a sub-category of Spacely Sprockets

Data Hub Basic course file

When deleting courses all 4 of the required fields must be included. Line 5 of the in the image is deleting a course.

Select this link to download the file shown in the image above. Note that the courses in rows 2 and 5 were added to the site before the file was processed.


Sub-Categories

Courses can be added to sub-categories and sub-categories can be automatically created via Data Hub Basic.
  • The default column header for this entry is "category". This is configurable in the Course property map.
  • The category and sub-category names are separated by a / in the csv file. For example, if your adding a course to the Spring 2011 category and the Math sub-category, enter "Spring 2011/Math".
  • Sub-categories in the file that don't exist on the site will be automatically created by the file. For example, if the file entry is "Spring 2011/Science", then the Science sub-category will be created if it hasn't been already.
Example file creating a course with sub-categories

Course CSV file FAQ:

  • Does the .csv file have to contain all headers/columns?
    No, just the required headers and the headers for data you want to change.

  • If we leave a field blank, does it override current data or ignore it?
    Blank fields in non-required columns are ignored.

  • Can we use the course ID number for the course template feature or does it have to be the short name?
    The template course's shortname is what is used for identifying the template course.

  • Can we include assigning the teacher to that role in the course to this file, or does it have to be a separate file?
    All role assignments are done via the enrol.csv file, and not by the course.csv file.
  • Can we create courses that are not enroll-able?
    Currently there isn't a method to set courses as not-enroll-able via Data Hub - instead you can create courses and set them to invisible, then run an update and make them visible.

Importing/Processing Data Hub Files

The 'Import file location', specified in the Data Hub block settings, is where the files should be uploaded for processing by Data Hub. To complete the import process for the example above I will follow these steps:
  1. Name the CSV file, for this example it's named 'course.csv'. Make sure the file name shown on the Course tab screen is the same, course.csv.
  2. Load the file to the 'Import file location'. The files are automatically processed every 24 hours.
  3. To process the file immediately, select the Save and Process button from the bottom of the screen on the Course tab.
A complete log of all actions is kept in the logs file, as well as (optionally) emailed to an address or addresses each time the DH script runs. The log file location is shown on the settings screen for the Data Hub block.

6.1 Using Template Courses

Data Hub Basic enables the creation of courses using Moodle courses as templates. A course template is an existing Moodle course - this course will be automatically backed up (without user data) and restored into each course that calls it as a template.

A common use for this functionality is automatic course rollover - when a site administrator wants to create copies of a previous semester's courses for faculty to use in a new semester.

Another common use is to create new courses from a template course - for example if you would like a custom block layout or a set of blocks not standard in Moodle, then you can create a course with that block layout, and set Data Hub to use that course as a template for creating new courses.

You can use as many templates as you like, so for example if one Department wants to create new courses with one set of blocks and/or block layout, and another department wants to create new courses with a different set of blocks and/or block layout, then you can accomplish this by setting one template course for each department's new courses.

The following image demonstrates how to setup a CSV file for this. The fields necessary to make this work are:
  • action - Enter "create".
  • fullname - Enter the name of the new course.
  • shortname - Enter the shortname of the new course. The new course's shortname must be different than the template course.
  • link - Enter the shortname of the template course.
The courses in rows 3 and 4 will be created from a course template. By entering the template course's shortname in the link column of file below, DH will automatically retrieve the zip file from the course and use it to create the new course.

Example file for creating a course from template

Any of the fields from the course properties map can be entered in the CSV file.

Select this link to download this CSV file.

Note - this sample file will only work if you have existing courses on your Moodle site with shortnames in the CSV file - such as PAP, EMM, etc. To use this file on your own site you would need to enter in shortnames of existing courses in the "link" column.


7 Importing Enrollment Information

Enrollment in Moodle is dependent on the context. If a user is enrolled as a student in a course, then they are given the role called 'student' - with permissions that are set for that role in Moodle's roles administration area, in the context of a course. A teacher could also be enrolled in a course - in this case they are given the role of teacher (with the permissions that are set for that role), in the context of a given course.

Data Hub Basic enables you to set the roles of users via the import file, once you have created courses you can enroll users in these contexts. The most common types of enrollment are for students/learners and teachers/instructors.

The Enrollment tab is where you can set the import parameters for enrollment information.

We can set the name of the csv file, as well as the file type (though currently only csv is supported). The file name entered here is the file name that Data Hub will process from the 'Import file location'.

Data Hub Enrollment tab

If you make changes to any property values, be sure to save your changes with one of the save buttons on the bottom of the screen.

The properties that can be set via the enroll.csv file are listed below with brief descriptions:

username: The username for the user that will be assigned the role.

role: The short name for the role the enrollment gives the specified user in the specified context/course. To find the role short name go to the Site Administration block > Users > Permissions > Define roles and the short names of the roles are in the 3rd column of the chart, Short name.

useridnumber: The idnumber for the user.

context: Set the context for the role at system, user, course, or block context. For example, user A is added as a teacher in course A or parent A is added to the parent role of student A.

timestart: The time for the enrollment in question - e.g. if you are enrolling a user via the csv file, and need to set a different enrollment time than the current time. If selecting a future date, the enrollment will not occur until that date.

timeend: The time/date that the user completed the course - used when importing completion data from another system.

instance: The name/shortname/username of the item you want to set the role at, e.g., the shortname of the course being updated.

group: The name of the group the user should be added to. If the group doesn't already exist it will be created.

grouping: The name of the grouping the user should be added to. If the grouping doesn't already exist it will be created.

execute: The actions for enrollment files are add or delete.


The required fields for enrollments are:
  1. role - role shortname
  2. username - of the user being assigned the role
  3. context - the role can be assigned at the system, user, course, or block level
  4. instance- name, username, shortname of the context the role is assigned in
  5. execute/action - enter add or delete

Data Hub Actions


The 'execute' field from the image above is represented by the 'action' column in the CSV files. DH tells Moodle what needs to be done with imported data via the Actions column. The following actions can be performed for enrollments:
  1. add - add users to a role in a specific context, e.g., a course.
  2. delete - remove users from a role
Data Hub Basic enroll file

In the file the following actions are taking place. Rows 2-4 are adding users to courses in a student role, the courses are identified in the "instance" column by their shortname. Row 5 is adding a user to the teacher role in a course. Row 6 is adding Roger Lee to the parent role for Lisa Lee, so the role assignment for Roger will be shown on the Roles tab of Lisa Lee's profile. Row 7 is deleting a user from a course.

Select this link to download a csv copy of the file in the image. Note that the users and courses in the file have to added to the site before this file could be processed correctly.

The Parent Role Use Case page of this book contains more information about setting up a parent role and assigning the parent role to a student user via an DH Basic upload.

Importing/Processing DH Files


The 'Import file location', specified in the Data Hub block settings, is where the files should be uploaded for processing by DH. To complete the import process for the example above I will follow these steps:
  1. Name the CSV file, for this example it's named 'enroll.csv'. Make sure the file name shown on the Enrollment tab screen is the same, enroll.csv.
  2. Load the file to the 'Import file location'. The files here are automatically processed every 24 hours.
  3. To process the file immediately, select the Save and Process button from the bottom of the screen on the Enrollment tab.
A complete log of all actions is kept in the logs file, as well as (optionally) emailed to an address or addresses each time the DH script runs. The log file location is shown on the settings screen for the Data Hub block.

7.1 Assigning Parent Roles

This use case will demonstrate how to:
  1. Create a parent role
  2. Create/upload a CSV file via Data Hub Basic to add users to the parent role of students
  3. Use the new parent role

Creating the Parent Role


To create the parent role go to the Site Administration block > Users > Permissions > Define roles. Go to the the bottom of the page and select the 'Add a new role' button. There are 5 permissions that will be allowed for this Parent role:
  • Users
    1. Edit user profile
    2. See all user blogs
    3. See all user posts
    4. See user activity reports
  • Course
    1. View user profile
This parent role is used with the Mentees block. The Mentees block is where the parent accesses the user information they are permitted to view when they are assigned to the parent role for a user. To add the Mentees block, go to the the site home page, turn editing on, go to blocks and select the Mentees block.


Creating/Uploading the CSV file


There are 5 required fields to create the CSV file to assign parent roles to users via Data Hub Basic:
  1. action: add this user to the following role in the following
  2. context: this user will be assigned a role in the user context
  3. instance: in this example the instance is the student user, enter the username
  4. username: enter the username of the user receiving the new role permissions
  5. role: enter the shortname of the role being assigned
Example file adding a user to the parent role of another user
The users in this example have already been added to the site. If users have not been added to the site, add the users via a user file or other methods first. Use the 'Import all' button located on the General tab to process user and enrollment files at the same time.

Any of the properties included on the Enrollment tab of DH Basic can be included in the CSV file. Some other properties that might be useful with a setup like this would be a start time and stop time for the role assignment.

Once the file has been created it can be uploaded to the site directory. In the following image I'm demonstrating a view from Cyberduck. Once the connection has been established:
  1. Select the file being uploaded.
  2. Drag and drop the file to copy it to the import file location. In the image below the import folder is the import file location. To find the Data Hub file locations go to the Site Administration block > Modules > Blocks > Data Hub > Import file location.
Cyberduck view

Next the Data Hub settings may need to be adjusted before processing the file. The following image is showing the Enrollment tab.

The file name needs to be changed to the file name that was uploaded to the directory so DH Basic can locate and process the file.

Enrollment tab

After selecting 'Save and process' above, the following screen appears stating whether or not the file was imported successfully.

Log file link

Selecting the log file link will display the log of actions taken.

Log file

The new role can be located by going to the student user's profile screen and selecting the Roles tab.

User profile screen

User roles screen



Using the Role


To demonstrate how the role is used I'll login as the user that was just assigned to the parent role. On the site home page the user will see the the student user they were assigned the parent role for in the Mentees block.

Mentees block

If the parent user selects the student user's name from the Mentees block, they will be linked to the student user's profile screen. From here the parent user can access the features they have permissions to use.

Parent view of student user profile screen

The courses the student user is assigned to can be linked to here from the Courses section of the user profile, each course name is a link to the course. The parent user does not have full access to the courses though, just the features they have permissions to use.

8 Log File

This page demonstrates the log file for a group of courses that are imported and processed by Data Hub.

I'm loading the following CSV file to the 'Import file location'.

Example of course csv file

Next I select the Save and Process button from the Course tab. Once processed, Data Hub Basic displays the import file status and a link to the log file.

Log file link

The log file displays the 4 courses DH Basic tried to create with the CSV file. The first three courses were successfully created. The fourth course was not created because I entered 'update' instead of 'create' in the actions column. Data Hub tried to update the course but it hasn't been created yet.

Log file

To access all the log files use the file location specified in the Data Hub block configuration screen. In the following image I've accessed the log folder using SFTP, the application/program shown in the image is Cyberduck. In this example the files can be downloaded by double clicking a log.

Accessing the log file folder via SFTP

9 Exporting Grade Information from Moodle via DH

Data Hub can export final grade information (number and/or letter grades) in csv format. This can be imported to an SIS/ERP record keeping program (Banner, Peoplesoft, SAP, etc.). What is provided by DH is a regularly exported csv file with either incremental data (grades changed/updated since the previous export) or a full export of all grade data on the system. Custom user profile fields can be added to this export.

Export files show course data for users in a CSV format. The export files include the following data:
  • first name
  • last name
  • username
  • user idnumber
  • course idnumber
  • start date
  • end date
  • grade
In Data Hub 1.8.8+ versions, custom profile fields can be added to the export report on the Export tab. The bottom of this page has more information about the new export features in Data Hub 1.8.8.

Export files are automatically generated every 24 hours by default, this can be changed by going to the Site Administration block > Modules > Blocks > Data Hub > Export period. Export reports can also be created manually at anytime by selecting the Export now button on the General tab of Data Hub.

Export now button

The export csv files will be saved in the Export file location. The file location is shown on the Data Hub block configuration screen located in the Site Administration block > Modules > Blocks > Data Hub > Export file location. The location can be accessed via SFTP.

Here's an example of a CSV export file from Data Hub Basic:

Example of an export file

All users assigned a student role in a course for the site will be included in the export file for Data Hub Basic. If a user is in multiple courses, each course will have a separate row in the file.

Currently there is not a way to set end dates for users in Moodle courses, so the end date will be the date the file was processed. Moodle 2.0 may enable a way to set end dates for users in courses.


Letter Grade and Custom Profile field export:


User custom profile fields can now be added to the export report on the Export tab of Data Hub. To add custom profile fields to the report:
  1. Select the Export tab to add/edit custom profile fields for the export report.
  2. Enter what should be entered in the column header for this field in the report.
  3. Select the profile field that will be added to the export report and give the entry a column header. For Data Hub import files, the column header would be the custom profile field shortname by default, the shortname is being used in the examples here.
  4. Select the Add button to add the profile field to the export report.
  5. Once the profile field has been added to the report it can be deleted, its position in the list can be changed, or it can be edited. Selecting the edit icon for a profile field on this screen will give you 2 options, change the header name or changed the profile field. When editing a selection on this screen, select the Update button when finished.
Export tab

The following example shows the export report generated by Data Hub Basic, the "Letter" column on the far right hand side of the report has the letter grade for users. The custom profile field added to the report is also on the far right hand side of the report after the letter grade.

Export file with custom profile field

10 Importing very large user and enrollment files:

Importing very large user and enrollment files:

Moodle relies on a timed procedure, know as the Moodle cron, to run many recurring tasks, such as sending notifications, forum posts, and messages, running reports such as ELIS reports and completion checking, and integration related batch processes such as LDAP synchronization. By default, Data Hub also uses the Moodle cron to process enrollment files. When these files exceed 40,000 lines of data, it may take too long to run, causing other operations to fail to run.

To address this issue for very large enrollment/user creation files, we implemented a separate system-level cron process to call the DH processes by itself which will not impact the Moodle cron from running regularly.

Disable cron setting
  1. This will turn off the standard cron process for Data Hub - this will stop DH from processing files, so please makes sure Remote-Learner support confirms that the DH Cron is operational before this is activated.
  2. This will be the path to your system's DH Cron.

If you need to run very large enrollment files, please open a support ticket to have the optional large file processing cron task setup for you - checking the box in the example above will not setup the DH cron for you, it will just disable the standard cron, meaning DH will no longer run.

Remote-Learner support will have to configure your server to run the actual cron to process the Data Hub files. Running the specialized DH cron also requires that your hosting level be at our Hosting Level 3, or above, or on our Red Hat Enterprise Virtual server system.

11 Automating Data Hub

Once you have developed a process to export data from your SIS/MIS/ERP/HR system into Data Hub's CSV format, you can further automate the process of data import using the standard SCP and scripted SFTP formats.

Please complete DH's setup and make sure the system is configured correctly and data imports are working correctly before adding the step of automating the process.

Due to the wide variety of different possible operating system configurations and SFTP/SCP applications, Remote-Learner's standard support for Data Hub does not include configuring your system to send files to Remote-Learner's servers1.

However, the SFTP protocol has been a standard for man years, and there are many sources for information on how to script it to automatically send files to a remote system.

Here are several tutorials that describe methods to script files to be sent automatically from Unix/Linux and Windows servers:

Windows using the free WinSCP tool:
http://winscp.net/eng/docs/scripting

Linux/Unix:
http://www.snailbook.com/faq/scripting-sftp.auto.html

Please note that these links are offered as examples, they are not meant to be comprehensive nor exclusive, and Remote-Learner's standard support for Data Hub does not include setting up or debugging SCP/SFTP scripts1.

1. Advanced support for building and testing an SCP/SFTP automation process may be available from our Professional Services Group, please discuss this with your Remote-Learner account representative.