xAPI Badging and Achievements at the University of Southern California (USC-ICT)

By Nick Washburn (Director Learning Division, Riptide Software)

 

The University of Southern California’s Institute for Creative Technologies (USC-ICT) is a US Army University Affiliated Research Center (UARC). Our team has been utilizing xAPI in our work with the University of Southern California’s (USC) Institute for Creative Technologies (ICT).

It conducts research and development for training across multiple domains including simulation, games, artificial intelligence models, virtual and mixed reality, and more. Among their many programs is the Captivating Virtual Instruction for Training (CVIT) program. CVIT is a multi-year research effort seeking to produce blueprints for mapping effective instructional techniques used in successful live classroom settings to core enabling technologies, which may then be used for the design and development of engaging virtual and distributed learning applications.

There are three different projects within the CVIT program using xAPI in some way. In our first project USC behavior scientists were looking to deliver a learning experience with badging and progress indications. USC has access to the raw interaction data in their LRS. They had specific xAPI reporting requirements to display in a learner’s view. Getting at the raw data in the LRS without an administrative dashboard was easy enough as their current learning experience and dashboards both access and display some near-real time xAPI data summarizing learner activity and results.

For CVIT, we use our HTML5 courseware (eLearning software) framework that includes both pre-baked xAPI recipes and a method for adding additional xAPI tracking. General concepts such as: activity, page and topic relate specifically to how the eLearning software is setup. A course is made up of one or more topics, and each topic is comprised of one or more pages. An activity is an interactive section such as a drag-and-drop exercise, scenario, video, assessment, which is usually on a single page. We also integrate to Phaser.io, an HTML5 game framework using HTML5 Canvas.

ACTIVITY: QUESTION VERB: INTERACTED ACTOR: USER

Contextual Data

Type

Description

activity

{string}

Associated Activity Name

questionId

{string}

Unique Question Id

questionText

{string}

Question Text

correctAnswer

{array}

Questions Correct Answer(s)

givenAnswer

{array}

User's Answer Choice(s)

possibleAnswers

{object}

Question's Possible Answer(s)

page

{string}

Associated Page Number

topic

{string}

Associated Topic Name

CVIT Intelligence Architecture Online Course (IAOC)

Within the courseware framework, we designed IAOC around a simulation game in which the learner is given tasks to complete training in a certain amount of tries. IAOC supports two use cases: to deliver standalone by individual course participants, or to be part of an instructor-led course accessed from the classroom. The instructor module supports blended learning (e.g., instructor-in-the-loop) and focuses on three categories of activities all using xAPI:

  1. Knowledge: Core IA comprehension (Text, Interactive Diagrams, Videos, Flash Cards)
  1. Application: Applying knowledge to problem scenarios (Game-based Learning). This is a video game developed in Phaser.io (an open source framwork for Canvas and WebGL powered browser based games).
  1. Evaluation: Causal performance reviews (Performance screens based on learning metrics).

xAPI is used to record statements each step of the way and our HTML5 courseware application interprets collections of those statements as learner achievements and badges. USC needed a system of badges and achievements with criteria of actions or successful activities that need to be met to “earn the badge”. We have done this kind of gamification before, as well as some badging, but this comprehensive use case required us to think about achievements more holistically. We didn’t want to code ourselves into a corner.

Thinking of this course using an “achievement engine” helped to abstract away the logic into more of a service type. For instance, if one must complete five videos to achieve a video badge, it would have to consist of five unique videos. We didn’t want learners to watch the same video five times to earn the achievement. However, this use case starts to point to a badging system, and some of these badges would represent core competencies or critical learning paths being achieved.

What if the instructors included an assessment after a learner watched all five videos, and the learner must get 80% correct to earn the video badge, or some other learner process flow? If we need to change the rules we can do it more rapidly by abstracting away the rules in an achievement engine. We can keep these systems reusable and interoperable when we develop logic and rules based on xAPI data outside of the LRS by querying it on behalf of the learner and/or mine for and react to patterns as the data comes into the LRS. Remember, the LRS is simply learning activity stream data. It holds the statement that the learner Mastered something, and the stream of activities and verbs it took to get there. What defines Mastery may change. In the case of an Achievements Engine, rather than hard code a solution based on what we think Mastery is today,  we can calculate a stream of xAPI activities and results that signify “something” but ensure it can be changed according to the learner process models (or business process flows) needed at a given time.

CVIT IAOC an example of the learner's achievement view

CVIT IAOC an example of the learner’s achievement view

 

CVIT IAOC learner dashboard

CVIT IAOC learner dashboard

Metric

Formula

Notes

Scenario Score

# Correct / (# Correct + # Wrong + c * # Hints)

For a single scenario attempt. c is a constant between 0 and 1, which might be manipulated at the user-level (e.g., should be an on-create property for a user)

Scenario Mastery

Avg (Scenario Score of Last 3 Attempts)

One of these for each scenario

Role Mastery

Avg (Scenario Scenario Mastery for Scenarios in Role)

One of these for each role

Role-Type Mastery

Avg (Role Mastery of Roles in the type)

My Role, Other, or All

Role Efficiency

# Correct / (# Correct + # Wrong + c * #Hints)

For all scenario attempts in a role

Entity Efficiency

# Correct and Clicked on Entity / (# Correct and Clicked on Entity + # Wrong)

For all scenario types, when "entity" was a valid answer. Only consider when "Wrong" or they clicked on entity (e.g., don't count if they clicked other right answers)

Below is a generated xAPI statement records the achievement earned by the user, John Smith

 {
    "actor": {
      "mbox": "mailto:john.smith@email.com",
      "name": "John Smith"
    },
    "verb": {
      "id": "http://adlnet.gov/expapi/verbs/achieved",
      "display": {
        "en-us": "achieved"
      }
    },
    "object": {
      "id": "http://elmnts.com/courseware/achievement/Basic%20Cable",
      "objectType": "Activity",
      "definition": {
        "type": "http://adlnet.gov/expapi/activities/objective"
      }
    },
    "context": {
      "registration": "1395fab2-1e95-40a7-98d7-a35d3ca2d576",
      "extensions": {
        "http://elmnts.com/courseware/pageId": "1",
        "http://elmnts.com/courseware/topicId": "2"
      }
    },
    "timestamp": "2016-06-27T19:48:40.308Z"
  }

CVIT Supervisor Development Course Refresher (SDC – R)

CVIT SDC-R is designed as refresher training and certification that features engaging gameplay and instructional videos. Using an HTML5/Game  library (Phaser.io) in combination with HTML5 courseware, supervisors can practice their interpersonal skills with a virtual workforce. SDC does not use a comprehensive achievement system but we do track completions, scores, progress and interactions using xAPI.

CVIT SDC-R Main Menu

CVIT SDC-R Main Menu

 

CVIT SDC-R Activity

CVIT SDC-R Activity

xAPI Client for Phaser.io

Both SDC and IOAC have a Phaser.IO game component to the learner experience. For both IAOC and SDC we use a “game listener” or xAPI middleware for Phaser.IO. The game programer/creator in Phaser does not need to know xAPI for us to get xAPI statements from the game they just need to publish messages from the game and we subscribe to them and convert the messages and data into xAPI. This is referred to as a  publish-subscribe pattern or  PUB/SUB and it is a great fundamental to remember when you want to get different softwares talking xAPI. If you have PUB/SUB option available it removes the need for the software engineers to understand xAPI, they simple need to publish events. Have you ever used Google Analytics? You just drop this little javascript on your html page and it works! This is probably the best way to describe our “game listener”. It’s an xAPI client that allows for easy xAPI statement generation beginning with published events in some other language. Incidentally we have been working on refining the xAPI client concept to share out to the world.

CVIT Advanced Situational Awareness (ASA)

The CVIT ASA courseware is intended to be a “train the trainer” type of application to support a week long in-person training. The in-person training is highly dynamic and popular and teaches skills in human pattern behavior recognition. The purpose of the CVIT program is to extend existing and identify new techniques used by successful military instructors in the delivery of classroom material. Those techniques are then mapped to core enabling technologies like: virtual humans, branching videos, and interactive exercises, that are incorporated into the design and development of engaging virtual and DL applications. CVIT ASA breaks down into six domains of practice and offers progressive activities that help to reinforce the six domain fundamentals. We use xAPI to track and score the users with a simple scoring and badging system. As well as the default interactions of the HTML5 courseware, we are also tracking data with USC requirements like showing if the learner changed the answer before they hit submit and more. Learners can re-take the exercises until they earn the highest badge.

CVIT ASA activity

CVIT ASA activity

 

CVIT ASA badge is awarded at end of graded progressive activity and badge persists in menu

CVIT ASA badge is awarded at end of graded progressive activity and badge persists in menu


NICK WASHBURN (Director Learning Division, Riptide Software)

Nick WashburnNick Washburn has a 20+ year career working with software companies and hi-tech entrepreneurs in distance learning, and web based software. Currently, Nick is Director of the Riptide Software Learning Division, bringing Riptide Elements® learning products to market. Elements core product development efforts align with the goals of the Advanced Distributed Learning directorate, and Nick is a contributing member of the community, researching, developing and working within the standards for xAPI. During 2009-2012 he led the design and development of an award winning multilingual training product line at a leading training provider for Fortune 50/500 companies. In 2006, Nick led the design and development of the award winning Future Soldier Training System (FSTS) used by the U.S. Army.

Speak Your Mind

*