Who are your top performers? Which employees go the extra mile to make everyone in the organization better? Are there opportunities for coaching or performance support that go unnoticed?
This article continues from my last xAPI Quarterly article on “Leveraging Existing Technology for xAPI Data” with a specific focus on forums and discussion groups. Forums and discussion groups provide a vehicle for teams to interact, post questions and have answers crowd-sourced through voting up or down and flagging accepted answers. While these tools are useful to the end-user, the data behind the various interactions can reveal a great deal about the behaviors of your end-users as well as helping to identify star performers. But how do we get this data?
One straightforward way is instrumenting your forum and discussion group engine to write xAPI statements. Using xAPI as a standard protocol to document student experience means that the data is portable across systems. Learning Records Stores can capture data from a number of different sources; mobile apps, hardware – such as internet of things enabled devices, document management systems and even an organization’s Learning Management System. Adding forum and discussion group data helps paint a more robust picture of a student or employee activities and interactions. As with most xAPI projects, it makes sense to invest time upfront to design how the statement should be written and what vocabulary – and specifically verbs – should be used. Remember, without consistent data entry, efficient data retrieval becomes difficult.
Let’s step through some of the normal interactions a user makes with a discussion group or forum.
Joining and Leaving a Group – One of the first things that a user must do before he or she can post to a group is join. Sometimes joining is initiated by the student and in other cases, a student is added to a group by a system or a moderator.
Let’s consider the example where a student is automatically added to a course forum because they registered for a course. In the example statement below, we can see where “Lisa Jones”, our actor in xAPI terms, “Joined”, our xAPI verb, the forum “Planning for Employee Advancement”, our xAPI object. In this case, an xAPI extension is used to note that the join was “Auto”, meaning it was completed by the LMS (in this case) when the student enrolled in the course that included a course forum.
In the case where a student decides to join a discussion group of their own volition, the xAPI statement is very similar in structure but the extension identifies that the join was “Manual”, indicating the student initiated the join.
By including this extension, we can filter reporting on the extension and trend the manual versus automated joins.
Of course when someone leaves a discussion group, we should capture that data too. In the example below, our same actor, Lisa Boss, left the forum on Competencies she joined manually in the last example.
Because we consistently track the Actor and Object, we can now report on information like the elapsed time a user – or the average time a group of users – are active forum member by comparing their “Join” and “Leave” dates.
Posting and Replying to Groups – Once someone is a member of a forum we may want to track their posts and replies. In an aggregate format, this data can help identify discussion topics that have a high-level of interaction. For example, a technical help-focused forum may have a topic or thread for each application in the Microsoft Office Suite. Tracking the number of posts (questions) and responses (answers) will show which applications cause the most confusion and may help a Learning and Development department identify training opportunities for their users.
In the xAPI statement below, Lisa Jones (actor) posted (verb) “Topic 1”; a new discussion thread. Where did she post it? The context identifies the “Parent” which, in this case, is the Forum proper. Not only can we tell how many topics or threads are created but who is posting these topics.
If you are posting a topic, say a technical support question, your goal is to receive a response (and a correct one as discussed later in this article). In the statement below, we can see a new actor, Richard Arnot, replied (verb) to the topic (object). The parent is still identified in the context as is the grouping, tying all the responses to the same topic together.
Voting Up, Voting Down & Accepted Answers – As important as receiving responses to a post may be, identifying the “right” or “good” responses is as important as is identifying “incorrect” or “bad” responses.
In this example, Lisa Boss is again our actor but this time she is “voting up” a specific reply. In the interest of space, the bottom of the statement has been cut but continues after the object to display the context just like the reply statement above. This lets us track which replies are voted up and by whom. Over time you can also trend whose (which actors) responses are being voted up regularly.
Conversely, the voted down verb can be used to track replies that are other users disagree with.
Often forums allow the person making the initial post or a forum moderator to endorse an answer or mark it as an accepted answer. This helps flag for other users that this solution ‘worked’ – especially in the case of a technical support forum.
This data too can be leveraged to identify star performers that go above and beyond by helping their fellow employees and/or other students.
Again, in the example below, we can see the Actor, Richard Arnot, the Verb “Accepted” the Object referring to the message proper and the context is used to tie identify the specific topic and forum.
Deleting a Post – As with most applications, a user may want to delete her own post. A moderator may also need to delete a post that is inaccurate or inappropriate. Within the LRS, xAPI statements are written as a chronological activity log and can’t be deleted according to the specification.
In the case of a “delete” the statement written to the LRS is a “void”. While the xAPI statement for inserting the response remains in the LRS, the “void” statement allows users to ignore it when querying the data. Void is also used in the case that an accepted statement is “unaccepted”.
Verb and Context Consideration – As mentioned above, one of the most critical considerations when generating xAPI statements is consistency. When we first looked at instrumenting forums with xAPI, we decided against storing complete posts by users because there was no valid need for reporting purposes. Instead, we decided reporting on aggregate data was the most important. For example, we can report on:
- Who starts the most topics?
- Who makes the most responses?
- Whose answers get voted up (or down) most often?
- Whose answers are most often marked as “accepted”?
- What is the overall activity level of students in a forum?
- For forums associated with courses, does activity in a forum have any correlation with performance in the course?
With this in mind, we began searching for existing verbs and context properties to use in a forum. Using a combination of ADL’s xAPI Vocabulary and the xAPI Registry we use the following verbs for specific user actions:
Posted a new topic in a forum
Replied to a forum topic
Joined a forum
Left a forum
Marked an answer as “accepted”
Voted up a response
Voted down a response
We developed the following rules:
- If a user marks an answer as “accepted”, then later “unmarks” the accepted answer, we void the original “accepted” statement.
- All messages contain the forum identifier in the context Parent.
- All messages contain the topic (original post) in the context Grouping.
So, what is next? Well, the collection and analysis of this data comes to mind. There are a number of tools to help visualize this xAPI data in different ways making it easier to interpret results. Yet Analytics is one group that focuses on data visualizations with a large focus on xAPI data.
What other sources of learning and experience data exist in our current ecosystem? What insight would could help you make training and development more effective and efficient for your students? What actions would trigger you to recommend a performance support resource or intervene with a coaching session? These are the questions that continue to drive the acceptance of xAPI as a valuable tool across the learning space and help training departments evolve into centers of human performance improvement.
DUNCAN WELDER (Director of Client Services, RISC)
Mr. Welder holds a Master’s of Education from Texas A&M University in Educational Technology and has over 19 years implementation of Learning Management Systems, domestically and abroad, to manage regulatory-compliance in high-consequence . Duncan has provided presentations to professional organizations including the Gulf Coast Process Technology Alliance, the e-Learning Guild and the Association for Talent Development. Mr. Welder’s career is founded in instructional design and e-learning. He is a certified as a Development Dimensions International facilitator, a Kirkpatrick Evaluator and facilitator of the Ohio State University curriculum development program. In addition to working in industry, Mr. Welder has held adjunct faculty positions at Bowling Green State University and the College of the Mainland.