As tools like Opencast Matterhorn continue to reduce the barriers to entry for capturing and distributing lectures, it is important to recognize that the availability of video alone is not sufficient to improve educational outcomes. Since January 2012, Entwine has collaborated with SWITCH on the Annotating Academic Video (AAV) project with the goal of creating a standardized, open and flexible tool/framework to enable Swiss University faculty, staff and students to annotate video across a mix of platforms including players, video management and learning management systems.

This post provides a first look at the features, architecture and our progress to date with the AAV project.

Why we need annotations in video management systems?

For the past decade, higher education institutions have been adopting lecture capture technologies to improve the academic experience. Although these technologies are proficient at capturing and distributing media to learners, they perpetuate an outdated and passive learning experience by not providing tools for students to enrich, enhance and share media with classmates.

The most common method for teachers and learners to interact with video is via note taking and discussions. The majority of education based video services do not take advantage of the annotations tools that YouTube delivered to the masses a few years ago. Taking into account that today learners and teachers are becoming less and less physically connected, this void is particularly painful as it means that there is not a simple method to comment on educational videos, such as lecture recordings. Simple interaction between learners and teachers through comments is not the only pedagogical benefit of an annotations tool. Most students take private notes during a lecture. The ability to annotate a video lecture has the potential to enrich notes by adding object and timing references to them. Sharing enhanced notes will help other learners and improve relevant information about search results.  AAV can also be used for assessment. For example, instructors can evaluate the annotations produced by a student to gauge concept mastery.

Academic Annotation Video Project

Screenshot of the current version 

Goals

A technical goal is to deliver an open source tool that is extensible enough to be used in a variety of educational settings in conjunction with different technologies relevant to the academic use of video, i.e. video players, video management systems and learning management systems. The result is an application that can be used with different lecture capture and delivery systems such as Opencast Matterhorn and with different players such as the Engage player from Matterhorn.

As for the non-technical goals and benefits, the annotations tool has to make video a full-fledged learning object in that it provides learners with adequate tools to actively capture content: Note-taking, highlighting, structuring, establishing relations or extracting excerpts. Most of what the learning process encompasses today is not possible with video – yet.

Benefits

As a pedagogical tool

In the first part of this blog, the lack of student interaction with video content was clearly expressed. This tool bridges the gap by introducing a toolset that allows users to easily annotate video. Learners can finally interact with video and enrich it with their own personal content. Furthermore, this service does not rely on a specific back end system or player. Being generic, this tool with its benefits can be integrated in any system as long as the interfaces for it are implemented.

As an Opencast Matterhorn third-party tool

The upcoming release of Opencast Matterhorn, 1.4, will come with another solution to annotate video. This one is directly integrated in the engage player and therefore strongly dependent on it. What are the benefits of another annotations tool if Matterhorn already has a solution?

First, the annotations tool makes annotations a the focus. The user interface is designed so its focus is on creating and visualizing annotations. Annotations are not just an afterthought, they are the main focal point. The tool allows end users to annotate video quickly and easily and view existing annotations. Secondly, as the tool is not dependent on the engage player, it can be used with any other player, like a native HTML5 player and introduces the option of using an iPad or other mobile device.

Features

The tool can be used to view and write annotations.

View

• Display annotations in two synchronized views: a timeline and a horizontal annotations list
• Highlight annotations in current playhead range
• Filter annotations and tracks
• Jump to annotations start time by selecting an annotation in a list
• Change tool layout
• Print annotations
• Export annotations as CSV file for statistic usage

Add/Edit

• Insert unstructured text annotations.
• Add new tracks.
• Edit fully the annotations in list
• Auto-pause mode: Video stops automatically during typing and continues after annotations insertion.
• In/Out buttons: Set the start/end timepoint of an annotation with the current playhead
• Define visibility of tracks and annotations (public/private)
• Change duration and move annotation by dragging on timeline
• Comment annotation

Structured annotations

Structured annotations are based on a set of defined labels divided in different categories. The categories can be created and filled by the teacher to give a limited set of labels to annotate a video.

For example, a video could show a patient being interviewed by a doctor. The categories and labels could be:

• Feelings
  • fear
  • anxiety
  • curiosity
  • happiness
  • confidence
• Reaction
  • Positive
  • Negative
  • Neutral

Now the students would have to annotate the interview with this given set of categories.

The structured categories can also have a scale to add more information to the labels. With the example above, a scale for the reaction category could be numeral scale, with value from 1 to 5, indicating how positive or negative the reaction is.

Students can also create categories and labels that can be shared using the export/import functions.

Features related to structured annotations:

• Create and edit category, label and scale
• Annotate with label and scale
• Present an annotation with labels on the timeline and table with their color and value
• Export/import categories
• Can be used in parallel with unstructured text annotations
• Structured annotations can be enabled/disabled

Architecture

Because the tool has to work with different lecture capture systems, players and learning management systems, the tool architecture is decoupled in three parts to make it as scalable as possible.These parts are illustrated in the diagram below:

Annotation tool architecture with an HTML5 player and Matterhorn as backend system. 

The player adapter

Any kind of web video player can be used with the annotations tool as long as it has its own player adapter and implements all the methods from the player adapter API. The annotations tool will communicate with the player through the player adapter.

The annotation tool

This is the core of the tool. It retrieves the annotations from the backend and allows one to create/edit annotations and synchronise them back. It communicates with the player and gets all types of events from it. It facilitates the synchronisation between the playhead, the timeline and the annotations table. The tool communicates with the backend through Rest Endpoints to get the existing annotations and maintains the new or modified one.

The backend (lecture capture system)

The tool is made to work with any kind of lecture capture system. It requires the implementation of this API and the related Rest Endpoint. This part is required to ensure the persistence of the data but for demo the tool can also works with the HTML5 localStorage.

Possible project extensions

A future project could be the use of the open annotation standard as a data model for this annotation tool. Although it would require an important refactoring. A good solution could be the use of this standard for the import/export format. The use of a standard is always a good practice to ensure interoperability between tools. Even if this young standardis not used in relevant tools now, the situation should change in the near futur.

The integration of social media is also an option that should be considered. Teachers and learners could share a video or annotation on a social media network or add comments/images as annotation content.

Annotations content could be extended to more than text. In the future annotations could consist of images or additional video clips.

Matterhorn integration could be refined. Currently the annotations integration only grabs episode from Matterhorn but does enhance the metadata when annotations are made. A deeper integration would mean a bidirectional communication with Matterhorn. The created annotations could for example be added to the search indexes to help users find relevant content.

More information about the project

As an open-source project, all the project’s resources are available on https://github.com/entwinemedia/annotations/. A follow up post is on its way presenting the technology stack used for it developement.

Additional informations is available on https://github.com/entwinemedia/annotations/wiki.

Project Funding

Provided by AAA/SWITCH

Once the service implementation and documentation have passed the community review, the new functionality will be added to the next version of Matterhorn 1.4.x.

Sox Service

The new SoX service, “matterhorn-sox”, is dedicated to run on a worker node and therefore is accompanies by a remote service implementation to enable audio analysis and leveling in a distributed environment.

Instead of just implementing audio normalization which first requires an analysis step we decided to expand the analysis step to an independent operation as well. After analyzing, three key values are being written to the media package’s audio metadata: “RMS Lev dB”, “Peak Lev dB” and “RMS peak dB”. The values can be obtained manually by executing the following commandline:

sox audio.flac -n remix – stats

The implemented normalization operation then uses the “RMS Lev dB” value and increases or decreases it to a target decibel value as requested by the operator. The resulting normalized audio file is then analyzed again and stores the same values as the analyzing operation to the media package’s audio metadata. This is done via the following SoX command:

sox source.flac normalized.flac remix – gain -5.4 stats  // decrease
sox source.flac normalized.flac remix – gain -l 3.4 stats // increase

This is at the moment all we need for an audio leveling service. SoX offers a lot more things to do with audio files (e.g. silence detection), so in the future it’s easily possible to extend the service with this kind of functionality.

Endpoint

The service’s REST endpoint is reachable under /sox/docs with the following two methods for analyzing and normalizing an audio track:

Links

Opencast Ticket - http://opencast.jira.com/browse/MH-6386

SoX Website - http://sox.sourceforge.net/

In SSL configuration, a way is described to configure Matterhorn to listen for and answer HTTPS requests. Instead of having to set up Matterhorn with SSL certificates, there is also the option of keeping it on HTTP and put Apache (or any other HTTP server) in front of it, handling the HTTPS requests and passing them on to Matterhorn through HTTP.

Configuring Apache Httpd

There is plenty of documentation available on how to set up Apache HTTPd with support for HTTPS. Enabling Apache as a frontend to Matterhorn is achieved by installing an SSL certificate and properly configuring HTTPS using these certificates, as well as enabling request forwarding with a little twist, as detailed in the two sections below:

Enabling Support for SSL

Again, enough has been written on this topic so it should be easy enough to find documentation on how to do this on your platform. Using the search engine of your choice, look for “Apache Httpd” and “mod_ssl” to get started. It eventually comes down to installing mod_ssl, creating a certificate request, having that request be signed by a certification authority (or create a self-signed certificate) and adding the certificate, the private key used to create it as well as the signer’s root certificate into the proper locations in the Apache HTTPd configuration file.

Then, after some tweaking and testing, Apache will be listening on port 443, which is used for https:// urls. For the rest of this document, we assume that a virtual host is being used to gather the requests for Matterhorn under the url http(s)://matterhorn.institution.edu.

Read more here.

We will first examine PcP technologies and the Matterhorn equivalent. Although architected around the same concept of highly customizable automated media processing platform, Matterhorn and PcP have a very different technology stack, thus the term “equivalent” throughout the article must not be taken literally. For example, while the MH Service Registry serves a purpose similar to what Xgrid provides to a PcP deployment, one can not be strictly replaced by the other, moreover even though they both provide load distribution they do not operate in the same fashion. The technical details will covered in an upcoming technical comparisons post.

1) Podcast Producer Nomenclature

In this table we cover the main concepts and technologies of Podcast Producer and, when it exists, the equivalent in the “Matterhorn world”.

2) Matterhorn Nomenclature

Contributed Features

• User groups
  Users can be grouped and groups can have access rights.
• Access control list management
  • ACL scheduling
    Apply ACLs in a time based manner.
  • ACL management
    Predefine ACLs per organization and give them names.
  • ACLs on a per episode basis
    ACLs can now be set on the episode level in addition to the series level to provide fine grained access control.
• Ingest from CloudStor service
  Service is a cloud file store. Matterhorn polls this service and ingests any uploaded media packages.
• Integration with Shibboleth authentication and authorization service
• RPM based deployment on a distributed setup
• Archive metrics (1.4)
  Monitor free and total space via JMX.
• Graylog2 integration
  Graylog2 is a tool for centralized logging.
• Branding for Engage based on the tenant
• Browse Engage by series

Contributed Fixes

• Several fixes for distributed and multi-tenant MH setups (everything 1.4)
  • Event handler synchronization
    Several concurrency issues caused handlers to interfere with services and each other.
  • Digest authentication
  • Media inspection property setting
    Properties in class CommonMetadata could not be set.
  • Workspace lock file synchronization
  • SearchServiceRemoteImpl forwarded only a small subset of query parameters
  • User propagation between stub and remote service
  • RemoteBase fix
    Could not handle 404 from series service.
  • Tenant config
    To support multi-tenant setups some config keys have been moved from the global config.properties to the organization config.
• Miscellaneous fixes
  • Tiff encoding of ffmpeg caused tesseract issues (1.4)
  • Default workflow did not archive metadata catalogs (1.4)

Feel free to contact us if you have questions about our work with Uninett or the new Matterhorn 1.4 features.

Because of this a media package from a previous release can’t be ingested directly into 1.4 without fixing the manifest. The MpConverter remedies this situation and in addition allows for building 1.4 compliant media packages from scratch, i.e. from media files, metadata catalogs and attachments. It is a command line tool that runs without interaction when applied to packages with a manifest and provides an interactive editing shell when building a new package from content files.

Fixing DublinCore Catalogs

In addition the converter fixes some malformed metadata fields. DublinCore catalogs need to have dates encoded according to W3C-DTF in order to be accepted. Capture agents have been seen to write these fields in a different format. Currently the following fixes are supported:

Support for other fields has been prepared but is not yet implemented. If you encounter a different format let us know so we can incorporate a fix for it.

Building MediaPackages From Scratch

Assume you have some tracks and metadata in a directory structure like below but no manifest file. To create a valid media package run the converter with the root directory as argument. The converter helps with building the package by highlighting fully defined elements and provides tab completion support for providing missing information.

|- track-0/video.mp4
|- catalog-0/dublincore.xml
|- catalog-1/dublincore.xml

Where To Find

Find the converter at github https://github.com/entwinemedia/mpconverter included is the source code and a precompiled binary to get you started quickly.

Batch Converting

For batch converting simply wrap the converter in a loop.

 $ for m in $(find . -type d -maxdepth 1); do mpconv.sh $m; done

What Else?

This little project is also a nice demonstration of how to use the Matterhorn libraries from another JVM based language, here Scala. If you like to learn something new and don’t want to play with the big boys (Haskell) give Scala a try! It’s very much worth it. And remember, it’s also from Switzerland .

Hope you’ll find it useful. Happy converting!

With the management of distributed applications comes the question of log management; not only in terms of centralization of the logs themselves (which typically are distributed on the different application hosts) but also in terms of finding the relevant information amongst the vast amount of data recorded in the logs. On it’s own, the open source lecture capture system Matterhorn does not address such challenges therefore Entwine recommends the integration of an external product/services.

While proprietary services such as Splunk or Loggly are available, Entwine investigated open source solutions such as Logstash paired with Kibana or Syslog-ng and ultimately chose Graylog2. An advantage of Graylog2 is its simple integration with JAVA application using Log4j as its logging facility via readily available appenders.

To install Graylog2 on RHEL/CentOS/SL 6 the step by step by Nathan Milford has proved to be the most functional.

In order to ship Matterhorn logs to Graylog2 , Entwine leverages the Graylog Extended Log Format (GELF) which is the ideal way of sending JAVA logs as it can deal with multi-line JAVA stack traces.

Since Matterhorn uses Log4j as its logging facility, it requires the use of a GELF appender for Log4j, while some GELF Log4j appenders exist, none of those tested functioned with the particular OSGi Matterhorn architecture and the Apache Felix framework. Entwine modified the GELF appender from Anton Yakimov to turn it into an OSGi bundle that attaches as a fragment to Pax Logging, thus making Matterhorn “Graylog2 ready”. You can find the pom to build the gelf4j GELF appender that will work with Matterhorn in the Entwine Matterhorn-Gelfj github repository ,  also included in the binary itself built from Anton’s source code using the included pom.xml

Don’t forget that you still need to make the GELF appender available to Matterhorn and Log4j, for Entwine RPM based deployment it means including the gelfj binary in the package and adding the path to the GELF Log4j appender to all the profiles for felix to load the module at start level 1.

Once the GELF Log4j appender has been loaded by Felix Log4j needs to be configured to use it via $MATTERHORN_HOME/services/org.ops4j.pax.logging.properties

Here are the details of the Log4j configuration for Matterhorn:

Include “graylog2″ to log4j.rootLogger key e.g. :

log4j.rootLogger=ERROR,graylog2,stdout,file

Add the required keys for graylog2 appender as per Anton Yakimov gelfj documentation e.g. :

log4j.appender.graylog2=org.graylog2.log.GelfAppender
log4j.appender.graylog2.graylogHost=GRAYLOG2_SERVER_URL
log4j.appender.graylog2.originHost=HOSTNAME
log4j.appender.graylog2.facility=matterhorn-graylog2
log4j.appender.graylog2.layout=org.apache.log4j.PatternLayout
log4j.appender.graylog2.extractStacktrace=true
log4j.appender.graylog2.addExtendedInformation=true
log4j.appender.graylog2.additionalFields={‘environment’: ‘DEV’, ‘application’: ‘Matterhorn’}

For many institutions, deploying a lecture capture system is a difficult task to manage as it involves many shared resources that staff, faculty and students, potentially across multiple campuses, must all share access to. The task is more difficult if there is no integration with directory services listing available users, or if the users are not grouped inside that directory in ways that are meaningful to the administration or consumption of lecture capture recordings.

As a newcomer, a role-based approach may seem confusing but it’s actually quite simple once you understand how it works.  This model allows an administrator to assign access privileges according to the “role” of a user or group. For example:

This means once a user logs in, the determination as to whether or not he/she has access to execute specific tasks (e.g. create a workflow, change metadata, watch video), is granted based on the set of role(s) a user is assigned, where the set is compiled of roles that have been assigned to that user individually and of roles assigned to groups that the user is in. The group based approach to roles and permissions management is more flexible and powerful than relying on (usually sparse) information in existing institutional directories or managing roles on a per user basis.

In order to complete the picture, here are several basic concepts to help you better understand the association between user, groups, roles and permissions:

• There are users and each user optionally belongs to any number of groups.
• A group is the holder of one or multiple roles.
• A role is the method for assigning permissions to a resource such as a recording.
• A permission gives a specific right to do something with that resource within Matterhorn

• A user can have multiple roles.
• A role can be assigned to multiple users or groups.
• A role can be associated with many permissions.
• A permission can be assigned to many roles.

Our Work

There have been significant changes and additions made to the codebase in order to add the group functionality, including:

1. A new REST endpoint for the creation and management of groups.
2. The REST endpoints for organization, users and roles was completely revised and now returns full object representations instead of just “mh_default_org” for an organizations but rather id, name, adminRole, anonymousRole, config properties, etc.
3. Previously the users and roles endpoint returned content from Matterhorn’s internal users and role database only. The revised endpoints now return all the users and roles from all registered user and role providers (e.g. internal ones as well as an external LDAP database).

The work on the backend led to a complete database refactoring with several new tables, as previously, many of the objects had only been kept in memory. Finally, as seen below, there is a proposal and implementation for a new “Security” section in the Matterhorn administrative UI for creating, editing and deleting the groups as well as managing group roles and memebership:

Entwine will discuss this topic further at the upcoming Opencast conference at UC San Diego, we hope to see you there.

MH-9316 - SearchServiceRemoteImpl#getSearchUrl does not use all query parameters.
MH-9315 - getForAdministrative methods do not check for the organization admin role.
MH-9290 - Services Documentation links aren’t linked to the right tenant in a multi tenant distributed environment
MH-9259 - Scheduling UI: if missing or invalid data when user submits, can’t see error message unless scroll to top
MH-9253 - The mediapackage metadata editor does not listen to window resize events

and to UNINETT for funding these patches:

MH-9274 - Encode, Analyze, and Distribute” configuration panel script does not work if loaded inside a dialog.
MH-9250 - EpisodesPermissionsUpdateEventHandler creates too much new version during updating

Please contact us if you are interested in putting Entwine to work on Opencast Matterhorn patches in December. Our goal is to help the community release 1.4 by the end of December and any support helps. We have exciting ideas and features we are working on for 1.5 which we will be discussing in a future post.

MH-9213 - Metadata used in integration tests lacks title
MH-9214 - MimeType parser does not honor multiple extensions defined in mimetypes.xml config
MH-9216 - StreamHelper sometimes throws NPE due to use of wrong logger object

Please contact us if you are interested in putting Entwine to work on Opencast Matterhorn patches in November. Our goal is to help the community release 1.4 by the end of December and any support helps. We have exciting ideas and features we are working on for 1.5 which we will be discussing in a future post.