fbpx

Leveraging Webinar Analytics to Measure User Experience

Any medium to large organization that has attempted to deliver a webinar understands the challenges of delivering live video at scale. However, despite the complexities of delivering a webinar, sometimes it is even more difficult to determine whether your broadcast went well.

To help answer this question, Vbrick has released Key Performance Indicators (KPIs) for live virtual events. These KPIs will help answer the question that often lingers after a Webinar: “How did it go?”

These new metrics will be found in the revamped Real-time Analytics for Live Events. This new video intelligence tool will help monitor and proactively troubleshoot your enterprise video distribution.

When we set out to define the KPIs enterprises should be using for live virtual events, there were a few core considerations that we felt were important:


Webinar KPI Foundations

  1. Webinar User Experience: The number one goal of a webinar is to communicate with your audience and thus user experience is priority.
  2. Video Stream Performance: Delivering your message is the most critical goal of a webinar. Rebuffering and errors interrupt your message and cause frustration to your viewers.
  3. “Apples-to-Apples”: These universal metrics should allow you to compare different event types and measure progress over time.

Rebuffering is a highly disruptive issue for a viewer to suffer during a live event, and is the pinnacle of poor user experience. Rebuffering may be caused when there is not enough bandwidth for the video quality to either be delivered by the host or loaded by the device. A study from the University of Massachusetts and Akamai showed that a viewer who experiences a rebuffer delay equal to 1% of the video duration will play 5% less of the video in comparison to a viewer with no rebuffering.

Some people might ask about KPIs such as bitrate, resolution, or other measures of video quality. However, context is critical for video quality metrics, as benchmarking a 4k video stream by its delivered resolution on a 1080P mobile device will do more harm than good. The short answer is that while video quality is important, delivering seamless video to your audience matters most.

This playback-first mentality prompted us to answer a few questions:

  • Can people see the stream?
  • How much time is the audience spending rebuffering?
  • When they start rebuffering, how long until the video plays again?

These questions then helped us develop the following KPIs for live virtual events:

Webinar Key Performance Indicators (KPIs)

Metric

Equation

Insight

Experienced Rebuffering (%)

This provides a high-level look at how much time your audience spent waiting for the stream to load versus watching your event.

Average Rebuffer Duration

This indicates how long users had to wait for a stream if they did start rebuffering and see a (dreaded) spinner.

Experienced Errors per Total Users

This indicates times when the audience was unable to see the video, which generally indicates either a configuration issue or overloaded network.

These metrics help you to see an at-a-glance indication of whether people are receiving your video content with minimum frustration or distraction. These KPIs will also be helpful to resolve the source of an issue because often issues are localized with a particular browser, device, or location that can skew these metrics higher.

The most common issue we see is attendees unable to participate in a webinar because there is not a lower-bitrate version that can be delivered over mobile networks or low-bandwidth environments. Pinpointing and resolving these types of issues can dramatically improve the experience for those users.

There are countless other metrics available, such as viewing time, bitrate, and user engagement with chat that should also be considered when evaluating an event’s overall success. However, these metrics provide a core foundational framework for measure the success of your video distribution for live virtual events. With these KPIs centered around technical performance pertaining to user experience, you will ensure that the most critical  


Author:

David Jeyes

Director of Product Management