You are here: Experiments / QoE Testbed

Quality of Experience Test Platform

The aim of this work is to understand how network design choices affect the quality of experience (QoE) enjoyed by the end user. To assess this we have designed and built a test platform and conducting tests with users.

In order to establish how QoE can be correlated with network design choices and key network performance indicators (delay, jitter etc.), a test platform has been built. The experimental system has been designed end to end, so apart from a physical platform the following have also been devised: a model for factors affecting QoE, in order to give a framework within which to interpret the results; a range of measuring methodologies that can elicit results in a standardised way and test conditions that allow, for example, group video chat situations to be assessed repeatably. In addition in order to make the insights gained from experiment carried out on the test platform the design team have looked closely at what key standardisation bodies have been working towards in the field of group communications.

The QoE Test Platform - Results

The test platform uses commercial off the shelf or freeware tools where available. In some cases the project has had to develop its own components. Supporting the platform and its experiments are a model for factors affecting Quality of Experience developed by Geerts. This helps us understand how context, user role and system parameters all affect the report the QoE.

         

 

 Schematics that help understand the User Experience Model developed by Geerts et al., that Vonect is using to help assess and undertand Quality of Experience

A range of test methodologies including Questionnaires, Interviews, Task scores speech patterns and physiological measurements based on biological reactions area available, the methodology used will depend on the experiment.

The system that has been built includes the Vconect built client for multiparty conferencing. This is the GUI through which test participant will experience video chat.

 The Vconect client, used as the GUI through which participants experience video chat.

 An ObserverControl Client for the experiment conductor and a tool for analysing experiment sessions. This allows the observer to see the emerging video chat and to interact if necessary to provide instructions etc. The end to end video chain has been augmented with capabilities that allow a range of network KPIs to be affected in a regulated way and these have been tied to a timing model that allows all interactions and observations to be compared against a common timing model. The end to end chain allows vividness to be affected by selecting different codecs; for interactivity aspects to be affected by changing the framerate, by introducing delay and jitter; for consistency aspects to be affected by changing both synchronisation of the audio and video streams received from a given user, and also by allowing the inter participant synchronisation to be varied.

End to end AV chain showing how various system properties can be affected in order to assess the impact on the QoE

More objective analysis that can be used to assess for example task performance or speech patterns can be revealed using the speech patterns analysis tool which can indicate more about the voice activity verbal or otherwise from the participants.

A view of the speech pattern analysis tool

The Test Platform – exploitation

Findings based on this experimental testbed will inform the use cases in the project. Beyond the project, a much wider exploitation opportunity exists by making the findings as widely known as possible. Vconect has already published the details of its test platform and will publish widely its results so that findings about the impact on network KPIs on QoE in group communications are as widely understood as possible.

One other formal and potentially effective route to making such findings known is to work through the ITU (International Telecommunication Union). The ITU makes recommendations about certain key performance indicators for certain types of communication. For example, for a telephone call, it recommends a one way delay less than 150ms and that echo cancellation should e used for delays above 25ms. In group video calls the key performance indicators are much more difficult to establish. However there are groups amongst the ITU working towards various recommendations related to group based communication. Vconect has attended ITU meeting and is working to understand whether there is interest in that body to specify KPIs for group based video calling. A first step in making the Vconect platform central to this objective would be to agree test designs and measurement metrics. The Vconect platform is a strong “straw man” with which the ITU can explore how best to make recommendations in this area.