Quantitative comparison of application–network interaction: a case study of adaptive video streaming

Abstract

Managing quality of experience (QoE) is now widely accepted as a critical objective for multimedia applications and the supporting communication systems. In general, QoE management encompasses: (1) monitoring of the key influence factors and QoE indicators, and (2) deciding on the appropriate control actions as specified by the management goal. Many multimedia applications, e.g., video streaming and audio conferencing, are able to adjust their operational parameters so as to react to variations in the network performance. However, such an adaptation feature is mostly based on a local client view of the network conditions, which may lead to an unfair allocation of network resources among heterogeneous clients and, thus, an unfair QoE distribution. In order to tackle this issue, there is the call for a cooperation between the applications and the underlying network, which includes application–network interaction (App-Net) in terms of: (1) exchanging information on the monitored QoE indicators, and (2) coordinating the QoE control actions. Various App-Net mechanisms focusing on specific use cases and applications have been proposed to date. This paper gives an overview of App-Net mechanisms and proposes a generic App-Net model that provides the means to realize a coordinated QoE-centric management. Based on the App-Net model, we develop an evaluation methodology to compare three App-Net mechanisms for managing QoE of HTTP adaptive streaming (HAS) against a baseline HAS service. The aim of this quantitative comparison is to explore the trade-offs between QoE gains and the complexity of App-Net implementation, with respect to the number of monitoring and control messages, achieved video quality, and QoE fairness among heterogeneous clients. Our ultimate goal is to set up reproducible experiments that facilitate a holistic evaluation of different App-Net mechanisms.

Introduction and motivation

The concept of quality of experience (QoE) models the notion of service quality as subjectively perceived by end-users [1, 2] and takes into account the effect of different influence factors (IFs) that are introduced by elements of the service delivery chain, such as end-user equipment, application servers, and network infrastructure, but also that of the end-user and context-related factors [2]. Managing QoE is now widely accepted as a critical objective for multimedia applications (e.g., for video streaming or audio conferencing) and the supporting communication systems. This objective has been boosted, in particular, by the emergence of popular content and service providers like Google, Netflix, Spotify, and Amazon. However, the Internet has not been designed for rigorous resource demands imposed by the multimedia services, while network mechanisms for a differential treatment of multimedia traffic have not been implemented on a wide scale.

As over-provisioning network resources is often economically unfavorable or technologically limited, e.g., for wireless networks, general solutions to cope with the current communication network limitations propose to adjust configuration and operational parameters of the multimedia applications so as to react to variations in the network performance. Many multimedia applications are able to dynamically adjust their behavior, with mechanisms ranging from fundamental TCP and UDP protocols to novel technologies such as HTTP adaptive streaming (HAS) [3]. However, such an adaptation feature is mostly based on a local client/server view of the network conditions, which may lead to an unfair distribution of the available resources among contending traffic flows and, hence, negatively impact QoE. Even if a fair resource share is achieved, heterogeneous clients may still receive different QoE levels [4]. For instance, a smartphone device and a laptop computer may be assigned an equal network bandwidth for video streaming, although a continuous video reproduction at the smartphone would demand less bandwidth.

The general QoE management cycle includes two phases: (1) monitoring of the key QoE influence factors (QoE-IFs), and (2) deciding on the appropriate control actions that adhere to the management objective. Since there are various QoE-IFs, monitoring strategies include design choices on, e.g., which quality indicators to measure, what are the measurement points, and how often are the measurements carried out. Measurements can be taken on different levels of the protocol stack, resulting in different granularity and types of the retrieved information (e.g., per-packet inspection or flow monitoring). However, we omit such details in this work and consider network assessment of the associated QoE-IFs, which will be referred to as network monitoring, and user-device or application assessment of the corresponding QoE-IFs, i.e. application monitoring (c.f. Fig. 1). Control actions, on the other hand, can be employed so as to adapt network behavior, application behavior, or both. For the network level, mechanisms such as bandwidth allocation (e.g., [4, 5]), routing reconfiguration (e.g., [6, 7]), and traffic prioritization (e.g., [8]) can be applied, which we refer to as network control. At the application level, for example, a client can request from a server another media content resolution and bitrate (e.g., for video) in response to bandwidth fluctuations, or adapt its playout delay (e.g., for audio) so as to alleviate consequences of network jitter, which will collectively be referred to as application control.

Fig. 1

Illustration of the QoE management aspects that consist of monitoring and control of application and network

The QoE management objective can be defined in a different manner as improving or optimizing QoE, e.g., to provide a fair assignment of QoE levels among end-users. In order to achieve such an optimization objective, an explicit exchange of information on the monitored QoE-IFs and of control instructions between the applications and the supporting network is required. The latter resembles typical cross-layer design approaches, which focus on the information exchange. So as to extend this notion in a way to include the solutions that aim at establishing a cooperative control interplay between applications and the network, we introduce the term Application–Network Interaction, or App-Net for short. With this in mind, several App-Net solutions also propose a coordinated approach for the QoE management (e.g., [5, 8]), which assumes a logically centralized entity collecting the application and network monitoring information, and then deciding on control actions between these two levels. In this work, we refer to such an entity as the policy manager (PM).

As the App-Net solutions propose different design choices, algorithms, and actions for QoE management, they influence QoE-IFs in different ways and to different extents. This fact calls for a comparison of the strategies to estimate their performance. However, the current works that present App-Net strategies discuss the benefits and impacts on QoE-IFs for diverse use cases and different quality metrics. Furthermore, different testing environments which are applied and the dissimilarity of the conducted experiments impair the comparability. In order to reveal and compare the performance of different App-Net approaches, a benchmarking is necessary to gain insights on how these approaches impact QoE in different scenarios. To the best of our knowledge, such holistic evaluations do not exist yet. In this work, we aim at comparing different App-Net approaches and estimating their complexity. While the goal is not to conduct a QoE study, but, instead, we discuss the impact of the interaction approaches on various IFs for QoE. Our contribution is, hence, three-fold.

• 1. We facilitate reproducible experiments by setting up a controlled environment and implementing three App-Net approaches based on the ideas from state-of-the-art work. To achieve reproducibility and comparability, the complete source code of this work is made publicly available.¹

• 2. We use this infrastructure to evaluate and compare the chosen App-Net mechanisms in different scenarios. With respect to the considered QoE metrics and scenarios, we go beyond evaluations in the original works in order to reveal pros and cons of the App-Net mechanisms with respect to several QoE-IFs.

• 3. We investigate the number of exchanged interaction messages as a first approximation of the complexity for a specific App-Net strategy.

Building on our previous work [9], in which we identified a generic App-Net model and provided an initial evaluation of two App-Net mechanisms for enhancing HAS QoE, this paper:

• 1. extends the overview of existing App-Net solutions to include additional state-of-the-art results and an insight into the QoE-IFs considered by each of them,

• 2. expands the App-Net model so as to provide a more in-depth discussion on possible interaction strategies, and

• 3. provides a quantitative comparison of three App-Net mechanisms for improving HAS QoE against a baseline HAS service.

The rest of this paper is structured as follows. “Generic App-Net model” gives an overview of App-Net solutions that focus on improving QoE and describes the generic App-Net model. Based on this model, “Evaluation methodology” explains the App-Net comparison methodology and the evaluation testbed that is used to gather experimental results. The quantitative comparison results are presented and analyzed in “Evaluation results”, followed by the conclusions and a future work overview.

Generic App-Net model

In the current highly competitive market, end-users can decide between a large number of service providers. It is, hence, an imperative for those providers to deliver their services in a way that offers end-users a rich experience, which is especially critical for real-time multimedia services. To reach a “good” end-user QoE, different IFs that are related to end-users, systems, applications/services, and context need to be taken into account [2]. With respect to end-users, the factors may encompass demographic information, user preferences and expectations. System-wise factors include typical features and performance of the supporting communication systems, such as network throughput, delay, jitter and packet loss, as well as server and user device characteristics in terms of, e.g., processing power, memory, screen size, and presentation features. Parameters referring to applications/services, on the other hand, comprise supported service features, available protocol support, media encoding, resolution and sample rate, but also content-specific information (e.g., content type). Factors describing the situation in which a service is being delivered relate to, e.g., user activity and mobility, time of usage, user location, and service cost.

Recent years witness a plethora of research results that focus on issues in managing QoE for real-time multimedia services [10, 11], such as video streaming and audio conferencing. The associated mechanisms differ in QoE-IFs that they consider, as well as level of support for QoE monitoring and control. Since a main prerequisite for improving or optimizing QoE is the level of support for application/network monitoring and control, our survey of the existing App-Net mechanisms is extended beyond our previous work [9] and uses the following classification:

• 1. Monitoring support:

  • (a) Application monitoring (AMN)—QoE-IFs related to applications/services and servers/user devices are assessed at the application level.

  • (b) Network monitoring (NMN)—QoE-IFs corresponding to the underlying communication network are evaluated at the network level.

• 2. Control support:

  • (a) Application control (ACT)—an application-level entity is informed on application-level or network-level QoE-IFs, and holds QoE control.

  • (b) Network control (NCT)—a network entity is reported on application-level and/or network-level QoE indicators, and runs the QoE control process.

  • (3) PM-coordinated control (PCT)—application-level and network-level QoE-IFs are conveyed to a PM, which coordinates control operations among applications and the network.

App-Net mechanisms for improving QoE

One of the most dominantly used multimedia services over the Internet is video streaming, which is commonly built upon the HAS technology and its widely accepted standard, Dynamic Adaptive Streaming over HTTP (DASH) [12]. DASH partitions video and audio content into short segments, of a few seconds in length, and stores them on a server. Each of the segments is produced in various representations that provide, e.g., different encoding bitrate and resolution so as to facilitate adaptation of video quality to varying network bandwidth. These segment representations are accompanied by a media presentation description (MPD) file, which contains all the meta-information on media content. General DASH operation consists of a client fetching the MPD file from a server that hosts the requested media content, and then choosing which segment representations to retrieve and play out. When deciding on a segment representation to download, a DASH client takes state of its buffer and estimated network throughput into account.

Adzic et al. [13] address DASH streaming in a mobile, bandwidth-limited environment. They present an ACT-based mechanism that prevents a client from switching to a higher bitrate segment and, thus, consuming additional bandwidth, if significant QoE gains cannot be achieved. While network bandwidth is assessed at the application level, common application-related IFs for DASH, such as encoding type and bitrate, video resolution and frame rate, are collected via AMN, but also additional, content-related factors regarding spatial and temporal characteristics. In [14], a QoE-driven management framework for Voice over IP (VoIP) services in mobile broadband networks is described. This framework builds upon application-level inspection of network conditions that are measured via packet delays, and then having communication end-points adapt audio encoding or packetization parameters when the performance is degraded.

QoE-aware DASH [15] is an ACT approach that adds a transparent proxy on the path between a DASH client and server to perform estimation of the available network bandwidth. This way, DASH clients can obtain and use more precise insight into network conditions when choosing which segment representations to download, in addition to typical application-related IFs such as encoding type and bitrate. A proposal for improving QoE for Skype VoIP services is presented in [16]. It discusses the strategies of adjusting forward error correction (FEC) coding scheme and injecting redundant voice traffic with respect to different voice encoders, so as to conform to network packet loss and packet loss burstiness.

Bouten et al. [17] address the issue of QoE optimization for multiple HAS clients and introduce intermediate proxies that detect the maximum segment quality levels the clients are permitted to retrieve, subject to the measured outbound proxy bandwidth. For this, a proxy runs a rate adaptation algorithm when a HAS client needs to establish or terminate a connection with the server and then the proxy directs each client which segment representation to download. An ACT-centric cross-layer framework for real-time over-the-top (OTT) multimedia applications, e.g., Skype conferencing, is described in [18]. The framework utilizes a simple NMN to detect congestion in mobile broadband networks, which represents a system-wise QoE-IF. This congestion information is then delivered to a Skype application, which decides on whether to invoke encoding-rate adaptation or network-level intra-flow packet prioritization.

Houdaille and Gouache [19] propose a bandwidth manager located in home gateway that allocates a fair share of network resources to each HAS session. The NCT manager monitors the available network bandwidth and then performs traffic shaping for HAS clients, which reduces the number of media content adaptations at the application level. NCT-centric mechanisms that specifically target YouTube are presented in [6, 20]. In [6], a software-defined networking (SDN) controller is notified on the current buffer level and media flow bandwidth demand, which are then used to choose less congested links for media flows.

Nam et al. adopt the SDN concept to increase QoE for HAS services [7]. The central part of their PCT-based solution is an SDN application, which calculates a new route when network congestion takes place. Monitoring is performed periodically by clients, which report application-level metrics such as buffer status, and by an SDN controller, which reports on network performance (e.g., jitter). A similar SDN-centric proposal that includes NCT with QoE-aware network paths is presented in [21], addressing different service categories (e.g., audio conversation and video streaming).

The QoE Fairness Framework [4] is a PCT approach for a fair QoE distribution among heterogeneous DASH clients. It realizes App-Net by monitoring network bandwidth, collecting user device features and MPDs, and reporting them to a PM. The PM then calculates appropriate media bitrate for each client and directs the clients which media representation to choose. Another PCT proposal for DASH services, which reduces video freezes (or stallings), is presented in [8]. There, network devices are regularly polled for traffic statistics, while clients report on application metrics such as buffer state. If a client buffer runs empty, a PM prioritizes specific media segments in the network and instructs the associated client to request the matching media quality.

NOVA, short for Network Optimization for Video Adaptation, is developed by Joseph et al. [22]. It implements a cross-layer joint optimization of resource allocation and quality adaptation. NMN entities regularly send network state updates to a base station, while AMN notifications from clients are only sent to the base station if video experiences a re-buffering event. A NOVA algorithm then computes bandwidth slices for each DASH client so that several QoE-IFs are optimized, the respective network controller performs the bandwidth slice allocation, while rate adaption is performed by the clients independently. Cofano et al. [5] describe several NCT strategies that target QoE fairness among different clients. The bandwidth reservation strategy uses application-level information so as to assign media flows to dedicated queues on network interfaces, while a hybrid strategy combines bandwidth reservation with media bitrate calculation (similarly to [4]).

Server and Network Assisted DASH (SAND) [23] is an ongoing standardization effort to specify a DASH messaging framework, with the goal to enhance multi-client QoE. The framework introduces a DASH-aware network element (DANE), which is capable of identifying and parsing MPD files, but also of modifying video segments. The SAND architecture proposes the following communication interfaces: client-to-DANE, DANE-to-DANE, and DANE-to-client. This allows the exchange of different message types and, thus, the interaction between DASH clients, media content servers, and the network devices. Exchanged information may include the intent of a client to request a video from a specific server, client capabilities with respect to connectivity, or information on the currently cached video segments at a particular server. SAND does not specify any adaptation logic, but provides a basis for the App-Net mechanisms by introducing DANEs and standardized communication interfaces.

More generic App-Net approaches are investigated in [24,25,26]. Ferguson et al. propose the participatory networking paradigm [24], which enables end-users, user equipment and applications to interact with the network. For instance, end-users can trigger application reconfiguration based on the network state or postpone initiation of, e.g., a conference call until sufficient network resources are available. OpenADN, or Open Application Delivery Networking [25, 26], introduces an abstraction layer between applications and the network, so as to enable the mapping of policies across various multimedia services to network behavior. OpenADN is based on the SDN principles, enabling, e.g., a network operator to customize message routing by chaining different middleboxes in the path.

Building the generic App-Net model

The approaches presented previously employ several distinct functions to realize App-Net, but all exhibit a common control loop. First, relevant monitoring information on applications, network, or both is collected. Then, control actions are computed and enforced, at network-level, application-level, or both. Neglecting implementation specific details of these App-Net functions, we group them into the following categories: application or network monitoring, and application or network control. The latter categories are used to identify fundamental building blocks and derive an abstract App-Net model (Fig. 2).

Fig. 2

Key building blocks for realizing application–network interplay

The model consists of four functional blocks, namely Application Monitoring (AMN), Application Control (ACT), Network Monitoring (NMN), and Network Control (NCT). AMN supervises the state of running applications, while NMN captures performance parameters that indicate the network state. Control functions of the model are shared between ACT and NCT—the former features adaptation actions implemented in applications, while the latter does the same for the network level. In order to jointly optimize behavior of applications and the network, and hence be able to achieve a fair QoE distribution, monitoring information about applications and the network may be provided to a logically centralized PM. Utilizing this information, the PM can decide on adaptation actions and instruct ACT/NCT to enforce them, thus realizing a PCT (joint control) approach.

The actual realization of the control loop varies between the App-Net approaches, which differ in monitoring accuracy, the degree of control support, and the implementation domain of the specific monitoring/control function. Such an App-Net cooperation may encompass decisions on whether assessment of key performance indicators and the adjustment of parameters are performed in the network, the applications, or both. Additionally, the App-Net mechanisms differ with regard to the frequency of control actions and technical implementation specifics. Thus, we classify the presented mechanisms with respect to the identified App-Net functional blocks (Table 1), also comparing monitoring accuracy and the frequency of control actions. Furthermore, we provide an insight into QoE-IFs considered by the App-Net mechanisms. Initial (I) refers to monitoring/control being performed only during service establishment, triggered (T) assumes that monitoring or control is initiated by an irregular event, while periodic (P) regards the actions in regular time intervals.

Table 1 Classification of the presented App-Net solutions with respect to frequency of NMN, NCT, AMN, and ACT actions

One important aspect that needs to be considered in the App-Net design regards network neutrality. Network neutrality (NN), which can be viewed from a technological, an economical or a legal standpoint, is still heavily debated and may be defined in different ways. It is most commonly referred to as the operation of a communication network that treats all of its traffic equally, independently of traffic source and destination, service/application type or content, etc. [27]. The surveyed papers do not explicitly elaborate on NN, but the approaches described in them violate to some extent the main NN principle when aiming to optimize or improve quality for a particular multimedia service (sometimes at the expense of other services). A thorough analysis of how the presented App-Net approaches conform to the NN design standards is not included in this paper. However, please note that PCT-based mechanisms which target QoE fairness come closest to fulfilling positive discrimination among end-users in terms of quality. In that sense, it is also interesting to note that some user studies (e.g. by Yiakoumis et al. [28]) have shown that end-users actually want specific services to have a differential treatment in the network.

Evaluation methodology

In this section, we first present a messaging service that is used to realize the information exchange as outlined in the abstract App-Net model. Afterwards, we highlight three PCT-based App-Net mechanisms, which are evaluated in “Evaluation results”. Finally, the evaluation testbed and the chosen performance metrics are described. The source code of the messaging service and the implemented App-Net mechanisms will be made publicly available upon acceptance of the manuscript.

Fig. 3

ØMQ-based messaging service for enabling the information exchange for different App-Net strategies

Messaging service for the App-Net realization

The interaction between application and network entities requires the exchange of monitoring and control information. In order to enable this exchange and to investigate different approaches on a common architecture, we develop a light-weight publish-subscribe messaging system that is based on ØMQ.² The central point of this system is a message broker (ØMQ-Broker), which receives all messages and forwards them to the targeted entities. Figure 3 illustrates the associated information flow. Application and network monitoring data is first delivered to a PM, which then forwards its control decisions to the corresponding application and network control entities via the ØMQ-Broker. Possible scalability issues of such an architecture could be solved by more advanced messaging architectures, if necessary.

Assessment of video quality and video quality fairness

We use the Structural Similarity Metric (SSIM) [29] to assess and compare the quality of transmitted videos. It is a full-reference metric, which means that all uncompressed and distortion-free video frames are used as a reference against the transmitted compressed frames. As luminance, contrast, and structure are considered for computing similarity, this metric reflects well the human perception and is a good approximation for video quality [30, 31]. The SSIM value ranges between 0 and 1, whereby 1 means equality to the uncompressed original content. Hence, we refer to video quality as the displayed quality, not considering other factors like video stalling or initial delay.

We furthermore investigate the fairness in terms of video quality, as this is especially interesting when end-user clients have heterogeneous capabilities, e.g., if they have different device resolutions. Larger devices have higher demands on network resources than those with a small resolution. Due to a TCP fair share, this may lead to a degraded video quality on larger-resolution (e.g., HD) devices. By fairness we assume that each client—independent of its device resolution—perceives a similar displayed quality, measured with SSIM. To investigate the video quality fairness, we use the QoE-fairness index for shared systems introduced by Hossfeld et al. [32] and adjust it for the usage with SSIM, resulting in the following formula:

$$\begin{aligned} F_{SSIM} = 1-\frac{2\times \sigma _{SSIM}}{SSIM_{max}-SSIM_{min}}, \end{aligned}$$

whereby $\sigma _{SSIM}$ denotes the standard deviation of the SSIM values observed for all end-user clients during the measurement runs. In our set up, $SSIM_{max}$ is of value 0.9907 and $SSIM_{min}$ is of value 0.8876, as these are the SSIM values of the highest and lowest video quality provided at the server. As we rely on SSIM to compute the video quality fairness, we will refer to it as SSIM-Fairness or $F_{SSIM}$ in the following sections.

Investigated App-Net mechanisms

For our evaluation we choose three PCT-based App-Net solutions that enhance QoIs or video quality fairness for HAS-based video streaming.

Quality of experience fairness framework

The first approach is called QoE Fairness Framework (QoE-FF) [4], which employs a centralized controller (PM) to govern the video quality selection among heterogeneous HAS clients and to produce a fair QoE distribution. This mechanism is illustrated in Fig. 4.

When starting a new video stream, a HAS client communicates its screen resolution, the requested video bitrate and the associated bandwidth requirements to the PM (1). While monitoring network load and active HAS clients, the PM evaluates QoE for each client using a mapping between the client-specific parameters, the network load, and the SSIM metric (2), computes the fair-share video quality levels, and enforces them at each client. A new video stream or a video stream termination triggers the PM to recompute the bitrate and, therewith, the QoE in terms of video quality for each client (3). The corresponding quality levels are enforced at each cl ient (4). As [4] does not provide details on how the network is monitored, we implement an approach based on network throughput estimation at the clients. Each client regularly estimates its available bandwidth by using time duration of a segment download and the corresponding segment size. This estimation is collected by the PM, which, besides the client-based bandwidth estimations, also considers download and idle times to derive an overall bandwidth estimation.

Fig. 4

Illustration of the functionality of the Quality of Experience Fairness Framework (QoE-FF)

Network assisted DASH fairness enhancement

The second approach, Network Assisted DASH Fairness Enhancement (NADE), is proposed by Cofano et al. [5]. Similarly to QoE-FF, this strategy targets the video quality fairness among heterogeneous clients in a shared system. For that, the mechanism allocates a guaranteed bandwidth slice to each of the active clients. The steps of this App-Net mechanism are detailed in Fig. 5.

Fig. 5

Illustration of the functionality of NADE (Network Assisted DASH Fairness Enhancement)

The network monitoring at the router queries the current throughput in fixed intervals and forwards this information to the PM (1). When initiating a new video stream, the client signals its screen resolution to the PM (2), which triggers a re-computation of the bandwidth slices. The PM is able to compute a fair video quality distribution among heterogeneous clients by using a mapping between each resolution/bitrate tuple and the corresponding SSIM value (3). By using this mapping, the knowledge about current throughput, and the number of active clients (including their resolutions), the PM decides on the bandwidth slice for each client or client type (4). The corresponding network configuration is then enforced (5).

Stalling prevention mechanism

The third approach is based on the work by Petrangeli et al. [8]. This App-Net mechanism avoids buffer under-runs in HAS clients by prioritizing download of video segments over other network traffic. We refer to the latter approach as Stalling Prevention Mechanism (SPM). Its functionality is illustrated in Fig. 6.

Fig. 6

Illustration of the functionality of SPM (Stalling Prevention Mechanism)

Network monitoring is realized by having a centralized controller (PM) regularly poll throughput of pre-configured queues in an SDN-enabled switch (1). To enable application monitoring, HAS clients are customized to add the video buffer status to HTTP requests, which are forwarded by the switch to the PM (2). The PM is then able to extract the application-level parameters and calculate whether a video segment will arrive before the corresponding client buffer runs out or not (3). In the case of a late arrival, the given segment is prioritized, while the associated HAS client is forced to request the lowest video quality so as to allow a fast refill of the buffer (4).

The outlined App-Net mechanisms differ with respect to the implemented monitoring and control blocks. While all of them use application monitoring, the SPM mechanism is the only one to monitor the application factors on a regular basis. In addition, all three approaches rely on periodic network monitoring. While the other approaches either implement network control (NADE) or application control (QoE-FF), SPM implements functions for both application and network control. We evaluate the described App-Net approaches based on the metrics detailed in the following subsection.

Evaluation metrics

QoE-IFs for HAS-based video streaming are initial buffering delay, video stalling, video playback quality, and the number of quality level switches [3]. Stallings during video playback have a high impact on user’s QoE, while initial buffering delay has a minor impact [33]. In the absence of stallings, video reproduction quality on the highest level dominates the user experience [31]. In that case, the number of quality level switches can be neglected. Accordingly, we rely on the overall stalling duration and the average video quality to compare the gains between the considered App-Net mechanisms. The investigated App-Net mechanisms differ in their complexity, which encompasses the number of involved functional entities, extent of their software implementation, and the number of exchanged messages. As a first step, we evaluate this complexity by considering the number of exchanged interaction messages sent via the brokering service.

Implemented testbed and evaluation scenario

Our testbed consists of four common personal computers. First one acts as a HAS server, second one as a network emulator, third one is running a PM and the ØMQ-Broker, while fourth one hosts six instances of the TAPAS HAS client [34]. The TAPAS clients use HAS heuristic Conventional [35] (which is a HAS baseline implementation). The HAS server stores the video Big Buck Bunny³ in three different resolutions (1080p, 720p, 360p). To obtain different video quality levels for each resolution, we encode the video clip using several bitrates. All bitrates used for video encoding and the resulting video qualities measured with SSIM are shown in Fig. 7. For 360p videos, the encoding bitrate ranges from 100 to 800 kbit/s in steps of 100 kbit/s. Hence, eight quality levels are available for this resolution. For 720p, we provide 16 different qualities by encoding the video with bitrates between 200 and 1700 kbit/s, again in steps of 100 kbit/s. Similarly, the bitrates used for 1080p videos range between 300 and 3000 kbit/s.

We divide the video clip into segments of 4 s long, resulting in 143 segments and a total video length of about 9 min and 30 s. As an example, the following ffmpeg command encodes a 360p video with a specific bitrate $br, splits it into MPEG-2 TS segments of 4 s long, and creates the corresponding play-list in m3u8 format:

Fig. 7

Bitrates used for video encoding (marked as dots) and resulting SSIM values for different resolutions. This also represents the video qualities available at the content server

The evaluation scenario involves six TAPAS clients running in parallel. We assume the clients to have different screen resolutions, 1080p, 720p and 360p, and consider two clients for each resolution. To emulate the different device capabilities, each client only requests the video play-list of the corresponding resolution, e.g., 1080p devices only request the 1080p video play-list and, consequently, only download 1080p segments. This experimental assumption eases the computation and comparison of clients’ SSIM values, as we only have to compare the compressed videos with the original video of the same resolution. Although the clients can not switch between different resolutions, they can still adapt the video quality, since several video bitrates are available for each resolution. We have to note that there is a specific range where video quality switches, either between different bitrates or different resolutions, are acceptable for an end-user. For bitrates below a certain threshold, the end-user will be annoyed, since the video quality or size is too small. Hence, the chosen video quality subset is sufficient to investigate different behavior of the outlined App-Net approaches.

One evaluation run lasts for 7 min, whereas the clients issue a video request randomly within the first 60 s of the run. Four different bandwidth limits are considered. The 1440 kbit/s limit corresponds to a link load of 90% if each client is streaming on the lowest quality available for its resolution. This limit is used to examine the App-Net strategies in a bottleneck scenario. For other experiments, we consider bandwidth limits of 2400, 4800, and 7000 bit/s. These bandwidth scenarios provide sufficient resources for a smooth streaming on moderate and good quality levels.

For each of the App-Net mechanisms, we investigate several configurations. Please note that the settings of one strategy only differ with regards to mechanism-specific, adjustable parameters, and that default configurations of the TAPAS player (e.g., initial buffer and buffer threshold) are not changed. For SPM, we investigate 12 configurations, which combine different values of network monitoring frequency (time intervals of half a second, 2, 4, and 8 s) and a safety margin for the computed segment arrival time (set to 1, 5, and 10% of the calculated value). For QoE-FF, we consider 100, 95, and 90% of the network bandwidth as available for the quality distribution algorithm. For NADE, we investigate two network monitoring frequencies (every second and every 10 s). Tables 2, 3 and 4 summarize the considered configurations for each App-Net mechanism. With respect to the QoE-IFs outlined above, there are only small differences between the settings of one App-Net strategy. For each App-Net mechanism, we, thus, choose one configuration as a representative. The representative configurations are marked bold in Tables 2, 3 and 4. For the sake of clarity, we will only illustrate the subset of representative configurations in the following subsections. The results for all settings and all scenarios can be found in Table 5.

Table 2 Configurations for SPM

Table 3 Configurations for QoE-FF

Table 4 Configurations for NADE

Table 5 Obtained results for the baseline implementation and all App-Net mechanism configurations. F denotes Fairness (in terms of SSIM and stallings). All results, except fairness indices, are mean values along with the 95% confidence interval

For each of the four bandwidth configurations, we perform 15 measurement runs to evaluate the baseline HAS implementation and all configurations of the three App-Net approaches. This results in the total number of 1080 runs, taking about 126 h and providing 6480 client log files.

Evaluation results

In this section, we discuss evaluation results in terms of key QoE indicators and the number of messages required for the investigated App-Net approaches. For the following evaluations, we focus on the phase between the second and the seventh minute of the experiment, i.e. the impact of DASH clients joining the system is neglected. The only exception we make is when examining the initial buffering delay. In this case, we consider the whole streaming session.

Stalling number and buffer filling behavior

In the experiment with the 1440 kbit/s bandwidth limit, the baseline and the considered App-Net implementations strongly differ in terms of video stallings. For higher bandwidth configurations, the stalling occurrence is negligible for all the approaches.

Fig. 8

Mean number of stallings per client at the bandwidth limit of 1440 kbit/s

Fig. 9

Illustration of the buffered video playtime with respect to different device types and different bandwidth limits

Figure 8 shows the overall number of video freezes for each App-Net mechanism, as well as for the baseline (BL). On average, clients suffer 8.4 stallings when using the baseline HAS mechanism. In case of QoE-FF, the mean number of stallings is increased to 37.7. The other two approaches, SPM and NADE, are able to reduce the number of stallings significantly.

A deeper investigation of the stalling behavior reveals that in the cases of BL and QoE-FF, the high resolution devices suffer the majority of video interruptions. This is particularly obvious for QoE-FF. When applying this mechanism, 37.7 stallings occur on average, whereby the 1080p devices contribute with 60.4, and the 720p devices with 52.7 stallings. The 360p devices can almost stream smoothly without any interruptions.

The stallings occur due to an insufficient buffer filling. The video buffer, or buffered playtime, is the portion of video data already available at the client, but not yet played back. Re-buffering is necessary if the buffer drains out, causing a video interruption that lasts until the next video segment arrives at the client. We depict in Fig. 9 the buffered playtime per device type to get a better understanding of the unfair stalling distribution when QoE-FF is used.

The y-axis represents the buffered playtime in seconds, the x-axis shows the strategies. The black dashed lined indicates the buffer threshold of 15 s configured at the DASH controller. As long as a client has a buffer below this threshold, it continues requesting segments. Otherwise, the client enters the idle state. Figure 9a shows the outcomes for the 1440 kbit/s bandwidth limit. In case of baseline, the 360p and 720p devices have a fairly high average buffer of 13.9 and 13.47 s. The 1080p devices, however, have a mean buffer of only 3.74 s. If QoE-FF is applied, the mean buffered playtime of the smallest device type nearly reaches the maximum buffer (14.5 s). Whereas, only 2.22 s are buffered at the 720p device and 1.54 s at the 1080p device. This is in line with the high number of stallings for these device types. SPM is capable to balance the buffered playtime among the different devices Despite its high resolution, a 1080p device has an average buffer of 6.63 s. The impact of the bandwidth slice reservation performed under NADE is clearly visible: All client-types have a similar average buffer. Contrary to all aforementioned strategies, the 1080p devices have the highest buffer filling level (13.63 s). On average, the 360p devices buffer 12.4 s of video content, the 720p device have a similar buffer length of 12.9 s.

The unbalanced buffer state in case of QoE-FF results in a high quantity of video interruptions for larger devices, particularly in low bandwidth scenarios. To explain this behavior, we distinguish two different phases for this mechanism. During the first phase, the clients constantly request video segments to quickly fill up the buffer to a predefined threshold. During the second phase, the clients request segments in a on-off manner, to keep the buffer on a level which allows both, a smooth playback and a fast reaction on changing network conditions. During the first phase, where video segments are constantly downloaded for a fast buffer filling, the diverse device types show different behaviors. The devices with a small resolution are instructed by the PM to request lower qualities in order to achieve a fair level of video quality among all clients. As the network resources are shared in a fair manner across all clients, these devices can faster build up their video buffer. The higher resolution devices, however, are directed by the PM to request higher bitrates to achieve fairness among the clients. As a consequence, these devices are not able to fill their buffer up to the threshold, continue to constantly request video segments, and do not enter idle states. Hence, the resources, which are shared in a QoS fair manner, do not suffice for the clients with larger devices to fill their buffer sufficiently for a smooth video playback. In the second phase, all clients have filled their buffers up to the threshold. Hence, they request different bitrates in an on-off manner, achieving a fair share of network resources with respect to video quality. As the second phase is not reached in this bottleneck scenario during our examined period, the devices with the higher resolutions suffer a large number of stallings. In a conventional scenario, the DASH heuristic would prevent this issue by switching to a lower quality for a faster transmission of video segments. However, the client-side DASH heuristic is displaced by the centralized PM application control, which does not consider a client’s video buffer. Monitoring the video buffers can enhance the mechanism’s performance w.r.t. stallings in bottleneck scenarios. Then, the PM can detect critical buffer states and decide about a lower quality for the affected clients, to support a smooth playback on the expense of a lower video quality fairness.

To summarize, NADE and SPM enable a smooth video playback even in low bandwidth scenarios, while the baseline approach and QoE-FF do not facilitate a smooth playback without stallings.

In scenarios with higher bandwidth limits, all mechanisms and the baseline allow all clients to sufficiently fill the video buffer. The outcomes of the experiments with a limit of 2400 kbit/s are shown in Fig. 9b. Please note that the results for the remaining bandwidth limits (4800, 7000 kbit/s) are hardly different and for that reason are not shown in the paper. Furthermore, the stallings are negligible for bandwidths of at least 2400 kbit/s and not investigated in detail here.

Mean SSIM and SSIM fairness

The average SSIM of all clients and the fairness in terms of SSIM are illustrated in Figure 10.

Fig. 10

Video quality fairness versus mean SSIM for different bandwidth limits

The x-axis represents the mean SSIM, the y-axis the shows the SSIM fairness.

For a bandwidth limit of 1440 kbit/s, each App-Net strategy outperforms the baseline’s SSIM fairness of $F_{SSIM}=0.62$. NADE and QoE-FF yield similar values of about $F_{SSIM}=0.88$. When SPM is applied, the resulting fairness is $F_{SSIM}=0.71$. In case of SPM and NADE, the mean SSIM is reduced while the video quality fairness is enhanced. In case of QoE-FF, the mean SSIM and the fairness are enhanced. This, however, comes at the costs of an increased number of stallings, which is illustrated in Fig.  8.

For a bandwidth limit of 2400 kbit/s, SPM and baseline yield similar results for both, the mean SSIM and the fairness in terms of SSIM. NADE enhances the baseline’s fairness by 0.25, resulting in an index value of $F_{SSIM}=0.74$. QoE-FF achieves an SSIM fairness of $F_{SSIM}=0.77$. However, for both approaches, the mean SSIM is reduced. In case of NADE, which yields a slightly lower fairness than QoE-FF, the average video quality is decreased to a lesser extent. A similar behavior is observable for the 4800 kbit/s scenario. In case of a 7000 kbit/s bandwidth limit, the baseline’s fairness of $F_{SSIM}=0.82$ is increased by NADE and QoE-FF to a value of roughly $F_{SSIM}=0.93$.

The results clearly indicate the potential of QoE-FF and NADE to enhance the video quality fairness among heterogeneous clients. However, in the absence of stallings, this enhancement comes on the expense of a reduced mean SSIM of all clients.

Initial delay

This section targets the results with respect to the initial delay. In the configuration with a bandwidth limit of 1440 kbit/s, QoE-FF and NADE enlarge the initial delay. The initial delay for baseline is on average 9.5 s. SPM slightly reduces this value to 8.88 s. When applying QoE-FF, the users have to wait 15.19 s on average until the video playback starts. NADE enlarges the initial delay to 14.49 s on average. To identify a possible correlation between device type and initial delay, we perform a deeper investigation of the initial delay for the different mechanisms with respect to the clients’ device resolutions. The results for all bandwidth limits are depicted in Fig. 11.

Fig. 11

Illustration of the initial delay with respect to different device types and different bandwidth limits

For the 1440 kbit/s bandwidth limit (Fig. 11a), baseline and SPM both show the behavior that devices with higher resolution need more time to fill the initial buffer level. Users of a 360p device have to wait on average 6.62 s until the video starts. Users of a 1080p device have to wait roughly twice as long. SPM slightly shortens the 1080p device’s initial time to an average of 11 s. For the remaining device types, no significant differences are observable. When QoE-FF is applied, the 1080p device has a mean initial delay of 19.65 s. The initial delay is enlarged for the other device types, too. Due to the overlapping confidence intervals, no conclusion can be drawn whether NADE preserves the trend of longer initial delay for higher resolution devices. However, the average initial waiting time is increased for the 360p and 720p devices compared to the baseline.

For higher bandwidth limits (Fig. 11b–d), all initial delays decrease. When baseline and SPM are applied, the 360p devices are capable of starting playback after a short waiting time, 720p and 1080p devices need slightly more time. This is similar for QoE-FF, however, for the 2400 kbit/s limit, it cannot be concluded whether 720p or 1080p devices have a longer intial waiting time, as the confidence intervals overlap. In the case of NADE, no conclusions can be drawn whether the initial waiting time is correllated with the screen resolution. This holds for all investigated bandwidth limits

To summarize, SPM and baseline provide the smallest initial delay. Further, the high resolution devices experience a larger initial delay than devices with smaller resolutions for all investigated cases. Further, the initial waiting time increases for higher resolutions if baseline or SPM are applied. This holds for all of the four investigated scenarios.

Number of exchanged messages

Based on the monitoring and control capabilities of a particular App-Net approach, different information is exchanged with the PM with different granularity. Consequently, the total number of exchanged monitoring and control messages differs between the mechanisms and their specific configurations.

We examine in the following the total amount of exchanged control messages and the share each of four different message types contributes. The types are defined by a message’s purpose. Hence, we differentiate network monitoring (NMN) and application monitoring (AMN) messages as well as the respective messages for controlling, namely NCT and ACT messages.

We depict two different configurations for each strategy, in order to highlight configuration-specific characteristics. The bandwidth limit is set to 4800 kbit/s. The illustrated behavior is representative for all other bandwidth configurations, where the number of exchanged messages slightly differs. The number of exchanged messages illustrated in Fig. 12 covers the time between 120 and 360 s of the experiment. Hence, the transient phase is excluded. In case of QoE-FF, the AMN messages serve as keep-alive messages and are sent from the clients to the PM before each GET request. As the number of GET requests is not influenced by the underlying QoE-FF configuration, no difference concerning the amount of this type of message is observable. In our implementation, network monitoring information is transmitted using the keep-alive messages (AMN). Since this information is logically separated from the AMN messages, they are illustrated in the figure. For both configurations of QoE-FF, $q_2$ and $q_3$, around 360 AMN and NMN messages are exchanged. The configuration affects the number of exchanged ACT messages. On average, 287 messages of this type are sent when applying $q_2$, where 95% of the bandwidth is considered at the algorithm. When configuration $q_3$ is applied, where the whole bandwidth is considered to be available, 180 ACT messages are sent on average. QoE-FF does not perform network control. Hence, no NCT messages are exchanged when using this strategy. In total, 1008 messages are exchanged for $q_2$ and 900 for $q_3$

Fig. 12

Number of exchanged messages, distinguished by the message purpose. Application Monitoring (AMN), Application Control (ACT), Network Monitoring (NMN), Network Control (NCT)

For the SPM configuration $s_5$, the network throughput is polled each 2 s, for the configuration $s_{11}$, this time interval is enlarged to 8 s. Consequently, the $s_5$ configuration leads to a higher number of NMN messages as compared to $s_{11}$. After each decision, which is triggered by a client’s AMN message before its GET request, both, network and application, are informed about the outcome. As the number of GET requests hardly differs for different configurations, there is only minor impact on the number of AMN, ACT, and NCT messages. In total, 1191 messages are sent when $s_5$ is applied, 1113 when $s_{11}$ is applied.

For NADE configuration $n_1$, where the throughput is polled once per second, 239 NMN messages are sent. With a network monitoring granularity of 10 s only ($n_2$), 24 messages of this type are exchanged. 31.5 NCT messages are exchanged with configuration $n_1$ and 26.5 for the second configuration. As the AMN messages are sent before each GET request, their number is not influenced by the configuration. Since no application control is performed using NADE, no ACT messages are exchanged. This leads to an overall number of 635 ($n_1$) and 411 ($n_2$) messages.

All examined strategies exchange a similar number of messages of the type AMN. However, whilst SPM requires each of the four message classes, there is no need for NCT messages in case of QoE-FF, and no need for ACT messages if NADE is applied.

Holistic comparison of the investigated mechanisms

In the following, the three presented App-Net strategies are compared to each other with respect to several metrics and under different network conditions. To illustrate the differences between the investigated mechanisms we rely on spider plots.

Figure 13 depicts the overall performance of baseline and App-Net strategies for a bandwidth limit of 1440 kbit/s.

Fig. 13

Illustration of the results for baseline implementation and the three App-Net strategies with respect to several QoE-IFs. Bandwidth limit 1440 kbit/s

QoE-FF clearly outperforms the other strategies in terms of minimum SSIM, mean SSIM, and the number of quality switches. Further, it enhances the fairness in terms of video quality to a similar extent as NADE. However, QoE-FF drastically enlarges the number of stallings, the overall stalling duration, and the initial delay.

NADE leads to an increase of the initial delay, too. It yields the highest number of quality switches but can enhance the baseline’s minimum SSIM. At the expense of the mean quality, NADE also enhances the video quality fairness. Furthermore, it is capable of reducing the number of stalling events and the stalling duration compared to the baseline implementation.

This also holds for SPM. Besides the stalling reduction and the overall stalling time, SPM is also capable to slightly shorten the initial delay. However, SPM leads to a higher number of quality switches and reduces the mean SSIM.

In each dimension, the baseline implementation can be surpassed by at least one of the strategies. However, none of the strategies can accomplish an improvement of the baseline with respect to all evaluated metrics.

Using the implemented QoE-FF strategy in the outlined scenario results in a severe degradation of the user experience due to the high number of stallings. This might even lead to user abandonment after experiencing a few stallings. NADE should be applied in a scenario like this, as it achieves both, a high SSIM fairness and a smooth streaming.

Figure 14 shows the results for a bandwidth limit of 2400 kbit/s.

Fig. 14

Illustration of the results for baseline implementation and the three App-Net strategies with respect to several QoE-IFs. Bandwidth limit 2400 kbit/s

The Quality of Experience Fairness Framework outperforms baseline and the other approaches with regard to video quality fairness, the number of quality switches, and minimum SSIM. However, it has the lowest mean SSIM. On average, there is less than a half stalling for any of the investigated solutions. The Stalling Prevention Mechanism is capable to stream without any interruption, and hence, slightly improves the baseline. Except from stalling number and duration, SPM and baseline hardly differ in their outcomes. NADE also enhances the video quality fairness, however, not to such an extent as QoE-FF does. In return, NADE yields a higher mean SSIM than QoE-FF. However, with an initial delay of 10.5 s on average, NADE needs the longest time until the video playback starts. As QoE-FF and NADE both enable smooth streaming whilst being fair, one of these mechanisms should be applied. Effectively, SPM has no significant positive impact, as the available network resources nearly allow an interruption-free playback.

Figure 15 shows the outcomes for a higher bandwidth limit of 4800 kbit/s.

Fig. 15

Illustration of the results for baseline implementation and the three App-Net strategies with respect to several QoE-IFs. Bandwidth limit 4800 kbit/s

As the fairly high bandwidth limit of 4800 kbit/s triggers hardly any actions for the Stalling Prevention Mechanism, its results are quite similar to the baseline’s outcome. However, the baseline’s small stalling number and stalling duration can still be reduced by using SPM. SPM and baseline achieve the shortest initial delay and the highest mean SSIM. Contrary, they suffer under a lower video quality fairness compared to QoE-FF and NADE.

QoE-FF achieves the highest SSIM fairness. In consequence of providing all clients a similar video quality, the minimum SSIM increases and the overall mean SSIM decreases. Compared to the baseline and all other App-Net strategies, QoE-FF enlarges the initial delay but also yields the fewest quality switches.

NADE also enhances the SSIM fairness and the minimum SSIM for the sake of a lower mean SSIM. It has the highest number of quality switches among all examined strategies and enlarges the initial delay compared to the baseline implementation. For this bandwidth configuration, we propose to apply NADE or QoE-FF as both of them enhance the SSIM fairness and slightly reduce the number of stallings compared to the baseline.

Figure 16 illustrates the results for the highest bandwidth limit of 7000 kbit/s.

Fig. 16

Illustration of the results for baseline implementation and the three App-Net strategies with respect to several QoE-IFs. Bandwidth limit: 7000 kbit/s

In this case, NADE is the mechanism that yields the highest fairness with respect to video quality. However, it also causes the highest number of quality switches. QoE-FF outperforms NADE, SPM, and the baseline with respect to the quality switches. The resulting fairness index for QoE-FF is lower than for NADE. For both mechanisms, there is less than half a stalling event on average per client, they are both suitable in this situation. Again, SPM should not be applied, as it cannot significantly reduce more stallings than the other mechanisms, but enlarges the initial delay and the number of quality switches.

Conclusion

This paper is a first step towards a systematic comparison of different QoE-centric App-Net interaction mechanisms. Firstly, we give an overview of such approaches and use them to derive an abstract interaction model based on the identified functional blocks. Secondly, we evaluate three App-Net approaches for adaptive video streaming.

We show that the investigated approaches differ with respect to the number of exchanged messages and their influence on stalling and average video quality. The stalling prevention mechanism requires a constant message exchange and is thus very costly. It reduces stalling if little network resources are available, i.e., if all clients can barely be supplied with the lowest quality. Its impact on the user-centric metrics is negligible if enough network resources for higher video qualities are available. The QoE fairness framework provides a fair video quality among heterogeneous clients for the price of a lower average video quality. As long as enough network resources are available, no stalling times occur. If the lowest video quality can be barely supplied, large stalling times are observed, indicating that the approach should not be used in such a scenario. The network assisted DASH fairness enhancement provides similar, slightly smaller results compared to the QoE fairness framework in terms of video quality fairness and average video quality. However, it allows a smooth video playback for low bandwidth scenarios. The results further indicate, that for a constant bandwidth scenario, the stalling prevention mechanism is very costly in terms of exchanged messages. The smallest amount of messages is exchanged when using the network assisted DASH fairness enhancement. Accordingly, this approach seems to be the most reasonable solution for constant bandwidth scenarios.

The presented results illustrate the importance of a detailed understanding of application–network interaction approaches and of identifying corresponding use-cases for them. The introduced framework is the first step towards a holistic reproducible evaluation of App-Net approaches that focus on improving QoE. For that, we provide an environment and the respective building blocks that allow an investigation of a variety of App-Net mechanisms. Firstly, the implemented ØMQ messaging broker facilitates the interaction of distributed components that execute monitoring and control functions. This messaging broker is easy to set up and enables a straightforward integration of additional entities and algorithms, which simplifies extensions of the framework with new mechanisms and its use for different applications. Secondly, we provide a basic set of concrete implementations of monitoring and control functions for DASH-based video streaming, as well as a corresponding evaluation methodology that combines various QoE indicators for DASH. Hence, we encourage the community to extend our work by integrating new App-Net functions and mechanisms, comparing them with the existing ones, and evaluating other multimedia applications.