In our sixth installment to our series “NPAWering Online Video”, Adri, one of our backend developers, explains how the Raspberry Pi helps QA and Beacons teams to complete technical tasks in a much easier way.
One of main tasks of the Quality Assurance Department at Nice People At Work is to validate that the beacons delivered to the customer behave as designed. The best way to assure the quality of the beacons is to mimic the exact conditions that a user might experiment during playback such as: limited connectivity, loading resources that no longer exists, among others.
Currently, there are several tools that provide the functionalities required for these tasks, however our QA engineers need to use a lot of different tools to perform different tasks which can be uncomfortable. Furthermore, some of these tools require “hacker” level skills to use them, like messing with Linux network interfaces.
So, we decided to create a tool that can simplify and speed up QA tests. The requirements for this new tool are challenging, and it has to provide functionalities for:
– Buffer creation
– DNS spoofing
– IP Blocking
– VPN Connection
– HTTPS decryption
– Packet sniffer
All of these capabilities must be provided in a unified, aesthetically pleasing, and user-friendly application.
We decided to use Raspberry Pi devices to develop this QA tool.
“Quote from www.raspberrypi.org FAQ”:
The Raspberry Pi is a credit card-sized computer that plugs into your TV and a keyboard. It is a capable little computer which can be used in electronics projects, and for many of the things that your desktop PC does, like spreadsheets, word processing, browsing the internet, and playing games. It also plays high-definition video. We want to see it being used by kids all over the world to learn programming.
In short, it’s a tiny a computer with a embedded Linux, so a developer can make almost anything with it.
In our design, the Raspberry Pi device connects to the enterprise network via ethernet and creates a Wi-Fi Hotspot in order to not interact in any way with the network. This Wi-Fi hotspot provides internet connectivity, through Raspberry Pi (and our enterprise wired network), to all the devices connected to the Wi-Fi. This provides us a way to tweak the network behaviour only for the devices connected to the Wi-Fi.
Creating a hotspot with Raspberry Pi as the host device is a very common search in Google, so making it is not a hassle. Depending on the Wi-Fi interface, one may encounter problems related with the drivers – good luck with that!
YOUBORA obtains the quality information from a user’s playback session. One of the quality metrics that our customers are interested is in measuring the Buffer Ratio (how much time the user is buffering content in relation with the length of the content). In order to do that, the beacon attached to the player reports buffering periods. So, the QA team needs to verify that this information is accurate.
In some scenarios, this is easy to simulate. For instance, in Google Chrome there is an Emulation Tab where the user can modify his network conditions. However, in some devices such as Set Top Boxes and some mobile devices, it not possible to do that without hacking, somehow, into the device.
Here comes Linux Traffic Control package to the rescue! With this package the user can control all the variables related with network interfaces such as latencies, size of queues, et cetera. For example, in order to simulate 1 mbit connection:
tc qdisc add dev wlan0 handle ffff: ingress && tc filter add dev wlan0 parent ffff: protocol ip u32 match u32 0 0 action mirred egress redirect dev ifb0
tc qdisc add dev ifb0 root handle 1: htb default 10 && tc class add dev ifb0 parent 1: classid 1:1 htb rate 1mbit && tc class add dev ifb0 parent 1:1 classid 1:10 htb rate 1mbit
What is found here is the configuration of a redirection to a wlan0 interface to a virtual interface where the are applied the queue discipline HTB for a rate of 1 mbit. As it can be observed, this is not beginners stuff, so in order to simplify this process we have created an script to enable limited network with a set of different network limitations. The webapp invokes this script in order to enable or disable the buffers.
In order to simulate a resource that fails to load, what the QA team usually did was modify their /etc/ hosts pointing the requested resource URL to an URL where it will fail, for example local host:
However, we end up in a situation where they can test browsers, but no mobile or other devices. In order to do that, the Raspberry Pi services offers a functionality to perform exactly that, but at network level.
The DHCP messages sent by the Wi-Fi Hotspot announces the IP of the Raspberry Pi as an Internal DNS Server. Then, the devices asks the IP to configure the DNS Server running on the Rasperry Pi. This DNS Server is a dnsmasq process configured to read a file similar to /etc/hosts, where a URL points to an IP.
The webapp simply writes on that file the desired DNS Spoofing configuration
We found that some devices, such as Google Chromecast, ignore the DNS entry in the DHCP messages sent over the Wi-Fi connection. After some time investigating the issue, we observed that these devices have “hardcoded” the DNS configuration to point to Google DNS (220.127.116.11). After further investigation, we found that if we block the IP 18.104.22.168, the devices works as expected upon the first instance (obtaining the configuration from DHCP messages).
Technically, this blocking is as easy as adding an entry to the iptables configured in the Raspberry Pi.
Some of our customers work on a Geo-Block scenario where their contents can only be consumed in certain countries. Since it is crucial to test their applications, it is necessary that our engineers bypass this geographical block. In order to perform this bypass, the Raspberry Pi has configured a VPN service to connect to multiple VPNs abroad in order to simulate the connection from different countries. This service is a simple openvpn service configured with a series of configuration files provided by our VPN provider.
There are some cases where the QA team needs to inspect the traces that the beacons send to YOUBORA. They usually perform this task watching the HTTP messages sent from the tested devices to our backend. However, in a secure scenario this cannot be achieved because the communication is encrypted end-to-end.
In order to achieve this, we installed a haproxy in the Raspberry Pi with our HTTPS certificates, which is able to receive and decrypt the incoming traffic and forward it to the appropriate destination.
In fact, this is a little bit more complicated because the QA engineers wants to observe the traces, so the haproxy must first receive the traffic, decrypt it, and send it to a special service inside the Raspberry Pi. This special service communicates with the webapp in order to present the incoming decrypted HTTP traffic to the user. While the HTTP traffic is observed in the webapp, the special service is forwarding the messages to its original destination in order to keep the tested application running without interfering.
The main drawback of the use of this tool is that the QA engineer using it has to configure the DNS Spoof module to direct the Raspberry Pi IP to act as our backend.
Since there are some situations where the HTTP traffic must be observed and analyzed, we include a Packet Sniffer inside the developed service.
This service is based on libpcap library which enables oneto programatically have an event for each network message that goes through the Wi-Fi network interface, enabling the user to observe the traffic without opening network analyzers like Wireshark.
The Raspberry Pi device may be small, but it is mighty. With a little tinkering, it can function as you wish. Whether it’s creating a Wifi hotspot or performing some HTTPS Decryption, this credit card sized computer can tackle a plethora of challenges. For this reason and many others, we embrace Raspberry Pi in our QA department, so we can help serve our customers better.
Just another thing to think about from us here at NPAW.
James Noeker on October 13th 2016
As the streaming market matures and user expectations surge, the ability to immediately detect issues and share up-to-date reports across an organization is becoming an essential part of the day-to-day work lives of video industry professionals. In response, NPAW has launched the YOUBORA mobile app,...
As streaming becomes an ever-more established behavioral trend globally, online video providers are faced with a double-edged sword. Demand for online video is higher than ever, but so too is competition for space in consumers’ viewing habits. User expectations are on the rise and, as it is easier...
The outbreak of the COVID-19 virus is drastically changing user behavioral patterns worldwide. As the virus spreads, streaming video providers must contend with sudden changes in variables including rising economic pressures, altered routines, moods, and priorities. We have gathered together a series...