Academy blog

How Our Quality Assurance Process Guarantees The Most Reliable Data

The integration process for YOUBORA Suite is a rigorous and in-depth process which guarantees the most reliable data on the market.

 

When you subscribe to a video service like the Adobe Suite or Microsoft Office, it’s first necessary to download the software and install it onto your computer. This is the integration process on a very easy-to-understand level.

The application will ask for certain permissions in order to integrate with your operating system and make sure it is able to run efficiently. Integrating your video player with a data analytics tool like YOUBORA Suite is similar to this, only much more detailed.

As a company, NPAW processes 6.5 trillion data events per year. It’s a ton of information and therefore needs to be monitored in order to guarantee that our customers aren’t being misinformed – this is where the QA team come in.

While many analytics tools for online video may be able to offer a plugin-and-play integration process or propose a time frame of only a day or two, the integration process at NPAW takes the necessary amount of time to guarantee the most reliable data.

Data is vital for our customers and however crucial rolling the product out is, nothing is as important to us as the accuracy of data.

Maria Lopez is our QA team leader and oversees all of the work we put into our integrations process. This process has changed over the years to accommodate product growth and new concerns within the video industry. Maria found some time between quality checking new features and integrating customers to take us deeper into our process to explain why we prefer to take our time.

“We are covering more parameters and metrics than ever before. This is constantly being reviewed, updated and customized to make sure each customer is being given the best quality assurance process.”

NPAW: Can you take us through the QA process?

Maria López: QA starts when a customer has integrated the plugin. We then send a document to the customers which gives us all the information needed to access the environment and whether there is a specific platform or configuration that needs to be looked into specifically. This also allows us to know whether we need to work around geo-blocking.

Geo-blocking is one of the most common drawbacks for us. We are testing a lot of applications from the United States, Australia, Japan and sometimes players block certain geo-locations, making it hard for us to check everything without getting cleared by the player and given access.

After they have fed us this information, we begin testing. Depending on the technology, we then go into our QA environment and test all possible configurations of device, location and CDN among many others to guarantee that data is being fed correctly to the YOUBORA platform and that this matches the numbers we see within the plugin.

Once we are finished testing, we send the customer a report. This includes every test case that we carried out and whether it passed the integration requirements. It is here that we need the customer to help us. If some data isn’t being sent or there are inconsistencies then we have to ask them to change something within their code which might be stopping their platform from reporting its data correctly.

Because of this, one integration may last two weeks, another may last months, it all depends on how many bugs we find and how capable our customers are of manipulating their code.

Once it has passed, we let the customer know they can move into production, guaranteeing a minimum assurance of quality according to our thresholds.

“Sometimes we’ll even send the whole team out to a customer in order to get the platform integrated as fast as possible for the customer.”

NPAW: Tell us more about thresholds

ML: We have thresholds for the most important metrics that are designed according to our benchmark data. This includes bitrate, join time, buffering and of course, plays. We have a lower and upper threshold for all of our metrics and make sure that there are no irregularities in the reporting of customer data.

The final step for us is then data monitoring while the customer is live. This step can last anything from two weeks to one month depending on the necessities of the customer and the amount of data they are streaming.

It’s very different using the system on your computer, feeding smaller and controllable amounts of data, compared to a customer who may be receiving hundreds of thousands of views a day. That’s why we track each customer to guarantee that they are continuing to receive the quality of data we strive to guarantee.

If we’re testing web video it normally takes two days to complete this part. It means we’re testing on Mac and Windows and then within each of those we’re testing within every browser.  It means testing on mobile and on consoles too so the process is extremely rigorous.

Sometimes we’ll even send the whole team out to a customer in order to get the platform integrated as fast as possible for the customer.

NPAW: How many people are in the QA team?

ML: There are six of us in QA, however, in terms of testing integrations, that involves five people. My job involves testing the functionality of new features within the UI. The integrations team is then divided into two. One team is focussing on the United States and Australia, the other is focussing on the rest of the world.

NPAW: How has the QA process changed over the past four years?

ML: The quality department began pretty much when I joined four years ago. But since then it has changed so much. In the beginning, we might only have checked the main events but we’ve learned to adapt to new demands from customers and have established our process as the best in the business. We are covering more parameters and metrics than ever before. This is constantly being reviewed, updated and customized to make sure each customer is being given the best quality assurance process.

We work very closely with the Plugins teams. However, many of the changes we have to adapt to come from the Product department. We’re always staying on top of new developments.

For example, when we updated all of our players from v5 to v6, this allowed us to access so much more data. This data was necessary for our sessions events data which can be accessed through YOUBORA Analytics. This means we are able to read and measure any event that their webpage can send us. For example, if they wanted to measure purchase events, the customer can also read how much purchases are worth so TVODs can track what actions a customer typically takes before making a big purchase.

We have also evolved our software to understand how many people are using each application within YOUBORA to help us understand where we should be focussing our efforts.

NPAW: What are the most important things you keep an eye out for?

ML: Of course, the most important thing is plays. Some customers may be charged per play so we make sure that the number of plays in YOUBORA matches that of their player so no customer is ever being overcharged.

Additionally, errors are one of the most important things a business can track. We have been building the Errors Dashboard recently and this is the perfect place for customers to see exactly how many errors are occurring, where in the playback they are happening and whether they lead to a view crash or not.


To benefit from the most reliable data on the market, speak to one of our team and learn everything about YOUBORA suite as well as how it can transform your video service – sales@nicepeopleatwork.com

Max Gayler on March 07th 2019

Related Post

NICE PEOPLE AT WORK will be live streaming the 080 Barcelona Fashion

It’s that time of year again, fashion designers and models are ready for today’s runway, and NICE PEOPLE AT WORK will be there too! From the 2nd to the 5th of February we will give you live, high quality and uninterrupted streaming of all the shows of the 080 Barcelona Fashion, so you won´t... 

2 Feb

NICE PEOPLE AT WORK and Brightcove become partners in Brightcove’s new player generation

YOUBORA Analytics is available in Brightcove’s new advanced player to maximize End User Quality of Experience in OTT delivery Brightcove and NICE PEOPLE AT WORK (NPAW) worked together to enhance Brightcove’s brand-new player with NPAW’s YOUBORA Analytics, the most advanced real-time metrics... 

10 Feb

Quality of Experience (QoE): Option or Need?

Quality of Experience has become an overused term in the online video industry, but also one of the big challenges that online broadcasters and OTT´s have to deal with. Even more now, a period in which users spend more time connected to the digital world than to reality. Moreover, expectations and requirements... 

16 Feb