Story Highlights

• Bengaluru-based Zapr Media Labs has developed a tech that listens to your ambient sounds to profile you for targeted advertising — all through your smartphone microphone.
• The tech makes its way into the user’s device with a code on third-party apps such as Chota Bheem games to Dainik Bhaskar to, likely, even Hotstar — all without explicit consent.
• Justice B N Srikrishna, who headed the team behind India’s draft privacy law, called out the Zapr model as a “breach”.

Have you ever felt that your phone is listening to you?

Have content recommendations occasionally been too good to be true?

What about that product pushed at you before you finally buy it the fourth time?

Ever wondered about any or all of these?

The answers: No, your phone is not “listening” to you in the strictest sense of the word. But, yes, all your likes, dislikes and preferences are clearly being heard by apps in your phone which you oh-so-easily clicked “agree” to the terms of which while installing.

How so?

If you are in India, the answer to the question will lead you to Zapr, a service backed by heavyweights such as the Rupert Murdoch-led media group Star, Indian e-commerce leader Flipkart, Indian music streaming service Saavn, and mobile phone maker Micromax, among more than a dozen others. The company owning Zapr is named Red Brick Lane Marketing Solutions Pvt Ltd. (Paytm founder Vijay Shekhar Sharma and Sanjay Nath, co-founder and managing partner, Blume Ventures, were early investors in Zapr but are no longer so, according to filings with the ministry of corporate affairs. Sharma and Blume are among the investors in Sourcecode Media Pvt Ltd, which owns FactorDaily.)

Zapr, in fact, is one of the few companies in the world that has developed a solution that uses your mobile device’s microphone to recognise the media content you are watching or listening to in order to help brands and channels understand consumer media consumption. In short, it monitors sounds around you to contextualise you better for advertising and marketing targeting.

A step back for the 30,000-feet view. If one is to survey the global internet landscape, Google pretty much owns search advertising, Facebook pulls in much of spending on social media, and Amazon is arguably ahead in e-commerce and its adjacent businesses. China’s Alibaba, Tencent and Baidu have staked their claim, too, thanks to the sheer economic heft of the Middle Kingdom.

As the internet expands wide and deep, next on the horizon is spatial search (think Google Maps) — and, even bigger, voice search (think Google Voice, Apple’s Siri and Amazon’s Alexa). Almost as a corollary, a bunch of companies is betting on voice blooming on its own helping better profile consumers. Advertisers globally spend some $650 billion annually and this cohort believes better profiling consumers by analysing their ambient sounds helps target advertising better. This group includes Chinese company ACRCloud, Audible Magic from the US, and the Netherlands’s Betagrid Media — and, Zapr from India.

Cut back to the Zapr headquarters on Old Madras Road in Bengaluru. One of the apps that inspired Zapr’s founding team was the popular music detection and identification app Shazam. But, its three co-founders saw opportunity in going further. “Instead of detecting music, can we detect all kinds of medium? Can we detect television? Can we detect movies in a theatre? Can we detect video on demand? Can we really build a profile for a user about their media consumption habits… and that really became the idea, the vision we wanted to solve for,” Sandipan Mondal, CEO of Zapr Media Labs, said in an interview last week on Thursday.

Shorn of jargon, the underlying Zapr tech listens to ambient sounds around you, analyses it, and profiles users based on their media consumption habits. “That data would be very useful in order to recommend the right kind of content and also for brands and advertisers to hopefully reduce the wastage and inefficiencies and make smarter decisions,” said Mondal, who co-founded the company in 2012 along with his batchmates from Indian Institute of Management, Ahmedabad (batch of 2010) Deepak Baid and Sajo Mathews.

Zapr claims to have the largest media consumption analytics database in India and helps television channels and brands to earn a better bang for their advertising buck. To be sure, advertising – even with the internet’s promise of better targeting – still is an inaccurate business with proxies, at best, helping measure its return on investment. More on this later.

Distribution deceptive

But, Zapr’s tech comes with privacy and data concerns – lots of it. The way its tech gets into your phone is dodgy: its code ride on third-party apps ranging from news apps to gaming apps to video streaming apps. You might be downloading Hotstar or a Dainik Jagran app or a Chotta Beem app on your phone little knowing that Zapr’s or an equivalent audio monitoring code sits on those apps to listen to sounds around you in an attempt to see what media content you are consuming.

In most cases reviewed by FactorDaily in a two-week exercise, it was not obvious that the app would monitor audio via the smartphone or mobile device’s microphone for use by another party (Zapr) for ad targeting purposes. Some apps hinted about Zapr’s tech at the bottom of the app description and some in the form of a pop-up – an app from Nazara games, for instance, mentioned that it required mic access to ‘Record Audio for better presentation’. Sometimes, the pop-up app would show up a few days after the download. And, often, the disclosure was buried somewhere in the app’s privacy policy.

None of these apps made it clear explicitly what the audio access via the microphone was for. “The problem with apps which embed this technology is that their presence is not outright disclosed and is difficult to find. Also, there is not an easy way to find out the apps in the PlayStore that have this tech embedded in them,” said Thejesh G N, an info-activist and the founder of DataMeet, a community of data scientists and open data enthusiasts.

With user privacy concerns more in focus than ever before after the Facebook-Cambridge Analytica fiasco, most apps try to stay away from – or be discreet as long as they are clear in the eye of the law – such tech embedded in them. But, as evident with the Zapr ecosystem, app monetisation is reason enough to give user privacy the short shrift.

On the outside, what Zapr says it does for users and app developers or media companies is straightforward:

• App developers are allowed to embed Zapr’s tech in their apps. In return, they will get better ad placement (read: better targeting) and higher revenues.
• As part of the deal, the apps give Zapr access to the user’s microphone to capture and analyse a person’s media consumption (via ambient audio monitoring).
• What users stand to benefit from this is still in a grey area. Yes, you get better movie recommendations but you may not be aware that you are helping advertisers target you and those around you better.

The phones-listen-in tech

The core technology behind how these apps work is called audio content recognition (ACR). What this tech essentially does is identifying a user’s content consumption on television, radio and other digital medium using the microphones on their mobile devices. The audio content captured is converted into data fingerprints which are then compared to an existing database of fingerprints to crossmatch the content and identify them.

“You essentially create a fingerprint of ambient audio. You take a second, a couple of seconds… you create a fingerprint. A fingerprint is called so because it is a digital representation of the original signal, a little digital file. And, at the backend you have the original content fingerprinted. Then, it is just pattern recognition. Match the fingerprints (with the backend) to see if there was a match,” explained Mondal.

Indeed, the media centre at Zapr’s Bengaluru office is analysing hundreds of TV channels in real time and is fingerprinting every second of their content. “Any second of content that passes through the servers at the media centre has a corresponding unique fingerprint which is timestamped and we know it belongs to this channel and that is archived away,” said Mondal, describing how Zapr builds it reference library media content fingerprints.

The tech periodically wakes up to create the fingerprint for analysis, says Mondal, emphasising it is not a continuous process. “Over a day or week, the total number of hours spent watching content is going to be limited and so it doesn’t make sense for this to operate the entire time. So, we learn based on the data which comes in, when are you likely to be consuming content, when you are definitely not likely to be consuming content and accordingly we vary that periodicity of the checks of the fingerprinting process.”

The software code here is not unlike the tech available on your mobile device on apps such as Shazam, acquired by Apple earlier this year. But, unlike Shazam, where you know exactly what you are signing up for, the way Zapr’s ACR tech reaches millions of customers is not clear or upfront.

Case in point: the Hotstar app, India’s most popular video streaming app with more than 100 million downloads. Hotstar, owned by Star India, has ACR technology embedded in it but the only way to discover the audio-monitoring capability of the app is to go to its privacy policy. One of the subheads in the policy mentions the presence of the ACR tech but it is not clear if it is powered by Zapr, though the description was similar to how Mondal explained it to FactorDaily.

“We have integrated SDK in our Platform that is designed to measure and analyze television channel viewership of Hotstar Users (“TV SDK”). The TV SDK initiated after obtaining relevant permissions uses the microphone on your device to capture audio samples (including ambient sounds) and create encrypted fingerprint files of such audio samples in real-time. The fingerprint files are matched against a database of known fingerprints of television channels and television content to identify and understand your television viewership patterns,” the relevant part of Hotstar’s privacy policy reads.

Last March, Hotstar and Zapr said they were partnering for mobile audience analytics, according to news reports. Details and scope of the partnership were buried in jargon and gobbledygook but one sentence stood out: “Zapr’s proprietary technology platform analyzes television viewership across 600+ channels in India providing targeted digital analytics and insight into offline consumption behaviour.” (Emphasis is ours.) FactorDaily can interpret “…insight into offline consumption behaviour” only as the ACR tech under the Zapr – and, likely, Hotstar – hood.

To be sure, in the week that FactorDaily downloaded the Hotstar app it did not ask for permission to the microphone. But, there has been at least one report that Hotstar does record ambient sounds.

When asked about the ACR tech in Hotstar’s app, Mondal, other open through the interview, was not forthcoming. “Unfortunately, I can’t speak about those things,” he said, offering to direct us to Hotstar.

FactorDaily’s queries to Hotstar sent on Thursday remain unanswered.

A dipstick survey of eight Hotstar users of the app showed that all of them were unaware of the ACR tech that the privacy policy talks about.

The law and informed consent

It is a matter of perspective but would you call the way the tech gets into your house sneaky or is it up for debate?

On being asked about the randomness and delay at times when the pop-up regarding Zapr’s SDK or the microphone access is displayed, Mondal pointed in the direction of the app developers. This is done by the app developer in order to not bother the app user with too many permissions at the time of the download, he explained.

All apps don’t display a pop-up at the first boot-up, he said. “Depending upon their user strategy, they always don’t want that pop-up to show up the very first time the app is updated. They have their own algorithms, sometimes it is on the third boot-up or on the fifth boot-up (that) they will request the user for permissions. Because often times if you catch the user at the right moment, they may be more willing to consent and so because the apps think they know their users better, they decide when they want to do it.”

But, an app will not have microphone access without the pop-up being shown and the user acceding to the request, Mondal added.

 

Since India does not have a privacy law in place yet, how do lawmakers or regulators elsewhere in the world look at such a practice? Let’s turn to a two-and-a-half-year-old regulatory ruling from another part of the world. A different ACR method used involves inserting an audio watermark into content which can then be picked by a device and used to identify the content. For example, a particular part of TV programming you are watching could be embedded with an ultrasonic audio sound that is unique and whether you are watching it or not be monitored. Silver Push, a Delhi-based company had developed a similar solution, which lead to the US Federal Trade Commission (FTC) warning a dozen app developers who had embedded the Silver Push code in their apps to not to use it due to privacy concerns. Silver Push has, since then, said that it has stopped using the product.

We asked retired Supreme Court judge B N Srikrishna, who headed the committee that drafted India’s data privacy and protection bill, what he thought of how Zapr distributes its technology on third-party apps. “That is totally a breach of privacy,” he said without hesitation. “Unless they give me clear cut indication that it is going to be used only by them (the app that hosts the code) for a specific purpose and not be shared with any other party, that amounts to breach,” he said in a phone interview.

Srikrishna felt that one of the challenges is the way different countries look at the law regarding data privacy and the lack of one in India had led to instances such as Zapr’s distribution. But, “the only thing is that the law has not yet been passed. Today, there is nothing that you can do about it… Once the law has been passed and becomes applicable then they will be caught under the law,” he said.

FactorDaily found the Zapr code embedded in more than a handful of gaming and news services going by their pop-ups, privacy policy or app description. This list includes NewsDog, an India focused news app from China, with over 50 million installs; apps by the two of India’s largest circulation newspapers Dainik Jagran and Dainik Bhaskar with over five million installs each; and the Rajasthan Patrika app, among others in the news business. Others include Nazara Games famous for their cricket-based and Chotta Bheem series of games; Games2Win; and, JetSynthesys that has developed popular games like Sachin Saga Cricket Champions (over five million downloads, according to the Google PlayStore). Another app ZEE5, a video on demand website run by Zee Entertainment, in its previous avatar as OZEE seemed to have Zapr’s ACR tech embedded in it, according to its privacy policy but the newly updated policy does not mention this though. These names are likely a small set of apps with the Zapr code out there.

We reached out to NewsDog, Hotstar, Nazara and Dainik Jagran about Zapr’s tech being embedded in these apps. NewsDog founder Forrest Chen, at first, said on an email that the app did not have ACR tech in it but in a follow-up mail said that it had used Zapr earlier but not for audio monitoring. Our conversation was on Friday. Later, the same day, NewsDog’s app description was modified to remove the mention of Zapr’s SDK but at time of publication of this story, the Zapr SDK’s presence was still in the privacy policy.

We are yet to hear back from the other companies.

One of the reasons for tech like this being used is because of the more precise nature of advertisement targeting and retargeting this can provide. GRP, which stands for Gross Rating Point, is a standard used to analyse the impact of an advertisement. According to Rahul Vengalil, CEO and co-founder of digital marketing, audit and consulting firm What Clicks, new technologies are being used to improve the GRP of advertisements and the way media content is being consumed.

Zapr’s checkboxes

Today, ratings provided by the television audience measurement agencies like BARC (short for Broadcast Audience Research Council) and TAM Media Research have relatively small sample sizes and are extrapolated to approximate for the entire user base. These kinds of extrapolations can come with its own set of problems. In comparison, the apps the ACR tech is deployed with have a much wider spread and also can assess consumption in real time.

“We have moved from consuming content from a single screen or TV in a household to multiple screens including television, mobile phones, computers etc in the same household and this had put up new challenges as well and new ways to push content to consumers and these new tech platforms are pitching to solve this problem,” said Vengalil, who also feels that these new methods also come with its own set of new challenges and problems.

Mondal pointed to the same and explained that unlike in more mature data markets like the US, where the industry has access to multiple sources of data – industry bodies, set-top boxes, ACR tech – to make data-backed informed decisions, that ability is lacking in India so far. “Why is there no good source of data that can really help the industry, empower the industry to take a data-driven decision and that was the genesis for the idea,” he added.

Zapr’s service consists of two platforms Zapr Insights, the analytics and consultative part of the company, and Zapr Engage, which helps clients with cross-device engagement and also help brands with media plans across television and digital. “You have no way of identifying, if I have a TV campaign running and have a YouTube campaign running, what is the duplication? If you don’t know that, then there is massive wastage happening,” explained Mondal.

For now, the company is only focused on the Android platform because of its reach in India, added Mondal, in response to a question why Zapr was not available on iOS, which is known to be more stringent with the way it monitors apps.

Mondal emphasised that Zapr is doing its best to ensure that concerns about privacy and consent are attended to as different from our experience with installing apps with the company ACR tech. Its agreement with partners talks of explicit consent before end-users, Mondal insisted. “Our business development team that has worked to onboard a lot of these clients are working with them (partners) on a regular basis.”

Zapr in the process of getting an external privacy and security audit done, which Mondal said will be complete by the end of the year; he did not name the agency doing the audit.

He detailed the procedure: “In terms of working with our publishers, mandating that there is language in the Google Play or the app store page describing what permissions are being asked for and why are they being asked for. Ensuring that the language isn’t hidden somewhere in line 342 of this massive terms and conditions that no one ever reads. So, we are very confident that when we say that our data is from users who are explicitly opted in, (it) is because we have a lot of these checks which ensure that we have partners who abide by what we believe is in their best interest.” The opt-in rate is between 50% to 80% based on the category of the app, he added.

Mondal added that Zapr stops analysing data from partners that don’t adhere to the privacy and consent terms the company has put in place. However, not too many cases have occurred where they had to do this with publishers.

On being asked whether he thinks users are aware that they are opting into being monitored and profiled by the apps, Mondal said: “I don’t know. I think as technologists we have to try our best to be as transparent as possible and not hide things behind legalities and jargon. I think it’s a process. I think all the discussions we are having today are helping us get there.”

Not illegal, still…

Evidently, there is a lot of cleaning up to be done even if legally, as Justice Srikrishna said above, there is little to hold against Zapr or its app partners. What is worrisome is that there is no way to find out how many of the apps installed on your device is monitoring and profiling you using Zapr’s technology. The only way to do so is to go through every fine detail in the pop-up, the app details and description in the app store, and the app’s privacy policy or ensuring that you carefully read and understand every pop-up on the app.

Such technology could potentially turn into a danger to society at large if used by, say, political parties, who are getting savvier the world over to deploy analytics while micro-targeting voters. It undermines the secret ballot in a democracy, Thejesh pointed out. “You are watching Leader #1 and you are a big fan of him but you don’t have to tell anyone because nobody will know. Now, these phones are giving away that information which you thought was private to you. They can prove that you are a big fan of Leader #1 because you are listening to them 50 times a day. Which is a huge privacy breach, that is what actually kind of scared me,” he said.

At this stage, if you realise that you have opted in by mistake and want out of being monitored and profiled, what do you do? There is no straight answer to this, either.

In almost all the cases we encountered, the details were buried in the app description or privacy policy which involved users having to visit a page on Zapr’s website and enter their Advertising ID — not a straightforward method. (The Advertising ID provided by Google Play services is a powerful tool for advertisers to help identify a potential customer. This article has a detailed description of how you are being tracked by Google and how to control the extent of tracking an even switch it off.)

General monitoring for content curation and personalization is a fair practice in some cases but with these apps and companies, it is difficult to identify the extent up to which a user’s content consumption habits are monitored and profiled. Without a data framework or policy in India, it is very difficult to find how and where you are being tracked and by whom because there is no.

“Currently there are no regulations to oversee how this data is being captured and used but with GDPR and India planning its own data regulation framework, it might become difficult for these apps to collect data,” said Vengalil. (GDPR is short for the European Union’s General Data Protection Regulation.)

A 2016 study titled ‘The Biggest Lie on the Internet’ conducted by Jonathan Obar, who teaches communication technology at York University, and Anne Oeldorf-Hirsch, assistant professor of communications at University of Connecticut, found that most users on the internet don’t bother with reading the long terms of service and privacy policies found on the internet. The case hasn’t changed much in two years and companies and services still bury important information and disclosures deep within the description, terms of service or privacy policies.

---