Deepak Agarwal dreams of a day when people would have a career coach just like Siri or Alexa. That day may be far out but the 44-year-old chief of artificial intelligence at LinkedIn is doing things that might just make it possible someday.

“When you come to Linkedin and say what you really are and what you want to become, our aspiration is to create a career coach for you,” Agarwal told FactorDaily in a recent interview.

Agarwal, a statistician with roots in Kolkata, now heads one of the most advanced teams (350-strong) working on AI and machine learning (ML) in the world. The team is baking AI into products while keeping an eye on the future.

Take, for instance, the personalisation of the ‘jobs you may be interested in’ feature: using AI and ML increased the number of job applications by 30%. It has also improved the click-through rates, cut down spam, and improved response rates for InMails.

The team also wants to build conversational systems that can improve a user’s experience on LinkedIn. In our conversation, we also spoke about the work that goes into weaving AI into the fabric of a large enterprise like LinkedIn. And, for those who are starting out, there’s some sound career advice in the interview. Edited excerpts:

How does LinkedIn use AI?

There is hardly any experience that doesn’t use AI at LinkedIn today. We often refer to it as the oxygen of all products. Whether you look at our consumer app or connecting you to the right people, all the recommendations are now through AI and machine learning. If you are coming to LinkedIn and getting engaged with the newsfeed, all the recommendations on the newsfeed use AI and ML. Instead of just being reactive (where we engage you when you come to the site), we are also more proactive. So, if there is some information that I should definitely share with you because it’s very relevant, then we also help you get that information more proactively by sending notifications. The red dot on the app reminds you there’s something very important. We don’t do that very often. We only do it when there’s something relevant.

If you’re looking at job recommendations, that’s obviously powered through AI and ML. It’s hard to do that manually because you have tens of millions of jobs. How do you do this matchmaking? Then, there are all the search products. People search on the app. People search for recruiters who are trying to source the right candidate. They have certain criteria in mind based on the job description and all their conversations with the hiring manager but it is still very open-ended. And, so, how do you actually use the search engine, type in the right criteria, and we surface the right candidate that you are likely to find relevant to you? But at the same time, we are also optimising the system to make sure that you’re shown the right jobs. If you are a salesperson, then we help you connect with the right decision makers. If you are looking to upskill yourself in a certain skill, then we recommend the right courses.

We have also been using AI in other areas like how do you keep the site safe? That is something Rushi (Bhatt) talked to you about. Even things like fake accounts, scraping… these are all the areas where we have been actively using a lot of AI to scale ourselves better. Otherwise, it will take a huge army of human beings to do that. And that job is not even fun for anyone. AI is pretty mature at LinkedIn. I would say it’s the state of the art and in some areas, we are actually setting the state of the art like we are defining what state of the art is. Some of the other companies have been adopting those technologies.

AI and ML has been at LinkedIn since 2007, this is before I joined. It was not as extensive as it is today and because the company was a pre-IPO startup. In order to deliver value to these data assets, we have to glean insights from data and that was a core component of the platform right from the very beginning. Over time, it really grew.

What was AI before 2007 at Linkedin? 



This was before my time but the first few successful products were the ‘people you may know’ or people search products. I joined in 2012, so I don’t know all the history before that. After I joined, the newsfeed, advertising, recruiting, job recommendation, these all started using AI and ML.

In 2012 when you joined, the AI hype hadn’t fully caught on. What was your charter like?

The charter was to slowly centralise all ML and AI we do at LinkedIn and strengthen the team. The team is around 350 people now. When I joined, there were multiple teams doing machine learning. The size of the team was 50 at that time. That’s primarily due to the impact the team has had. LinkedIn has also grown over that time. So, the RoI you get out of these efforts grows with the scale of the platform. In 2012, we had less than a hundred million members. Now we have half a billion members, there’s a lot more activity on the site. It’s not like the company didn’t believe in AI at that time. More like, whatever investment we had was adequate.

But as we grew, as we introduced new products. For instance, we didn’t have native ads at that time. We didn’t have sales navigator at the time. As we even made more and more investment, we knew that these areas can only succeed if we have machine learning and AI built into it and we continue to invest in that.

How does the team function? How do you interface with other teams?

We have a centralised organisation and almost two-thirds of the organisation is product-facing. So they are embedded in the product and product engineering. At Linkedin, we have functional units and then we all get together to build the product. One-third of the team is the platforms team. They are building machine learning capabilities, libraries, platforms that all the other teams can use. We don’t want every product team to build their own platform. We have one platform that can be used by all of them. Of that one third, there’s also the team that’s building the professional knowledge graph. If you look at the total number of ways users have described their titles, there are 160 million variations. So we standardise all of that to a canonical taxonomy of 24,000. That structure we add is used by all downstream applications. For instance, if you are posing a job on LinkedIn and you’re looking for the title, we’ll show you suggestions and that’s all coming from that.

How does the AI academy work?

In the last few years, our teams have grown and now we’re slowly realizing that in order to scale ourselves beyond what we do right now, we need more people with machine learning expertise. We find it difficult to hire that many people in ML. There aren’t that many in the market.

Is it because of the talent grab?

It’s both. So the demand is creating a lot more scarcity. If you look at 10 years (back) from today, the same process was being followed as we follow today to train the workforce. But there wasn’t as much scarcity as it is today. Every company, especially in the technology sector, realises that they need this pool of talent to add RoI to their business. So there’s more scarcity and hence it is more challenging to have that many people and we cannot wait for that long.

We said, let’s look at the anatomy of machine learning. There are many different machine learning problems we solve. Each of them has varying degrees of complexities. There are a lot of machine learning problems where the complexity is not as high as some of the problems that we solve. We realised that for those problems, we can train a very well versed software engineer and they can actually pick it up.

That’s why the AI academy was born. You will hear many companies having such efforts. We’re not training software engineers to pick up theoretical machine learning skills. For that, we still encourage them to go out and take courses. But, just taking a theoretical course like deep learning AI and getting your certificate does not mean you can do machine learning. I got my degree in 2000 and it took me 10 years to learn machine learning. You read something in textbooks and then you have an applied machine learning problem, it’s totally different. So, our goal is to really help software engineers learn that. Our goal is to provide them with more vocational training which can help them improve the systems that they are responsible for today by infusing more AI and ML. So you’re already running that system now with this new skill set, can you intervene and introduce new variations to the algorithms and deploy it and see what happens through rigorous A/B testing?

What areas are you setting the benchmarks?

There are two areas where we are leaders. One is building large-scale recommender systems. I think we have created technology that is the best in the industry. I have not seen anyone else have better technology than what we have. The other area is a more holistic optimisation. Like how do you optimise the entire ecosystem? A good example is email optimisation. I was describing earlier what reactive and proactive recommendation (is). How do you actually balance that and not spam people? We have published extensively on that. In that, we have been able to use a lot of sophisticated technology. A lot of people in the industry have approached us.

What’s next for AI and LinkedIn?

We want to keep it real. One is to make the whole end-to-end machine learning process more automated. Today it’s not. This may not sound very sexy but it (still)  actually adds a lot of value. We are training people who are not trained in machine learning to be able to deploy machine learning in our system. But if the end-to-end processes are not simplified, then it will be very hard for anybody to make them productive.

It starts from data preparation, maintenance, monitoring, all of that has a lot of manual steps today. That has a high barrier to entry because the sophistication has increased over time. Unless you take proactive measures to make this process simpler, it could be hard for them. So this is definitely one thing that we are all working at. It involves collaboration across all engineering.

The system should optimise the computations across different tasks. There’s a lot of commonality. How do you abstract that out and not do the redundant tasks? We’re working with our foundations team for model deployment. Model deployment should become like software deployment. There should be as much rigour as there is in software development. Today that’s not the case. How do we create that process that will make it as if you’re installing software? Then how do you ensure that the whole thing is running reliably? Just like we have software systems that are 99.99 % reliable. So, the same kind of disciplined approach that we have a software engineering has to come here. This is one of the top projects for LinkedIn engineering for the next few years.

We’re also working on conversational AI. In today’s world, the human-computer interaction is different. We believe in the future, in an ambient computing world, the way humans and machines interact will change and one of them will be through conversation.

Do you have plans to move to a voice interface? And probably something more futuristic later?

Could be. That remains to be seen. The core AI technology doesn’t change. As long as the speech recognition accuracy could be improved, whatever we build on top of that could still work if we solve the conversational interface. The hard part in conversation interface is not just the speech part which many people are working on. We’re not spending too much time on that. We’ll just take it from Microsoft (LinkedIn was acquired by Microsoft in June 2016). But the part we’re focusing on is how do we create this conversational back and forth that will help you accomplish the objectives you come to LinkedIn to.

For instance, if you’re a recruiter today, how do you make it easier? You’re driving in your car, talking to the machine and telling what you need and the machine through a few back and forth tells you the set of candidates you should reach out to and then you are in your office and you send them an email and you’re done. Most of the time that you save, you spend in creating a great interview experience for those candidates. Today, that time is spent sourcing candidates.

Creating that conversational interface is harder because you say something and the machine has to understand the context and respond and ask the right question. Even for human beings, it is hard. We have been investing in that area. We have gotten some early success in a very constrained set of problems. This is a long way to go.

Can you give us some examples? 


Smart replies is a good example. The machine is generating a reply but then you as a human can curate it. We’re also doing something in the help center. There are a lot of customer complaints and how do we provide an interface to our members so they get resolutions quicker.

Our aspiration is to create a career coach for you. Like you come to the Linkedin site and you tell us what you really are and what you want to become. And then we have a coach. An intelligent assistant that helps you achieve your career goals. That’s an aspiration and we are far from there. How can we create this technology where there would be an interface which will make every professional more productive and help them achieve their objective way more efficiently? That’s how you save time for professionals. That’s a big goal and it touches many aspects.

How far have you come on that journey? 


We’re just getting started. There’s a long way to go. The technology is still very early. I think that’s true for everyone. Anyone who tells you I have a conversational assistant that works very well is just lying. We all as a community have a lot of work until it becomes something that we can use on a day-to-day basis. We have Alexa… they’re very good at of the tasks. We’re trying to do something similar for our world. We’ll continue to roll out these things. To create technology that’s perfect will take time.

For someone to pursue a career in AI or ML, what’s the right way of looking at it?

It’s a very interdisciplinary space. People from various disciplines are approaching problems in different ways and there’s systems engineering aspect it, a math aspect to it, a data aspect to it. Different people can start from different areas but I believe you need to have some core strength. For instance, if you are a systems engineer, you need to have some of the basic math foundations. But at the end of the day, I think it will be good to realise that this is still more an empirical science.

You’re collecting evidence by doing empirical analysis of data and you keep accumulating data and evidence and that’s how you solve problems. That appetite is something that should inspire you and motivate you.  You need to have the love to solve applied problems through data. I would not say you need a degree in this or this or this. The field is going to change a lot.

How do you avoid algorithmic bias? Because humans have a lot of bias. People discriminate when it comes to hiring all the time. You don’t want that to creep into your algorithms.

So, there are two kinds of biases: good biases and bad biases. We want to reduce the bad bias. I think it’s very difficult to create something which is unbiased. Whenever you have an algorithm that collects data, every data point you collect is a biased opinion. Machine learning is about taking all these opinions and then doing the best you can. Completely eliminating bias is not possible, in my opinion. What we can do as what we do at LinkedIn is figure out what are the good biases and what are the bad biases and start measuring the bad biases and then start tweaking your systems to reduce the bad biases. That’s what we’ve started doing. There’s a long way to go.

Prabhakar Raghavan of Google told us about growing empathy towards algorithmic bias within enterprises. What are some of those biases you come across?

So, if you look at our recruiting product, there’s a possibility that there could be gender bias. First thing we do is to measure it and if the bias exists, we tweak our algorithms by adding some constraints into our algorithm to reduce the bias. That’s the high-level approach. And the particular approach for a particular product depends really on the details of these biases. We have a committee at Microsoft dedicated to mitigating bias. We are active participants in that. We’ve also published some reports on this.

How did you get started in the field?

I did my PhD in statistics. I was contemplating going into academia, becoming a professor. I was very close to accepting an academic position. Then, I interviewed at AT&T Labs. After my interview, I was like if I get a job here I’m not going anywhere else. Luckily, I got the job. After four years there, the telephony business was going down and at some point, Larry Page had started a lab in New York and came to Bell Labs. After he gave that talk, the brain drain started to go to Google. He was selling his architecture of using commodity pieces to do distributed computing. It sounds basic now but at that point it was big. Then, I moved to Yahoo because I used to use their mail product a lot. At some point, Yahoo Research was publishing more papers than top academic institutions in the world. But all good things have to come to an end.

How did you come to LinkedIn?

One of the key projects I worked (at Yahoo Research) was optimising their front page using ML and AI. That project, codenamed Coke, was very successful in 2007. It was the heydays of Yahoo. We were not using AI before, it was done traditionally using editorial inputs. After that project, this radically changed. Even New York Times started doing the same thing after we did this. I also worked a lot on advertising.

The executive sponsor of that project was Jeff Weiner (today LinkedIn’s CEO) and that’s how LinkedIn happened. At the time in 2012, they were slowly winding down their research. We had a new CEO and I was considering many other places but LinkedIn was top of mind. I can do what I do at many places. Like if I ended up at Netflix, I’d help people see better movies. I’m not saying that that is not a noble thing to do. But then I thought helping people find economic opportunities is more profound than actually helping people find better movies. Sooner or later you will find better movies. But finding great economic opportunity doesn’t have any impact on one individual. It was more inspiring.

Tell us more about your journey.

I was born and brought up in Calcutta (Kolkata today). Did my undergrad in statistics from the Presidency College in Calcutta. That’s how I got interested in data. The idea of using data to quantify uncertainty and help make decisions was fundamentally fascinating. My initial inclination was to become a physicist. But then I learned about statistics. It was not mainstream, but I just found it very fascinating that you can do something like this. After undergrad, I went to the Indian Statistical Institute.

Then I came to the States to do my PhD. The education in India gave me a very strong mathematical foundation. In those days, the computing power was not readily available in India. I remember we just had one mainframe computer and there was like this two hours in a day when you can access that. After coming to the States, compute power was more readily accessible and that’s when I started to do applied statistics. During my thesis, I worked with ecologists to study deforestation patterns in Madagascar. That was beautiful. That’s how I got interested in doing statistical modelling with big data.

It had satellite imagery, topographic data, geographic information systems, survey maps and not only was the data big, it was also very heterogeneous. The goal of the thesis was how to combine all of this data sources and make sense of how deforestation happens in Madagascar and provide pointers to policymakers to prevent that from happening. It was a lot of fun. That’s when I decided my next job would be to do statistical modelling for big data. That time, only two places had big data and were interested in modelling. One was AT&T with all the call graph data. The other was NASA. I didn’t get a job there because I was not a US citizen. At AT&T, I really picked up computer science. Then, I went to Yahoo.

---