On May 9, Netflix launched its own research website. This highlights the focus Netflix has on Deep Learning and Data Science. The site is extremely well designed showing vertical classification of the different areas that Netflix research works on along with the horizontal business areas where Data Science is deployed at Netflix. It has some great articles with everything from video encoding to A/B testing where they use Data Science. I found the website to be very comprehensive making it a go-to destination for things Netflix Data Science from different verticals to jobs.
MyAI Interview Questions articles for Microsoft, Google, Amazon, Apple, Facebook, Salesforce, Uber, LinkedIn have been very helpful to the readers. As a followup, the next couple of articles were on how to prepare for these interviews split into two parts, Part 1and Part 2. If you want to find suggestions on how to showcase your AI work please visit Acing AI Portfolios. Career Insights check out the interview I did with Jesse. Now onto the Netflix Data Science Questions article…
To maximize the impact of their research, Netflix does not centralize research into a separate organization. Instead, they have many teams that pursue research in collaboration with business teams, engineering teams, and other researchers. From our publications, we can deduce that they are focused on the applied side of the research spectrum, though they do pursue fundamental research and think that has the potential for high impacts, such as improving our understanding of causality in our data and systems.
Netflix moves quite fast. There is one phone interview with the recruiter and another detailed one with the hiring manager. There are two onsite interviews with around 4 people first time (data scientists/engineers) and 3 people (higher level execs) a second time. There is a mix of product, business, analytical and statistical questions. Statistical questions mostly revolve around A/B testing: hypothesis testing. There are a couple of SQL questions too. Analytical questions usually include a hypothetical problem to analyze and metrics to evaluate product performance. Higher level executives mostly focus on background and past experience.
- Netflix Research Blog: All Articles
- Deep Learning for Recommender Systems: Talk Slides
- Reliable ML in the Wild Workshop (ICML 2017): Making ML Reliable at Netflix
AI/Data Science Related Questions
- How would you build and test a metric to compare two user’s ranked lists of movie/tv show preferences?How best to select a representative sample of search queries from 5 million?
- Given a month’s worth of login data from Netflix such as account_id, device_id, and metadata concerning payments, how would you detect fraud? (identity theft, payment fraud, etc.)
- How would you handle NULLs when querying a data set? Are there any other ways?
- What is the use of regularization?What are the differences between L1 and L2 regularization, why don’t people use L0.5 regularization for instance?
- SQL queries to find time difference between two events given a certain condition.
- Given a single day with a large sample size and a significant test result, would you end the experiment?
- What do you know about A/B testing in the context of streaming?
- How do you prevent overfitting and complexity of a model? How do you measure and compare models?
- How do you know if one algorithm is better than other?
- Elaborate on the recent project you developed for your company.
- Why do you use XYZ method? Elaborate on how to improve content optimization?
- What technology or item that most people feel will be obsolete in the future do you not agree with?
- Why Rectified Linear Unit is a good activation function?
- How should we approach attribution modelling to measure marketing effectiveness?
- How would you determine if the price of a Netflix subscription is truly the deciding factor for a consumer?
- If Netflix is looking to expand its presence in Asia, what are some factors that you can use to evaluate the size of the Asia market, and what can Netflix do to capture this market?
- Say the CEO stops by your desk and asks you whether or not we should go into an untapped market. How would you determine the size of the addressable market and the factors the Netflix should consider before deciding to enter the market?
Reflecting on the Questions
The data around Netflix questions is sparse. The high level questions resolve around A/B testing, recommender systems and foundational knowledge questions around regularization and activation functions. This is different from the other companies we have looked at previously where focus was more foundational. All job openings are usually senior level. Good experience combined with good preparation can surely land you a job at the largest international evergreen content cinema in the world.
Consumable List: Netflix Data Science Interview Questions
This article was also featured on KDnuggets: https://www.kdnuggets.com/2018/06/netflix-data-science-interview-questions-acing-the-ai-interview.html
The AIconics are the world’s only independently-judged awards celebrating the drive, innovation and hard work in the international AI Community. A panel of 20 judges from around the world thoroughly reviewed competitive entries from the leading innovators in the AI Space.
Here are the 2018 winners:
Best Application of AI for Sales & Marketing
RTB House is a global company that provides state-of-the-art retargeting technology for top brands worldwide. Its proprietary ad buying engine is the first and only in the world to be powered entirely by deep learning algorithms, enabling advertisers to generate outstanding results and reach their short, mid and long-term goals.
Best Innovation in AI Hardware
Pure Storage helps innovators build a better world with data. Pure's data solutions enable SaaS companies, cloud service providers, and enterprise and public sector customers to deliver real-time, secure data to power their mission-critical production, DevOps, and modern analytics environments in a multi-cloud environment.
Best Innovation in Robotic Process Automation
UiPath provides a complete software platform helping organizations automate business processes. The company's mission is to eradicate tedious, repetitive tasks and let software robots do the grunt work. They enable businesses and organizations to develop an agile digital workforce by providing a state-of-the-art platform for software robots orchestration. Their products automate across all internal or web-based applications/databases and have unmatched solutions for Citrix, SAP and BPO automation.
Best Application of AI in Financial Services
WorkFusion's Intelligent Automation empowers enterprise operations to digitize. WorkFusion combines robotic process automation (RPA), AI-powered cognitive automation, workflow, intelligent conversational agents, crowdsourcing and analytics into enterprise-grade products purpose-built for operations professionals. These capabilities let enterprise leaders digitize their operation, exponentially increasing productivity and improving service delivery.
Best Intelligent Assistant Innovation
Artificial Solutions® is the leading specialist in Natural Language Interaction (NLI), a form of Artificial Intelligence that allows people to converse with applications and electronic devices in free-format, natural language, using speech, text, touch or gesture. Delivered through Teneo® – an ultra-rapid NLI development and analytics platform – it allows business users and developers to collaborate on creating sophisticated, humanlike natural language applications in record time without the need for specialist linguistic skills.
Best Innovation in Deep Learning
IBM Watson Studio
IBM’s experiment-centric deep learning service within Watson Studio allows data scientists to visually design their neural networks and scale out their training runs while auto-allocation means paying only for the resources utilized. Watson Studio accelerates the machine and deep learning workflows required to infuse AI into your business to drive innovation. It provides a suite of tools for data scientists, application developers and subject matter experts to collaboratively and easily work with data. Build, train, deploy and manage AI models at scale, and prepare and analyze data, in a single, integrated environment.
Best Innovation in Natural Language Processing
Clarabridge helps hundreds of the world's leading brands understand and improve their customer experience. Using advanced text analytics, Clarabridge transforms survey, social, voice and all other forms of customer feedback into intelligence used to empower confident, decisive action across the business.
Best AI Start-Up
Luminance helps lawyers to categorise, review and analyse thousands of documents at speeds no human can match. With AI taking the burden of low-level cognitive tasks – like those common in due diligence, compliance, insurance or in-house contract management – lawyers can optimise their practice, working smarter, faster and more effectively.
Best Application of AI in the Enterprise
LiveTiles is defining the market for the intelligent workplace by giving developers and business users tools to easily create dashboards, employee portals, and corporate intranets that can be further enhanced by artificial intelligence and analytics features.
Best AI Application in Healthcare
Aifred Health is using AI techniques to develop a clinical decision aid for physicians to improve treatment efficacy for individuals suffering from depression. The foundation of this clinical tool is the data collected by researchers investigating patient response to depression treatments in clinical trials .
Artificial Intelligence and IT Service Management
As AI transforms the workplace, IT Service Management will both enable this process as well as be transformed itself through the new efficiencies coming in.
Not just the stuff of sci-fi films anymore, AI is transforming technology in people’s daily lives, Amazon’s Alexa and AmazonGo being important examples. According to a Gartner report, there are three key requirements that define AI:
It needs to be able to adapt its behavior based on experience.
It needs to be able to learn without being solely dependent on human instructions.
It needs to be able to come up with unanticipated results.
Based on these criteria, the AI that we deal with regularly, like Siri and Alexa, are examples of ‘weak AI’, as they are built to accomplish very specific tasks. ‘Strong AI’ or ‘General AI’, which is the end game, is still a distant dream.
Nonetheless, weak (or narrow) AI by itself packs in enough exciting potential to revolutionize the workplace. The opportunities it presents can, however, be squandered if its deployment is done without proper human governance. This governance will largely fall into the hands of corporate IT Service Management (ITSM) teams.
AI’s role in the ITSM revolution
At the same time, AI has the potential to transform ITSM too, allowing staff to delegate mundane tasks to AI software and focus on more strategic issues. A learning, conversational AI experience will be critical for AI technology to succeed, and will revolutionize ITSM in the following key ways:
An AI-automated front line
Currently, risk and uncertainty is rife in old-style self-service portals, making companies reluctant to divert their human ITSM frontline resources away from basic phone handling. AI-enabled chatbots can help develop automated ITSM solutions that are better at customer query interpretation, assistance without human intervention and providing a personalized end-user experience.
Good ITSM operates many vital ‘back-end’ processes like incident management and change management that keep IT systems running. AI can not only make this process more efficient – for instance, when connected to IoT devices it can be notified instantly if a smart device starts malfunctioning, without the end-user having to report it – but also make business aware of ITSM’s important place as an enabler.
All knowing AI
AI-powered ITSM can efficiently handle large volumes of data and decipher patterns, resulting in real-time insights, predictions of problems and recommendations to fix them. Plus, it can source answers to difficult queries from across the Internet as well as pool data from multiple organizations to provide better solutions.
Ultimately, humans will remain vital for delivering AI-enhanced IT services. AI’s rapid evolution will allow it to work alongside humans to create a more efficient workplace. It will also allow IT staff to become business enablers and productivity transformers, while technology does the heavy lifting.
Healthcare’s AI Market to hit $6.6 bn by 2021, says Accenture
The projected 1000% increase in the healthcare AI market from 2014 to 2021 is driven by factors like greater consumer acceptance, change in healthcare models and rapid increase in investments.
A new Accenture report, which looked at investments, revenue growth and acquisitions in the AI space, predicts that the healthcare AI market will reach $6.6 billion by 2021 from just $600 million in 2014.
The rapid growth is driven by factors like the move to population health, and greater acceptance of machines in healthcare delivery by consumers, driven by experiences in other services.
Accenture’s survey of over 3000 consumers shows that one in five US patients have already used AI-powered healthcare services, like robots, virtual clinicians, and home-based diagnostics.
In keeping with the move in the wider healthcare industry from volume of care to value-based models, business and venture funds are funding development of products and systems based on AI.
Examples: Use of AI by Anthem and Cigna to curtail opioid addiction; funding by Optum Ventures of the startup Buoy Health that has developed an AI-powered digital health assistant that helps patients better understand symptoms and advises on next steps.
Regulators are stepping up approvals of AI systems and related products.
Example: Approval of three of the seven robotics products developed by Bionik Laboratories, a venture whose solutions have been used for treating neurological disorders in over 200 hospitals in 20 countries.
The future of AI in healthcare
According to Bionik’s CEO, Eric Dusseux, there will be a steady evolution of AI both in the medical industry and beyond. Humans have long relied on technology to improve efficiency, productivity and process quality. Innovations like AI, machine learning and brain-computer interfaces will encourage continued use of technology in the medical space to further optimize patient treatment and care.