An Attempt to Overcome the Challenge of Measuring Progress in AI

Measuring the progress of AI can be complicated, but Stanford’s AI Index is a good start.

The AI Index

Predicting when new developments in AI will take place has been famously difficult, but The AI Index from Stanford is an attempt to measure this, on some parameters at least.

The Approach – Measuring Hype or Actual Progress?

The AI Index tries to aggregate data across several ‘volume of activity’ metrics, like VC investments, academic conference attendance, papers published, etc. It also tracks creation of AI-related software at Github, interest in machine learning packages, and sentiment of AI-related news articles. But, this covers AI hype as much as it does progress, and the two might not always be correlated. Hype can also be cyclical in nature. To remedy this, the AI Index uses another metric.

Assessment of Progress of AI on Tasks

Measuring performance of AI systems on narrow tasks is useful. For instance, performance of computer vision in image annotation (great) or answering questions about images (not so great). But, it’s also very easy to measure – devise a metric that can be easily calculated, create a competition with a scoring system, or just compare new software with the old version.

It becomes more difficult to map narrow-task performances onto general intelligence. Computers are superhuman at chess now, or even Go, but does it mean we are any closer to general intelligence? The AI Index doesn’t attempt to offer a timeline for general intelligence because no one really knows how to measure progress. What it can do though is track the specialized performance of algorithms on tasks previously reserved for humans, like predicting skin cancer better than dermatologists. This shows that progress in AI over the next few years is likely to resemble a gradual rising tide, rather than a tsunami of general intelligence breakthrough.

Ethics of AI

Another challenge faced by the AI Index is to identify success measures by AI’s impact on people’s lives. These include the interactions between humans and AI systems; our ability to program values, ethics and oversight into these systems; and society’s flexibility in adapting to AI trends.

Conclusion

AI progress is a race for which we don’t know the endpoint or how to get there. This makes measuring it a daunting task. But the AI Index, as an annual collection of relevant information, is a good start.

Download the AI Index report at AIIndex.org


Leave A Comment
LATEST | POPULAR
QUANTIFIED SKIN™ - COGNIFYING SKIN CARE

Powered by Deep Learning. Our platform analyzes the contextual and behavioral signals of users from images and sensor data.

SNAPS - AUTOMATED CHATBOT

Snaps Helps Brands Deflect Tickets, Drive More Conversions & Delight Customers Through Proactive And Always-On Customer Service Automation.

DATMO - PLATFORM FOR SEAMLESS SCALING

Datmo tools help power up your existing model workflow. A new standard built by engineers, for engineers.

BIZTEXTER - CLOUD TEXTING

BUSINESS TEXTING A.I. SMS CHATBOTS

FACTMATA.COM - CREDIBLE MEDIA CHECKER

A crowdsourced community platform powered by advanced machine learning for tackling fake news and misleading content on the web. Sign up with Email to stay up to date on our plans.

Keep updated on all things AI

Sign up to be able to Star, save and post new products and services.

Sign up