Have you ever sat and wondered why your project failed while your competition is yet again described as another AI success? Or sighed in despair when another project landed in a trash bin or was put in a drawer?

TechRepublic and Gartner both claim that approximately 85% of AI projects fail and the reasons include budget, timing and not delivering on goals. But what does it actually mean?

In this article, I’m:

1. exploring 3 areas where we make most of mistakes leading to AI failure:
   1. Data
   2. Planning
   3. Management/execution
2. and what we can do differently.

 

Data – AI failure no. 1 reason

 

When it comes to data, the failure is often related both to quantity and quality. 

 

Quantity

 

When we have a large dataset, we have a possibility to (even if we don’t have to) use more various solutions than with a smaller size one. But it’s not only that.

Digging through the variety of samples may also help us in finding the potential edge cases for our algorithm. With that, we can understand and define how it should behave in the unknown situation and understand how unknown situations can influence our model.

Why are unknown cases important? We want our models to be robust, meaning behave as expected even if the conditions are not the ones that we assumed or knew. A good example are named entities. When we train a system to translate voice to text, we can’t possibly show how every person pronounces names of their friends, family or places they visit. Especially when we use many languages. But we still want it to work.

How can you fix the lack of data? Of course the solution should be case-by-case, but some that are used are: 

1. augmentation – modifying data we have in order to expand it,
2. generation – using data to generate other cases.

 

Quality

 

More crucial problem is data quality. A single case I saw most often was that the team did not understand the process of collecting data. Such information is invaluable. It can give you a hint about potential biases in the process itself and what may be missing.

It can lead you to understand how the data should be cleaned, processed and potentially supplemented with other examples. This is the first thing you should do before you even touch the data. It could’ve changed a lot of projects like Amazon hiring bot or teacher grading in the US.

We also have to stop attaching ourselves to data just because it’s data – not every example is a good example, some may be broken or biased. We can also use explainability methods to find potential bias sources in data by producing a biased model.

 

Both?

 

There is however a third way – a data-centric AI. It allows humans to work in a loop with algorithms being created. But it’s a topic for a whole new article and will be covered in the next one.

 

Fast track to AI success: Planning

 

Apart from data, planning is in my opinion the second most crucial step of an AI project.

 

Explore vs. Exploit

 

The exploration project is the one in which we try to create something totally new or we want to explore our knowledge about the world or about possible solutions. The exploitation one is related to the things we already know and the area that is familiar to us. 

The example of an exploration project on a global, humanity scale, may be creating an artificial general intelligence (AGI) whereas the exploitation project may be classification of a handwritten digit from MNIST testing dataset. 

But we can also understand exploration vs. exploitation in terms of the technological stack or domain experience we are familiar with. Exploration projects have higher risk of failure, but we still tend to choose them because of the higher benefit when we succeed. If you’re not yet sure if you are able to deliver a result you’re wishing for – choose an exploitative project or try to shift as many exploratory tasks into exploitative ones.

 

Performance

 

As a tech person I often tended to focus on mathematical metric for the project, so the one which can be directly optimized by my model. But this led me to forget about the actual business metric that I am optimizing. 

The project is not meant for us to shine, the project is about finding a solution to a well-defined problem that we or someone is putting in front of us. Define both mathematical and business metrics, and try to see how they interact with each other. I gave a real life example of what is the difference between them in one of my tweets: click here to see it. 

 

 

 

Understanding

 

Last but not least, we can’t plan a project well if we can’t understand the problem itself. Our task as a Data Scientist is of course to help others in execution of a project, but also to listen to them carefully, to what their needs are. 

We can be distracted by a technological stack we have and not be able to adjust the solution to the requirements. In the planning stage, we need to understand how the technology will influence the end users, how they will use it. We also need to simplify our thoughts as much as possible to make the process most straightforward to every person who participates in it, no matter the experience. If you’re interested, I wrote a whole article about how to communicate better in Data Science.

 

Management and Execution – from AI failure to AI success

 

I’ll discuss 3 issues here:

1. management techniques,
2. communication,
3. automation.

 

Management Techniques

 

Because AI projects rely heavily on programming and delivery, we tend to approach them in a similar way as an IT project, like for example website development. However, the main difference between them is that in the initial stage of an AI project, which is a more exploratory one, we don’t often have a possibility to set up measurable results and we can’t really plan further into the future. 

This is why a classical waterfall or agile techniques may fail. And this is why it is better to adapt a technique during a project when we move from exploration to delivery, by firstly having broad tasks restricted by time and move into more defined tasks restricted by performance.

 

Automation

 

In AI projects we often tend to make everything custom and build everything from scratch to show a spectrum of tools that we know. But actually the sooner we start automating some processes in model and data development, the better. 

For example if we are still collecting data at the same time we are performing model experiments, we can automate data changes tracking using tools like DVC to be sure we know which data were used for training for a given model. We can also automate model search using AutoML methods, and performance report generation using simple continuous integration in our repository.

 

Communication

 

Finally I can’t stress enough how important communication is between the technical part of the team and the manager. We need honesty and not just blindly follow the tasks that are given. 

Some situations where you have to raise your concerns are: 

1. a potential of a large technical debt in the future, 
2. a plan to create a solution out of multiple elements, all having high risk of failure,
3. when you don’t understand what the aim of delivering a particular task is.

Very often a 3rd one is uniquely important, because only when all team members understand a direction we want to pursue, can they equally contribute and cooperate on the project as partners.

 

Conclusions

 

If you reached this point – I applaud you. 

The solutions I presented here are not one-size-fits-all fixes. Many of those won’t be applicable in all, very complex problems. 

But I believe having awareness may stop us before taking a step off the cliff with our AI project. After all, not every problem requires an AI solution.

 

Sources

[1] Alison DeNisco Rayome, Why 85% of AI projects fail