The 7 biggest big data pitfalls

Many big data projects stagnate

The potential benefits of big data are great. It can give new insights into consumer behaviour, show where your organization can work more efficiently, predict future changes and much more. But many companies plunge into big data without being sufficiently prepared and so easily fall into one of the many pitfalls. Gartner predicted in 2015 that about 60% of big data projects would not go beyond piloting and experimentation. How do you make sure your organization does not fall within that large group? These are the seven most common pitfalls.

The seven pitfalls and how to avoid them

A summary before the explanation.

1. Not having a clear application in mind

Many companies want to get started with big data analytics right away without a clear goal. They follow the issues of the day and go along with the hype. But collecting data indiscriminately without knowing what you want to do with it is a good way to waste a lot of money and not get much further. Before you put time, money and resources into big data, think about what you want to do with that data. Which decisions do you want to take faster, what do you want to be able to do more efficiently, which consumer behaviour are you looking for? If you know what you want to achieve, you know which data you want to record and how to use it. Only then do you achieve visible results.

The city of Dublin, for example, collected traffic data with the aim of improving traffic and safety in the city, by placing sensors on roads and in buses. And Safety Region Noord-Holland Noord managed to save twenty minutes in emergency care for heart patients, where every second counts. Data can even save lives.

2. Looking only at your own big data

Many companies do not look beyond the data they collect themselves, but analyzing data from other sources can have great added value. Data from social media, web content and data providers can supplement the internal picture with useful data from external sources and so bring new insights. Look not only within your own company and industry, but also outside them. If, as a shopkeeper, you only look at the products you sell yourself and not at what the competition sells, you probably miss crucial insight into consumer behaviour. That behaviour changes faster than ever, and to remain competitive you have to be agile and able to anticipate changes in the market.

Besides your own data and that of other sources there is also so-called zero data: the data that is not (yet) recorded. Sometimes the answer you are looking for lies in a corner you have not thought of. By staring blindly at the data you do have, you miss the chance to collect new data that can provide the missing insight. So keep looking for new angles for relevant data.

3. Not using the right tools

Many organizations still misuse spreadsheets as a data warehouse. A lot of time and money is lost on this, because a spreadsheet is anything but efficient for that purpose. Most spreadsheets contain errors, and it takes a lot of time to fish them out. Whole meetings are lost on interpreting the figures and checking their correctness, while data should prevent precisely that kind of problem. It is about facts, not feelings. So choose BI tools that suit your purposes.

4. Flouting privacy and other regulations

The use of big data brings many advantages, but also many temptations. Data has to be handled ethically and with respect for privacy, and for that reason all kinds of laws and rules are attached to the use and storage of data, such as the GDPR. Handle the data carefully and make sure you meet all requirements. Do you have the right to collect and use the data? Is the data anonymous? May you sell the data?

A fine example is Nestlé Germany, which already worked so carefully that it stayed ahead of the GDPR and did not have to take extra steps to change its policy. You do not have to seek out every limit: not everything that is legally permitted is desirable. And paying hackers to keep a leak of personal data quiet is simply perverse.

5. Ignoring confirmation bias

Sometimes you would like to see your desired result during an investigation and you only look at data that supports it. This is called confirmation bias: you (unconsciously) favour the data that supports your opinion or result and exclude the rest. It is better to carry out research without wanting to reach a particular result. Most professional data analyses are carried out by trying to reject a hypothesis on the basis of a so-called null hypothesis, instead of proving it. The use of data is precisely about taking decisions based on facts and information, and not on gut feelings and opinions.

6. Using a sample that is too small

If you analyze a small sample of data, there is a good chance that you will draw the wrong conclusions. A small sample is often not representative of the whole. It is usually impossible to say whether a value is a statistical outlier, and therefore whether you should include it in the analysis. If you extrapolate the results of a small sample to a much larger data set, the results are likely to be misleading. So always include enough data in your analysis.

An example: there is a car park with five cars in five colours: blue, red, green, black and white. Based on this information you could conclude that each of these colours has a 20% chance of occurring, and that purple cars, for example, do not exist. With a little common sense you know that this is nonsense, but the same principle applies to other small data sets.

7. Not having the right people in house

You have the infrastructure, you have the data, but you do not have the right data specialists. If you build a data lake, for example, it is important to bring in the right expertise to interpret and catalogue the data, and to make sure that the quality of the data remains guaranteed.

The data scientist in particular is a much sought-after role, but make sure you look for a real person and not a jack of all trades who will solve all your BI problems single-handedly. The trick is to work with teams whose members complement each other in the necessary areas, and that are agile enough to solve problems.

Further reading

• Big data
• Blog: The 20 best tips for pure decision making
• Blog: The ethics of big data and AI: temptation and fear
• Blog: 6 reasons not to build a data lake
• Blog: When every second counts: time is muscle

About the author

Jack Esselink is a very experienced trainer and speaker on data and AI. After studying business economics he worked at several software companies, including Cognos and IBM, most recently as Data Science & Machine Learning Evangelist at IBM, giving training courses, workshops and keynotes worldwide.

About Diamond Decisions

Diamond Decisions helps organizations become better by enabling better decisions. We don’t start with data, dashboards or AI, but with the decision that needs to improve, and work back from there to KPIs, data and technology. We do that in four steps: inspiration and alignment, business modeling, analytics and data platform and data-driven decision makers. That way BI and AI don’t remain an IT project, but deliver value every day.