
What is data mining and how does the data mining process work?
Data mining finds patterns you would not think of yourself. Whether they are worth anything depends on the decision you want to improve.
Data mining in short
- Data mining is finding relationships, patterns and correlations in structured data with the help of machine learning, statistics and database techniques.
- The de facto standard for the process is CRISP-DM, with six steps: business understanding, data understanding, data preparation, modeling, evaluation and deployment.
- Data mining looks for patterns in data (the what); process mining looks for deviations in the business process (the why). Text mining does the same for unstructured text.
- Start with the decisions you want to improve in the primary process, not with the data sources.
- Most failed projects do not fail on technology, but on data quality, unrealistic expectations and neglected communication with users.
What is data mining?
Data mining, artificial intelligence and big data are often mentioned in the same breath. The same goes for deep learning, data science and predictive analytics. Although these concepts have a lot to do with each other, there are essential differences. To avoid confusion we use one definition:
Data mining is finding relationships, patterns and correlations in structured data with the help of machine learning, statistics and database techniques.
The ultimate goal is to gain new insights that are hidden in data. You use them to take better decisions, to optimize processes and even to innovate. A much-cited example: one fine day a father was surprised by all kinds of offers for products that women often buy during pregnancy. The supermarket knew before he did that his daughter was expecting. It shows the power of data mining and at the same time its ethical side: the retailer put the daughter in an awkward position. And without basic knowledge of statistics you quickly draw the wrong conclusions.
The building blocks of data mining
Data mining does not function in isolation. You need at least the following building blocks:

- Data sources: select and extract data from high-quality sources.
- Data warehouse: combine data from different sources and bring it together.
- Data discovery: turn the data inside out; know your data through and through.
- Data mining software: apply statistics and machine learning to the collected data.
- Data visualization: visualize the patterns and relationships so that everyone can understand them.
- Better decisions: make decisions data-driven and improve them structurally.
Although the arrow in the figure runs from bottom to top, we advise you to start with the decisions you want to improve in the primary process and not with the data sources. A clear goal gives your data mining project the right focus.
Where are the roots of data mining?
The roots go back to the 18th century, when Bayesian probability, probability theory and regression analysis made their entrance in mathematics. The term data mining as a concept only surfaced in 1990 in database circles. At first people spoke formally of knowledge discovery in databases (KDD); later the term data mining became more and more common. Given this history and its exploratory character, data discovery is a generally accepted synonym. Data science is the collective name for the data sciences in the broadest sense.
What is the difference with process mining?
Process mining and data mining are often confused. Both fall under the broad umbrella of business intelligence and analytics, and both increasingly use machine learning algorithms to discover hidden patterns, relationships and irregularities. Yet there are more differences than similarities.

- With data mining the focus is on the patterns within the data. The American chain Pizza Hut, for example, tries to discover patterns in customer behavior. Using AI techniques, the company wants to recommend pizzas on the basis of the current weather and the place where customers live or want to eat their pizza. A certain weather pattern implies a preference for a specific pizza.
- With process mining the focus is on the deviations in the business process. Why is the pizza not delivered on time, and what explains the differences in baking time? Process mining works with event logs, audit trails and timestamps.
Another difference is that data mining traditionally works with more or less static tables, while process mining can nowadays also monitor business processes in real time. With data mining chance plays a large role, while with process mining you can also analyze a predefined problem. Data mining looks for correlations and patterns; process mining looks for causal relationships. Or, as the platform TechTarget puts it: data mining is concerned with the what, the patterns themselves, while process mining seeks to answer the why.
Data mining versus process mining
The main differences side by side.
Data mining
Process mining
Focus
Patterns within the data
Deviations in the business process
Question
What is going on?
Why does it go this way?
Input
Tables with more or less static data
Event logs and timestamps
Looks for
Correlations
Causal relationships in processes
What is text mining?
Text mining, also called text analytics, is less well known than data mining. That changed with the appointment of Jan C. Scholtes as the first professor of text mining in the Netherlands. In his inaugural lecture he describes the difference. Data mining is the analysis of transaction data in relational databases, such as credit card payments, to which characteristics like date, location, age of the cardholder and salary can be attached. By combining this data you expose patterns in interests or behavior.

With text mining you discover relevant patterns and characteristics through automated analysis of unstructured information. That lets you search better, analyze deeper and surface insights that otherwise stay hidden. For some years now AI has also been able to generate text: generative AI.
The idea of data mining is simple, but the number of pitfalls is large. In essence it is an interplay of four areas: people, organization, technology and data. In all four you will find small and large hurdles.
The data mining process: 6 steps (CRISP-DM)
When you set up a data mining process you can hardly avoid the CRISP-DM standard, the Cross-Industry Standard Process for Data Mining. Since the late 1990s it has been the broadly accepted de facto standard. It is not built on an academic foundation, but grafted onto daily practice.

- Understand the business question: formulate the objectives of the project clearly and translate the requirements from a business perspective. The result is a problem definition and a preliminary plan of approach.
- Understand the data: collect the data and get thoroughly familiar with it. You recognize data quality problems, stay alert to possible biases, gain first insights and formulate hypotheses about hidden information.
- Prepare the data: work from the raw data towards a definitive dataset that serves as input for the model: selecting tables, records and attributes, and transforming and cleaning data. You repeat these tasks as needed.
- Model the data: select and apply different algorithms and calibrate the parameters towards optimal values: the training phase. Some algorithms set specific requirements for the form of the data, so you often have to return to step 3.
- Evaluate the model: validate and evaluate the model thoroughly. Are you sure you will achieve the business goals with it? Is the performance sufficient? Have you overlooked anything important? Then decide on the use of the results.
- Deploy the model in production: deployment can be as simple as a report with a prediction of whether a customer will renew the contract, or as complex as a repeatable data mining process embedded in a step of a business process. The user must understand in advance which actions are needed to actually use the model.
Deployment is generally not the end of the project. Often it is about applying live models within the decision-making processes of an organization, such as personalizing web pages in real time or repeatedly scoring new leads. One thing is certain: with a well-designed and well-filled data warehouse you reap the benefits of data mining much faster.
How Diamond Decisions applies data mining
Step 1 of CRISP-DM is where most of the value is decided. In business modeling we establish which decision the model has to improve and what a better decision is worth. The data platform covers steps 2 and 3: reliable, combined data with one set of definitions. And a model only delivers when decision makers trust it and use it in their daily work, so deployment includes the people.
9 examples of data mining
Organizations do not always advertise their use cases, because they are competitively sensitive. The most appealing example undoubtedly comes from the film classic Moneyball: data mining turned the traditional baseball world upside down. A number of classic applications in business and government:

- Retail: supermarket chains and web shops use associative techniques to uncover consumer patterns, such as which items are often bought together.
- Smart farming: in precision agriculture, weather patterns and data on crop growth and fertilization are analyzed.
- Manufacturing: why does one machine part wear much faster than others? Factories use data mining to discover errors in the production process and to predict preventive maintenance.
- Healthcare fraud: abuse and fraud in healthcare come to light when you look for deviating patterns in patient behavior, medication use and claims and compare them with the protocols.
- Quality of care: large-scale data mining and big data analytics in hospitals bring patterns in clinical pictures and hereditary diseases to light fairly quickly.
- Financial services: algorithms detect fraud by flagging deviating patterns in payment behavior and human behavior.
- Telecom: models have been used for decades to analyze calling behavior and switching behavior, in order to retain customers longer.
- Government: data mining software is used to monitor and analyze benefit patterns and tax compliance.
- Public space: governments use the software to discover patterns for crowd control, traffic regulation and crime prevention.
ICS: a textbook example of data mining
The case of International Card Services (ICS) may not be the best-known example, but it is a fine one. Every step of the process, from acquiring new customers to retaining them, and from acceptance to stimulating the use of credit cards, is underpinned by data mining algorithms that greatly increase the effectiveness of the process steps. The number of fraudulent transactions fell by no less than 50% within a year, and card usage rose by 20%. Not only ICS benefited, but the customer too. Computable reported in 2009 that ICS won an Excellent Performance Award for its use of BI.
If you work with personal data or apply advanced models, make sure you comply with privacy legislation (GDPR) and the EU AI Act.
Data mining tools and techniques
All major enterprise software suppliers, such as SAP, Oracle and IBM, offer data mining tools, as do suppliers specialized in business intelligence. You do not always have to open your wallet: you can also use open-source languages and tools such as Python and R. Some suppliers specialize in text mining or in particular techniques such as classification, clustering, regression, association and detecting outliers. Whatever you choose, recognizing patterns in datasets is central.

According to the suppliers, tools are becoming more user-friendly: users without any programming experience would also achieve appealing results, thanks to no-code or low-code and drag-and-drop functionality. Some suppliers specialize in a particular sector, such as agriculture, industry or education. Specific market knowledge can help to fathom your business problem quickly.
Five rules for selecting data mining software
Suppliers sometimes advertise with exotic terms such as “predictive data mining software” to broaden their scope. The lack of clear definitions makes the market less transparent and disrupts an orderly, objective selection process. These five rules can save you from a bad buy:
- Do not fall for sales tricks: suppliers work with basic and premium versions. Text mining, for example, may only be available in the premium version.
- Use your negotiating power: suppliers are almost always willing to give discounts, certainly for large numbers of licenses. Salespeople have targets too. Street prices always differ from list prices.
- Be critical of the number of licenses: every extra module or feature usually has a price tag, and not every user needs all functionality and plug-ins.
- Watch out for sharp offers: behind free versions there is usually a subscription model that runs via upgrades to premium versions.
- Get the most out of your investment: good documentation is important and not always in order with free versions. And you do not learn data mining from a book: practice makes perfect.
Making data mining a success: 6 tips
Many data mining projects never reach production, and that is often not due to the technology. The organizational side is ignored, communication with users is neglected, the project is purely technology-driven or management has unrealistic expectations. Usually these go together. Keep the following in mind:
- Use the CRISP-DM model and follow the six steps closely.
- Be very alert to data quality; it can make or break the project.
- Put together a multidisciplinary team with representatives of business and IT.
- Make sure the team has enough knowledge of and experience with data mining.
- Collect as much relevant data as possible; do not limit yourself to internal data.
- Select the type of algorithm that fits the problem.
Talk to the author
Herman van DellenDiamond Decisions
Discuss your question with Herman
Tell us which decision needs to improve. In a no-obligation conversation we show you where your data delivers the most.
Schedule an introduction →Step 3 of our approach
Diamond Data Analytics
A data warehouse that starts with your most important decisions and is ready for conversational AI: a working prototype within 24 hours.
Explore Data Analytics →Frequently asked questions
01
Is data mining something very different from artificial intelligence?
Broadly speaking the two are comparable and often use exactly the same algorithms and models. But AI is more comprehensive and can also analyze text, images, video and sound. You usually use data mining for structured data and text mining for unstructured data such as text.
02
What is the difference between data mining and process mining?
Data mining looks for patterns and correlations within data: the what. Process mining looks for deviations and causal relationships in business processes on the basis of event logs: the why.
03
How much data do you need for data mining?
In principle you can also get interesting patterns from very small datasets (fewer than 1,000 rows and a few columns), as long as the results are statistically significant. For really refined knowledge you need much larger datasets (more than 10,000 rows and many columns). A more extensive decision tree contains more knowledge and therefore has more value.
04
Can data mining predict?
Yes. With data mining you can classify (this is a red car) and cluster (put all red cars together), but also predict, for example the chance that a customer will buy a red car or renew a lease contract.
05
How important is good data quality?
Very important: the success of data mining stands or falls with data quality. A model trained on unreliable data produces unreliable predictions.
06
How do I find out whether there are biases in my data?
That is not easy, because many aspects can be relevant: data entry, the design of workflows, selection at the gate, KPIs with perverse incentives and the way the data is modeled. Talk to the people who enter the data, watch them work for a while and carry out a process analysis.
07
Is data mining feasible for a small or mid-sized company?
Yes, provided you start small: one recurring decision, the data you already have and a simple technique such as a decision tree or a cluster analysis. Open-source tools keep the costs low. The limiting factor is rarely the software, but the quality of the data and a clear question.
Further reading
- Neural networks, cluster analysis and linear regression: three techniques explained.
- Process mining: finding deviations in business processes.
- Data science and machine learning: the wider field.
- Data quality: the precondition for reliable models.
Sources
- Chapman, P. et al. (2000): CRISP-DM 1.0: Step-by-step data mining guide. SPSS.
- Scholtes, J.C.: Text mining: de volgende stap in zoektechnologie. Inaugural lecture, Maastricht University.
- Dutch BI Award: jury report 2009 edition (ICS).
- Business Insider: Pizza Hut is building an AI that recommends food based on your local weather.
About the author
About Diamond Decisions
Diamond Decisions helps organizations become better by enabling better decisions. We don’t start with data, dashboards or AI, but with the decision that needs to improve, and work back from there to KPIs, data and technology. We do that in four steps: inspiration and alignment, business modeling, analytics and data platform and data-driven decision makers. That way BI and AI don’t remain an IT project, but deliver value every day.

