Just like basic statistics, understanding a Data Science Lifecycle (DSLC) is foundational knowledge for Data Scientists, Data Analysts, and Data Team Managers. Here is a brief introduction to the tried and true process of CRISP-DM.

CRISP-DM stands for the CRoss Industry Standard Process for Data Mining. It is a standard process for knowledge discovery consisting of 6 phases that can be applied across a wide range of applications. The 6 phases are Business Understanding, Data Understanding, Data Preparation, Modeling, Model Evaluation, and Deployment.

According to KD Nuggets Surveys in 2007 and 2014, CRISP-DM is the most widely used methodology for data science and analytics projects: over 40% of survey respondents reported they use it.

Crisp-DM is designed to improve the speed, efficiency, and accuracy of data analysis by iterating through 6 phases:

The CRISP-DM diagram shown below makes it easy to visualize how the phases flow together:

These steps aren’t meant to be followed linearly. As the arrows in the diagram indicate, some of the steps will lead sequentially from one to the next. In this way, it is a cyclical and iterative process.

For example, when starting a new project, it would be normal to spend a significant amount of time iterating between business understanding, data understanding, and data preparation in order to have a firm grasp on the problem and the data.

The original CRISP-DM 1.0 Guide established the methodology as a hierarchical process model, consisting of sets of tasks described at four levels of abstraction: phase (high-level abstraction), generic task, specialized task, and process instance (low-level abstraction).

The model was designed to be a high-level framework that included a strategy for mapping the generic process model to the specialized level. These are the mapping steps provided by the original guide:

1. Analyze your specific context
2. Remove any details not applicable to your context
3. Add any details specific to your context
4. Specialize generic contents according to concrete characteristics of your context
5. Rename generic contents to provide more explicit meanings in your context for the sake of clarity

Many teams have used this mapping strategy and published extensions to the CRISP-DM model. These extensions provide practitioners with tested best practices for data mining in various industries!

The CRoss Industry Standard Process for Data Mining was a result of special funding by the European Commission (EC) in the late 90s. The program’s objective was to establish a standard process for data mining, as the name suggests.

The project statement noted that the rise in High-Performance Computing and problems with interpreting vast amounts of data had led to a need for a process of knowledge discovery that was fast, well-understood, reliable, and valid across a wide range of applications.

To solve the problem, the EC formed a special interest group to broaden the basis for development and testing without sacrificing the efficiency and effectiveness of a small, tightly-focused consortium. The special interest group would also help facilitate the dissemination and exploitation of the results. 

The vision was that data warehouse vendors and data mining tool suppliers could exploit the process model to enhance their product and service offerings. The user partners could exploit the results of the project internally to improve business intelligence and decision-making.

The EC saw the business potential and benefit of developing a standardized process for the industry. “[CRISP-DM] will make large data mining projects faster, more efficient, more reliable, more manageable, and less costly. A widely adopted process should foster the development of a multitude of data mining tools which support it, thereby significantly contributing to promoting a profitable use of HPCN technology,” according to the project’s stated objectives

Over two decades later, CRISP-DM has stood the test of time. With the evolution of the field of data science, there have been some major expansions to encompass things like data engineering, data analytics, and machine learning engineering.

Additionally, there are a number of published extensions of the model. Many researchers and practitioners in various industries have systematically analyzed their processes and published their best practices in the form of CRISP-DM extensions!

While there have been many specializations of the model, others have built upon it and revised it completely: DASC-PM, TDSP, etc. Yet, CRISP-DM remains the most cited data mining process model in academic papers and by professionals.

CRISP-DM is a standardized process for solving real-world problems with data. It is a simple process made up of 6 phases and you can iterate through these phases in the order that makes sense for your project.

There are many ways to customize it to any particular use case and many published examples to learn from. It’s the best way for small teams and individual data scientists to structure end-to-end projects to deliver value in a systematic way.