A well-structured project goes a long way in helping you to achieve project goals in a clear and well-defined format. Discovering data-driven insights from the enormous data collections can sometimes be overwhelming, and scientifically structuring these data analysis projects can help towards efficient analysis and decision making as well as effectively communicating the insights to a broader audience.
Consequently, this article serves as a guide to highlight the framework with its nine important stages and their intended goals used to scientifically structure projects based on data-driven decision making,
Overview and Motivation: This is an important step in any project which demands to brainstorm. It highlights the reasons behind the start of your project and its intended goals. It eventually gives a clear idea about the research area of your project and effectively highlights what data-driven insights the project ultimately aims at.
Project Objective: This step clearly defines the goal or the objective of your project. It further helps in the formulation of the initial research questions based on the data source.
Data Source: This step helps to understand the data source used in your project in terms of its varied aspects. It gives an overview of the data source in terms of the origin of the collected data, its size, and as well as the information on the number of features and instances in the data.
Related Work: This step gives a background of the related work in the area of your project. It aims to provide an overview of the research being carried out in the targeted area by your data analysis project to highlight what important contributions your project will make. In scenarios where the same data source is used in other related work, you can highlight what your project aims different by using the same data source in comparison to the previous work.
Initial Research Questions: This step details the Research Questions (RQs) formulated at the initial stages of the project based on a primary understanding of the data but without a detailed Exploratory Data Analysis.
Data Wrangling: Data Wrangling consists of different steps that transform data from raw into a clean format, which is appropriate and accurate for data analysis. The different steps include, Examination of the Input Dataset: This step includes visualization of the input dataset to generate its statistics and effective summarization, Dataset Cleaning and Processing: This step includes cleaning of the input dataset to eliminate the missing values, duplicate rows, column renaming & reordering etc., finally writing the cleaned dataset back to the file for further analysis , Exploration of Cleaned Dataset: This step includes visualization of the cleaned dataset to generate statistics, analyzing through various data visualization plots the different variables in the dataset, examining the correlated features etc., and Data Preparation: This step make data ready for the different RQs through the removal of the unwanted features, addition of new columns etc.
Exploratory Data Analysis: Exploratory Data Analysis (EDA) is the process of visualizing the main characteristics in the data before its formal modeling to discover data patterns and verify the initial primary assumptions made on the data. This step further helps in effectively restructuring and reformulating the initial RQs.
Final Research Questions: Exploratory Data Analysis provides a feasibility check on the initial RQs formulated. The EDA phase helps to get a better understanding of the data in relation to the project objective. Hence this leads to the modification, removal, or addition of new RQs. Thus the outcome of this phase should be the formulated final set of RQs, which will be answered through the project.
Data Analysis and Modeling: This is the crucial step in a data analysis project, where we employ sophisticated algorithms and modeling to answer the formulated research questions. To further structure and communicate the data-driven insights to a larger audience effectively and clearly; for each RQ, it would be a good practice to structure them into five informative steps as, Algorithms Selected, Reason for Algorithm Selection, Analysis and Modeling, Observations and Applications.
Customer Behavioural Analytics serves as a real-time example of a data analysis project carried out at a research university and structured according to the insights shared in this article.