W hat if I ask you to think about data science work? Chances are that images of notebooks, data plots, algorithms and programming snippets pop into your mind. When talking about this sort of work very often we focus on tools and techniques useful for specific tasks ranging from things like classification or NLP, through cool new python libraries and some effective computation tricks, up to the latest jupyter notebook trends. What seems to be largely missing from this picture is: how the actual modelling tasks that need to be solved get formulated?
If we step back to think about the high level goals that your typical data scientist tries to achieve — two quite different workflows emerge.
First up — exploratory data analysis. In such case, the focus is put on understanding your dataset, finding some interesting and useful patterns within and presenting them in a way, which will allow decisions to be made. In this sort of work, usually little to no guidance is given from the dataset’s “owner” except for a general direction like finding some anomalies, looking for inefficiencies, recurring patterns or the like. You usually go where the road leads you and your main task is just to get some insights into the processes that actually generated the data. The work usually ends when you simply run out of the designated time slot.
Another type of work involves creating some sort of predictive model which will be used on previously unseen data. This might be a one-shot thing, if the purpose is only to present and discuss the obtained results with the board of directors. This might also end up in actually shipping your model into a working computational system where it will live and thrive for a long time. Whatever the case, the people who asked you to do this sort of task probably have something (more or less) specific in their minds, so even before starting your work you need to precisely define what are you actually to do.
Because chances are that the data science work is actually spawned by some roughly defined business needs — we have a clash of two realms. The former works best when surrounded by precision and rigor, while the latter stems from the often messy and vague real-world needs. In order to be successful you need to have a translation between these two.
Basically you need to formulate the requirements in the business domain language, translate them into well-defined analytical tasks and determine measurable success criteria. Sounds abstract? Let’s look at some examples.
Sometimes discovering what you actually need to do is straightforward. If you are asked to create a spam filter plugin for a given email client, you can quickly translate this into a binary classification task and assume that your model gets access to email content and metadata. Defining the success criteria will require assuming some acceptable ranges for metrics like accuracy, true and false positives etc. — you can estimate these based on how the currently available solutions perform .
Other times things get a bit more complicated. Let’s say that you’re running an e-commerce website and you would like to include product recommendations on your homepage. This is your business need which can be translated into 2 modelling tasks:
Now we still need a success criterion to tell us whether our results are sufficient. In this case we want to actually use the recommenders if they add some value to our homepage. A simple criterion would be: use the recommenders if they are able to obtain a click-through-rate which is higher than the one currently calculated for the homepage. So in this case you evaluate your new approach on whether, or not it improves the existing state. Another angle would be to compare the performance of your model compared to for example using only the baseline best-sellers part. Either way, you in a sense bootstrap your success criteria .
In real-world scenarios data science tasks cannot be taken for granted — you need to define them. To create a useful definition remember about translating the business language into modelling tasks and creating clear and measurable success criteria. Without this it is difficult to decide whether what you created is complete and actually useful.