{"id":167191,"date":"2023-03-22T10:59:05","date_gmt":"2023-03-22T09:59:05","guid":{"rendered":"https:\/\/liora.io\/en\/?p=167191"},"modified":"2026-08-09T18:44:42","modified_gmt":"2026-08-09T17:44:42","slug":"management-of-unbalanced-classification-problems-i","status":"publish","type":"post","link":"https:\/\/liora.io\/en\/management-of-unbalanced-classification-problems-i","title":{"rendered":"Managing Unbalanced Classification Problems : Part 1"},"content":{"rendered":"\n

Classification of unbalanced data is a classification problem<\/strong> where the training sample contains a strong disparity between the classes to be predicted. This problem is frequently encountered in binary classification problems<\/strong>, especially in anomaly detection.<\/strong><\/p>\n\n\n

This paper will be divided into two parts<\/strong>: The first one focuses on the choice of metrics specific to this type of data, and the second one details the range of useful methods to obtain a successful model.<\/p>\n\n\n

Part I: Choosing the right metrics<\/h2>\n\n\n

What is an evaluation metric?<\/h4>\n\n\n

An evaluation metric quantifies the performance of a predictive model. Choosing the right metric is almost mandatory when evaluating Machine Learning models<\/strong><\/a>, and the quality of a classification model depends directly on the metric used to evaluate it.<\/p>\n\n\n

About classification problems, metrics generally consist in comparing the real classes<\/strong> to the classes predicted by the model. They can also be used to interpret the predicted probabilities for these classes.<\/p>\n\n\n

One of the key performance<\/strong>concepts<\/strong> for classification is the confusion matrix, which is a tabular visualization of the model predictions against the true labels. Each row of the confusion matrix represents the instances of a real class and each column represents the instances of a predicted class.<\/p>\n\n\n

Let’s take the example of a binary classification<\/strong>, where we have 100 positive instances and 70 negative instances.<\/p>\n\n\n

The confusion matrix below corresponds to the results obtained by our model:<\/p>\n\n\n

\"Predicted<\/figure>\n\n\n

It gives an overview of the correct and false predictions.<\/p>\n\n\n

To summarize this matrix in a metric, it is possible to use the rate of good predictions or accuracy. Here it is equal to (90+57)\/170 = 0.86.<\/p>\n\n\n

The choice of an appropriate metric is not obvious<\/strong> for any Machine Learning model, but it is particularly difficult for unbalanced classification problems.<\/p>\n\n\n

In the case of data with a strong majority class<\/strong>, classical algorithms are often biased because their loss functions try to optimize metrics such as the rate of good predictions, without taking in account the data allocation.<\/p>\n\n\n

In the worst case, minority classes<\/strong> are treated as outliers of the majority class and the learning algorithm simply generates a trivial classifier that classifies each example into the majority class. The model will appear to perform well but this will only reflect the overrepresentation of the majority class. This is called paradoxical accuracy<\/strong>.<\/p>\n\n\n

In most cases, the minority class represents the greatest interest<\/strong> and that we would like to be able to identify, as in the example of fraud detection.<\/p>\n\n\n

The imbalance level varies<\/strong> but the use cases are recurrent: disease screening test , failure detection, search engine, spam filtering, marketing targeting…<\/p>\n\n\n

Practical application: Churn Rate<\/h4>\n\n\n

Let’s assume that a service company wants to predict its churn rate<\/strong>.<\/p>\n\n\n

Little reminder: the churn rate\u2019s ratio<\/strong> is the following : lost customers \/ total number of customers, measured over a given period, usually one year.<\/p>\n\n\n

The company wants to predict for each customer whether he will end his contract at the end of the year.<\/p>\n\n\n

We have a dataset containing personal information and contract characteristics of each customer of the company for the year X, as well as a variable to know if he renewed his contract at the end of the year.<\/p>\n\n\n

In our data, the number of ‘churners’ corresponds to 11% of the total number of customers.<\/p>\n\n\n

We decide to train a first logistic regression model <\/strong>on our prepared and normalized data.<\/p>\n\n\n

Surprise! Our code displays a good prediction rate of 0.90!<\/p>\n\n\n

This is a very good score, but our goal is to predict<\/strong> the possible departure of customers. Does this result mean that out of 10 churners, 9 will be identified as such by the model? No!<\/p>\n\n\n

The only interpretation that can be made is that 9 out of 10 customers<\/strong> have been well classified by the model.<\/p>\n\n\n

To successfully detect naive behavior in a model, the most effective tool is always the confusion matrix.<\/p>\n\n\n

A first look at the confusion matrix shows us that the good prediction rate obtained is largely influenced by the good behavior of the model <\/strong>on the dominant class (0).<\/p>\n\n\n

In order to evaluate the model related to the desired behavior on a class, it is possible to use a series of metrics from the confusion matrix, such as precision, recall, and f1-score, defined below.<\/p>\n\n\n

\"F1-score\"<\/figure>\n\n\n

Thus for a given class:<\/p>\n\n\n