Custom category classification - customer service emails

[LimeLeaf]

Hi community,

currently I'm working on a project where I want to categorize customer emails into certain business cases. Therefore I'm trying to use the MS AI Builder capability's 'custom category classification model' combined with power automate . As I didn't found that much regarding this topic I want to share my experience and some useful links here. Please feel free to share your experiences with the 'custom category classification model' any participation is highly welcomed.

I'm trying to explain what I'm doing with fruits so hopefully it helps to understand the whole thing better.

We got as almost every company a shared mailbox where customer service is sitting on. In this mailbox different kind of business cases (apples, bananas, grapes, pears and pineapples) arrive daily. Out of all these apples, bananas, grapes, pears and pineapples  we want to process apples and bananas. So we want to use the AI custom classification model to identify these apples and bananas between all the other fruits. 

What I did first was to train the AI model with apples and bananas but this didn't do the job. The testing results reveled that the AI categorized pears also als apples as they are very close when you only know apples and bananas. Even some pineapples where categorized as apples or bananas without any logical context at first glance. I used about 60 apples and 60 bananas to train the AI at first. As I noticed the results where not satisfying I tried to train the AI with just unique parts of these fruits. But this doesn't made the results any better. 

I considered the following recommendations for the training sets:

(Source: https://powerautomate.microsoft.com/en-us/blog/tag-incoming-emails-with-ai-builder/)

Now I'm training the AI model with apples, bananas, grapes, pears and pineapples to make sure it can hopefully distinguish between e.g. apples and pears. As this is now a much more time consuming process than just dealing with apples and bananas I hope that this procedures helps. 
 
Here some useful links that I found so far:
How 2 - https://powerautomate.microsoft.com/en-us/blog/tag-incoming-emails-with-ai-builder/
MS Documentation - https://docs.microsoft.com/en-us/ai-builder/text-classification-overview

What are your experiences so far with training or results of the custom category classification model?

Cheers

[JoeF-MSFT]

Hi @LimeLeaf - thanks a lot for taking the time to share your experience using AI Builder. We love reading posts like this. 🙂 I'm sure other readers of this post will benefit as well. Providing a good number of samples for each category, and balanced numbers across categories is indeed key. 

What's next for this project? Is this for a customer of yours? Is the customer satisfied with the solution? 

Feel free to share any additional feedback and I look forward reading the experience from others in the community as well. 

[LimeLeaf]

Hi @JoeF-MSFT,

it is a customer project indeed. 
As soon as I got the custom category classification up and running I will use the entity extraction AI to extract custom values from customer mails. After this I will do validations steps using power automate and power automate desktop. Then further processing the mails according to extracted business cases and their extracted values in subsystems like SAP by using power automate desktop. 

I'm already very satisfied how the "entity extraction" with custom values works out. As mentioned above where I'm struggling currently is the part of category classification. I'm currently in the phase of creating new learning datasets which represent a good cross section of all categorys that are arriving in customers service mail box.

[JoeF-MSFT]

Thank you for sharing @LimeLeaf !

[LimeLeaf]

Hi community, Hi @JoeF-MSFT ,

as I am currently collecting new datasets for a new training of a custom category classification model I discovered that the customer has about 50 different business cases arriving in the targeted customer service mailbox. These cases sometimes are not only clearly apples, bananas and pineapple. They even inhabit sometimes apples inside of pineapples. As some customers write emails regarding multiple business cases in just one mail.

I'm really doubting that these 50 different business cases and their combinations can be clearly identified by the custom category classification model. As this seems to complex at the moment to distinguish between the different cases and combinations from my point of view. Even if I would provide the AI with thousands of learning examples having the same quantity of each business case. As this collecting and categorization has to be done by me currently - to have consistent training data I have to invest 50+ hours. In the end I maybe discover that the AI is not capable of clearly distinguishing these cases and my customer will be really unhappy. 

And another thing came to my mind regarding this topic. What if in the next months suddenly additional business cases arrive in the service mailbox. As far as I understand the process of learning I have to recollect and retrain the  the AI so the model is capable to distinguish again between relevant and not relevant emails.

Currently I really think that the custom category classificiation model might work in a mailbox where just 4 clearly identifiable mails without to much complexity arrive for the rest of the live of the mailbox. But in case of a mailbox where to many different business cases arrive and where is change to the business cases even (seasonal change) the model might fail.

Currently I'm pretty unsure if and how to progress from here. I'm thinking about other solutions here.
Any similar experiences here in the community? 

[LimeLeaf]

Hi community,

we discovered that over 100+ different business cases arrive at our customer service team mailbox.
As the data collection and maintenance of an AI model in this dimension would be to time and cost extensive for us we decide to go another way.

Instead of trying to solve this problem by AI, we are now focusing on an customer self service portal and smart forms to reduce the amount of incoming customer mails.

Cheers