cancel
Showing results for 
Search instead for 
Did you mean: 
Reply
ZakXrm
Regular Visitor

Ai Builder Prediction model performance - High Score & Testing

Hi, just looking for some recommendations in relation to testing with the Ai builder. I'm currently using the Prediction Model with some Timesheet entries used to track the length of time worked per project. These are broken down into days, time, client, project, work done and user performing the work. 

We use a field called, "Is Correct Rate," as a yes/no field, which I've current set the model to use for predicting future outcomes and use the below fields to influence the data: 

  1. Billable Rate
  2. Project
  3. Team Member
  4. Task
  5. Client

Currently, every test I run outputs a score of 98-99%, being run on a data set of 6000+ records and based on some initial testing, is working incorrectly and marking every "Is Correct Rate," field of new records as false. Billable Rate seems to stay consistently as the most influential field.  

Any recommendations in regards to why this would constantly score 98-99% and any information with split testing and training to train the Ai builder more relevantly? Thanks in advanced for any info anyone can provide here. 

1 REPLY 1
Microsoft
Microsoft

Re: Ai Builder Prediction model performance - High Score & Testing

Thank you for bringing up this issue.

 

As we discussed, you kept seeing high training score because the training dataset you provided is very imbalanced towards the positive end. And this is not asking you to change anything on your training data to make it more balanced. We would like you to keep using dataset that could represent your production data distribution. And AI Builder team is already aware of the issue that Accuracy isn't a good measurement for imbalanced training set. We will soon release a new grading system to measure your model performance which could cover both balanced dataset and imbalanced. And you can decide whether or not to use that model based on it's grade, is it an A, B, C or D model (A is the best, and D would most likely require you to fix some issue).

 

And for this particular model, you'll get a B which might reflect the true status better.

 

Helpful resources

Announcements
MBAS Gallery 2020

MBAS Gallery 2020

Watch Microsoft Business Applications Summit sessions on-demand.

firstImage

New Ranks and Rank Icons released on April 21!

The time has come: We are finally able to share more details on the brand-new ranks coming to the Power Apps Community!

Power Platform 2020 release wave 1 plan

Power Platform 2020 release wave 1 plan

Features releasing from April 2020 through September 2020

Top Solution Authors
Top Kudoed Authors
Users online (2,325)