Issue with PVA Analytics

[kzaidi]

Hi all,

We noticed that PVA Analytics capture testbot sessions in the analytics as well which is kind of weird. You could test it by entering a question in the Test bot window and it would appear as a session in the sessions excel file (downloaded via Sessions Tab in Analytics Menu). Is there a way to separate the actual sessions triggered via the demo website and the ones started from the test bot window (typically not needed as they are for testing purposes only).

I am thinking to get ChatTranscripts file from Power Admin Center to see if there's any indication in there. 

Please share your views, suggestions, and any other similar findings with analytics

Thanks

[renatoromao]

Hi @kzaidi ,

I don't think so, look at this documentation here: View billed sessions - Power Virtual Agents | Microsoft Docs

If you are looking for a different way for reports, you can create your idea here: https://aka.ms/pva/community/ideas

Sessions that use the embedded test chat or that only leverage capabilities included with the Power Virtual Agents for Microsoft Teams plan don't count toward the billed sessions. The Billed Session report doesn't distinguish between the sessions included in the Power Virtual Agents for Microsoft Teams plan and those that are not. You might see an inflated number of billed sessions in the report.

[kzaidi]

I have read this before and its kinda surprising for me too, please cross-check and let me know if you are experiencing the same.

Thanks

[renatoromao]

Hi @kzaidi ,

Yes, unfortunately, same behaviour here. 😞

[mboninco]

---

Hello @kzaidi @renatoromao 

Except for Billed Sessions, the analytics page includes sessions from PVA's test pane. The thinking is this would help users discover how analytics work and aide in the development for their bot. In most cases the sessions from the test pane should be very small in comparison to those from the deployed bot.

The conversation transcripts which can be downloaded from Dataverse contain a flag IsDesignMode that can be used to differentiate sessions that are from the test pane.

---

Mike Bonincontri
Senior Program Manager

Microsoft

[kzaidi]

Thanks @mboninco  for your reply,

Yeah, I scanned the Chattranscripts file very carefully yesterday and found that channelId is null for test sessions and IsDesignMode is true. However, the reason for merging test sessions with the actual billed sessions is still not convincing. As a chatbot manager, one would be interested in the actual analytics, not the test ones.

Cheers

Kanwal