Announcing the .net Core SDK for Common Data Service (CDS) - External Client Development

[MattB-MSFT]

This Post has been superseded by net-SDK-for-Dataverse-Dataverse-ServiceClient 

Folks, 

As many of you know, we have a existing .net full framework SDK that we provide for folks to develop client applications that communicate with CDS.  This SDK has existed for a long period and arose from the Dynamics 365 aspects of our platform. 

We have continued to support this SDK over the last many years over many versions of our platform.   

We have been working to update this SDK package to better address the needs of the current platform and the technologies that many of our partners who develop client applications to integrate with the Power Platform use.   We are now ready to share this with the larger community for testing and feedback. 

Today we have posted the alpha version of the Microsoft.Powerplatform.Cds.Client on nuget.  along with a few additional packages that we are using to split out the Dynamics Sales / Service and Marketing centric messages and helpers from the core CDS platform SDK

This version of the SDK libs supports the following and has the following notices: 

1. .net full framework 4.6.2, 4.7.2, 4.8 and .net core 3.0, 3.1 
2. We are supporting only Client Secret / Cert flows for .net core.  On .net framework additionally we support OAuth User flows
3. We are shipping the initial drop using ADAL.net for the Auth Lib.   We MAY change to use MSAL shortly 
   1. MSAL would enabled us to support User Interactive flows on .net core. 
4. The Namespaces of areas WILL change.  We do not know what they will change too just yet but they will change before we “release” this.
5. The Assembly Names Will likely change at least once more. 
6. The Message types that are part of the client have been reduced to CDS Core server messages only.  Things like “QualifyLeadRequest” have been removed to their own Nuget package ( Microsoft.Dynamics.Sdk.Messages ) 
7. We will likely ship more extension packages that will contain the “CRM” messages,  though over time, we will likely split the namespaces of those messages up based on service line,  think Field Service or Sales or Customer Service, etc..
8. Plugin Development using this Client is NOT supported at this time. 

We have pushed the first version up to nuget,  its tagged as preview and I encourage you to read the release notes we provide with the nuget packages. As with most of our Nuget packages that are intended as tools or for developer consumption, we extensively comment in release notes. 

Samples and such will be updated overtime on the PowerApps Samples GitHub Site as we move forward with the evolution of this capability.   That said, any of the existing CrmServiceClient samples can be used as a base to start.

From a scenario point of view,  we are particularity interested in any issues or challenges when using these library in either Asp.net Core or Functions scenarios. 

Links for the packages:

https://www.nuget.org/packages/Microsoft.Powerplatform.Cds.Client

https://www.nuget.org/packages/Microsoft.Powerplatform.Cds.Client.Dynamics

https://www.nuget.org/packages/Microsoft.Dynamics.Sdk.Messages

for 99% of your applications working against Dynamics, you should only need Microsoft.Powerplatform.Cds.Client and Microsoft.Dynamics.Sdk.Messages.

if your working against CDS Only,  you should only need Microsoft.Powerplatform.Cds.Client.  If you do not experience that, or find missing messages in the CDS only scenarios, please let us know. 

Note: Please be aware that during the Alpha \ Beta phases, these nuget packages are not supported via official channels. 

Support is aware of them, however you will likely be directed back to the community forums for support during the Alpha \ Beta phases.  A number of our dev's and PM's do monitor this channel and can respond to questions and feedback. 

As we progress this process it is our intent to deprecate then retire the older Microsoft.Xrm.Tooling.* Packages and replace them with updates built on this new set of Nuget Packages. 

Thanks All, and we look forward to your feedback on this.

Update:  We have setup a github issues site to track issues for this effort: 

Update#2:  We have now begun posting the code for the CdsServiceClient on this git Hub repro for reference. 

Please find the site here: https://github.com/microsoft/PowerPlatform-CdsServiceClient 

MattB-MSFT.  

[fberasategui]

---

@MattB-MSFT wrote:Plugin Development using this Client is NOT supported at this time. 

Is this expected to be supported in the future?

Also, why are the new packages not .NET Standard-based? 

Shouldn't that enable them to be used for plugin development, as well as client scenarios? Meaning they would still continue to be executed on .NET Framework inside CDS, but also on .NET Core outside the platform.

ILMerge-ability is desired as well. I understand Microsoft does not support this, but in practice it is being used on production in a LOT of projects. I'm not sure if this would be affected 

On a side note: can we please switch to Github for these discussions? I understand there is a low/no-code side of the platform intended for people who might feel comfortable with this forum, but for us code breathers it really isn't. This is the third time I type this stuff here because my post was rejected due to some weird HTML stuff, and then I got these errors after trying to login.

And btw, are you ever planning to open source the SDKs? my reverse-engineered version of the plugin debugger integrated right into Visual Studio yearns to become legal so it can be shared with the wider community.

[MattB-MSFT]

Thanks for your feedback,  and In answer to your questions:
( bit of a book here 😉 )
 
Why not .NET standard:
.NET Standard 2 is missing several methods from .NET framework 4.8 that our SDK requires. The .NET standard team is aware of this and is planning to address it in the future.  When they have support for it, we will provide a target for .NET standard.
 
Plugin/Sandbox in general:
There are a number of enhancements and changes in flight with the Plugin system currently.  Support for tech stack's other .NET Framework are being consider, but require substantial investment to bring to market and thus take time.   .NET Core is one of the tech stack's being looked at.
 
We are logically separating External Client development from Plugin development, and we wanted to bring the .NET core based clients tech to market as soon as we could make it viable.  Additionally, We are in working on splitting up the Namespaces, message names, and in some cases renamespacing sections of our platform away from the combined "crm" model to more closely match how we are building, servicing, and operating the system.  ( thus the Alpha tag on the current release train ). 
 
With regard to ILMerge: 
This comes up a lot and I will try to explain the reasoning here.
The reason ILMerge is used is one of the key reasons for several enhancements in our Plugin system currently in desgin.  ILMerge is unsupported, its unserviced and is technically unowned inside Microsoft.  It was a Microsoft research project that was never taken forward and those updates that have been done have been done by teams using it internally.  That it works as well as it does today is a nod to the programmers of the .net framework and the author of the ILMerge tool.   "We" cannot officially support it for use with our platform as the PowerPlatform team cannot take on the responsibility for bug fixing ILMerge itself, which would be the byproduct of PowerPlatform's Official support of it.  That said, we do not actively block the use of it in our platform because we know that its require for some scenarios currently.
 
With regard to OSS and GitHub:
It’s a fair ask,  the reason we didn’t do it at this time was that to take this discussion to GitHub would require us to have a Repro on GitHub with the source posted, we could do an empty repro that would not really do much more then this board.  To get to the code posted state requires us to setup an large amount of process and infrastructure to comply with our commitments around our SDL policies that we do not have time to do at this time.   That said we DO want to do it at some point in the future.  it really a matter of getting the aforementioned process and overhead support in place to make posting to git hub more useful than just having a non-compiling copy of the code post there that folks can comment on.
 
With regard to your comment about reversing the PRT and using part of it in another project:
I get the use cases for this, however to put this in our prospective: 
We OSS'ed our login code , several tools and the prt in CRM v5 and v6.  Because that code exists, we spend a lot of time on support cases where the primary reason is that someone took the v5 or v6 code and integrated it a custom tool set or project and sold or used that in a deployment, due to the changes we have made of the last 6 years or so the code stopped working and we are asked to solve it for them.  This is why we invested so heavily into Nuget Modules over the last 7 years or so to allow folks to just take an update to a nuget package to stay current, and we worked very hard to maintain backward compatibility in the client interface code ( CrmServiceClient and such ).  

Specifically with regard to the debugger of the PRT,  this is actually something we are discussing turning into its own Nuget package vs shipping with PRT as we do today.   Please send me a direct message with your use case, I would like to understand what your working on there and why you have made some decisions.  It could be we can share how to use the debugger.exe that is part of PRT natively with you and avoid you have to have forked the code.

 
Hope that helps clear up some of our decisions here.
MattB-MSF

[fberasategui]

Matt:

First of all thanks a lot for hearing me. I will try to address your thoughts with as much detail and as clearly as possible. This will probably get long too 😉

The Message types that are part of the client have been reduced to CDS Core server messages only.  Things like “QualifyLeadRequest” have been removed to their own Nuget package ( Microsoft.Dynamics.Sdk.Messages 

This is excellent. IMO a much wider separation needs to be drawn between the CDS platform and the Dynamics 365 product. I mean both in the documentation and in a practical sense such as this package separation. Keep doing more of this please.

As an example: I've been working for more than 4 years now in both existing and greenfield projects where the usage of Common Data Model entities (Opportunity, Lead, etc) and vanilla "Dynamics 365" functionality is practically ZERO and instead we have a lot of custom business functionality supported by custom industry-specific or client-specific data models and business processes. 

TL;DR: we're using CDS as a business application platform and don't care very much about CRM functionality and default entities. Keeping these aspects as separate as possible is a plus for us.

From a scenario point of view,  we are particularity interested in any issues or challenges when using these library in either Asp.net Core or Functions scenarios. 

I'll report back as soon as I get a proof of concept up and running on Azure Functions V3 on .NET Core using the new CDS Nugets. At this time we're stuck on Azure functions V1 precisely due to the need to target .NET Framework and use Microsoft.Xrm.Sdk.dll

There are a number of enhancements and changes in flight with the Plugin system currently

It would be awesome if you could communicate the general technical roadmap and would be willing to discuss it with the community. This is a philosophical matter, and might be a big change, but a lot (most?) of the Microsoft dev stack has converted in recent years to a workflow where the will make their roadmap public and are willing to discuss it with the larger community (think how the C# language team went from the Anders/Mads era of "benevolent dictatorship" to the current Github-all-the-things approach). As a developer who invested a decade exclusively in the MS stack, this is tremendously valuable.

Support for tech stack's other .NET Framework are being consider

I imagine this has to do with getting plugins out-of-process in the form of Azure Functions or something similar? If so, keep in mind it is important to maintain existing functionality (synchronous, single transaction, friendly business error messages for the end user, and so on)

To be fair though, this might make sense for greenfield projects, but there's no way existing projects with heavy business logic will migrate to a non-C# tech stack. For instance it makes no sense for me to write a CDS plugin in javascript, but that's mainly because I hate js and dynamic languages in general.

NET Core is one of the tech stack's being looked at

.NET Core, on the other hand, makes a lot of sense as a target platform for both client and in-process (plugins) CDS code, because it enables us to design application architectures where Model classes and Business Logic is shared among these, and I no longer care about whether my business logic executes in a plugin, or inside an azure function, as long as I have a valid IOrganizationService instance to work with.

For instance: I have this abstraction which makes it pretty much transparent for application code whether it's running in an Azure Function or inside a regular plugin.

We are logically separating External Client development from Plugin development

Please consider that it is very important to maintain the current compatibility level. I don't want to have to differentiate plugin code for business logic from other business logic that executes elsewhere. The entity model is the same, the IOrganizationService interface is the same, and whatever abstractions I write on it are the same. As an aside, the LINQ provider is of critical importance too and it is critical that it will continue to work in all currently supported scenarios.

I will PM you a solution diagram to illustrate our current preferred architecture for CDS based applications.

With regard to ILMerge

I understand and agree with everything you've said about this subject. If the platform enables me a different, supported way to reference my own assemblies/Nuget packages inside plugins I will gladly get rid of ILMerge too ;). When we have 30+ plugins in a single VS Solution each build/rebuild becomes slow because it has to ILMerge 3 or 4 assemblies into each one of the 30+ plugins 😞

Again, thanks for taking the time to read my post, I greatly appreciate that.
I will email you about the debugger and with additional details of the designs and tools I've come up with during these years working on XRM/CDS.

btw, you guys rock so much. Keep going.

[MattB-MSFT]

Regarding Plugins:

we do not have much to say specific feature wise on this right now aside from "there is a good deal of work going on". When we get closer to knowing what lands and when, we will start talking about it publicly 🙂

Regarding non .net plugins types…

You would be surprised the number of requests we get to support non .net languages 😊 Non-net plugins will necessarily have a different method of interaction.

With regard to compatibility with the .net framework in the alpha, The current alpha is fully compatible with all the features of the full framework system, including Linq,

MattB-msft

[Menik1]

Amazing news that we finally have .net core support!!

Will we see async/await being added for organizationrequests? Azure Functions, for instance, are billed by cpu time, memory usage etc... Would be great to not have to take cpu time while waiting for the response.

[Toftager]

Great stuff, have many customers asking for .Core compatibility, so looking forward to this becoming a non beta. Will start looking into this right away. 

What is the outlook of this running on a on-premise installation? Specifically one running AD Auth. I believe this would work with claims, but could not get it working with AD auth. 

[ben-thompson]

@Toftager  From memory (it's a while since I've done it) Certificates work better than shared secrets in the AD world.

Have you followed all the steps in https://docs.microsoft.com/en-us/dynamics365/customerengagement/on-premises/deploy/configure-the-dyn...  as it's possible you may be missing a step out.

[kevind2000]

Plugins are based on WCF Message Pipeline, are CDS re-engineering away from WCF on the server side in order to be reborn into the Cloud?

[AhmadAnwar]

Any love for F#?
Currently, the plugin system doesn't recognize FSharp.Core.

It would be great to register plugin written in F#.