General Dataverse Advice

[JasonSealy]

So I am in charge of setting up a database in Dataverse. Over the last several weeks I have been researching, migrating data to Dataverse, and configuring the database. I have learnt lots through this time but I have some high-level questions that I have not been able to find answers too (mainly because my questions are situation specific) and I need advice. If you have any input on any of the following questions it would be greatly appreciated.

Context: I am working for a small engineering firm whose primary purpose in converting to Dataverse is to use the quick low-code features in Power Apps to quickly create applications that can speed up the current paper processes. I am in a Computer Science Intern position and will be leaving the company for school in early August. I want to leave the company with a good frame work for them to build from where I left off (there will not be someone replacing my position and I will be training co-workers before I leave who do not have a background in software development). I will be making a high-level and company specific "software development manual" so co-workers can refer to it if they need help.

Questions:

1. Application lifecycle management (ALM). Within Dataverse there is environment and solution strategies to initiate a ALM system. Is this overkill to start with? I don't think the company would need ALM for the next 12 months but I don't want to start Dataverse on the wrong foot and be a pain down the road to have to switch everything to work with ALM. Currently I have decided to set-up a single environment with a basic app in a single solution and not initiate a ALM system.

2. Common Data Model (CDM). Is it worth using CDM in Dataverse, from research I personally don't have much excitement for CDM. I think this is because I don't know much about it and the benefits it could bring. It personally doesn't sound like it is overly useful compared to creating custom tables. I like knowing about the data model of my entities and the CDM is confusing but maybe it is worth investing time to understand it? Currently I have decided to not use and CDM tables.

3. Multi-application integration. My company currently uses SharePoint for many of its file storage. We see the benefits of SharePoint for being a useful collaboration tool but were concerned about its security and scalability; hence the transition to Dataverse. However from research it seems like it may be useful to use SharePoint and Dataverse together as a team. Currently I don't have a concrete plan but was thinking that we would have photo/file storage on SharePoint and table/number storage on Dataverse. This is because from some research it seems that Dataverse storage is expensive and does not support photo/file storage as good as SharePoint. Is this a good idea? I am open to recommendations for how I should delegate what data gets stored where, or if we should eventually remove any use of SharePoint.

4. Where is data validation in Dataverse. One question that has been a headache to figure out and still don't feel like I know the answer too. So to my understanding, Dataverse has some basic data validation for each table using Business rules and then has better data validation for Model-Driven and Canvas apps. What I don't understand is why there isn't more focus on table validation, because if you have multiple apps with different developers using the same table, then data could be inconsistent in the table because each developer for their app could have different methods for data validation. Is there a reason for why there is better validation at the app level? ... because I don't understand why.

5. Invest in a full-time software/database developer? When I leave at the end of summer there will be no one who is tasked to develop the software/database for the company. It would be a shared effort between the co-workers who are not real developers. My biased opinion would be to pressure them in finding someone to replace me as soon as I leave. Nevertheless I must be cognizant that companies must make smart business decisions. So I am wondering how important it would be to have someone working full-time as a skilled software/database developer and how soon the company hire that person. Could a Dataverse system be administrated by a bunch of "citizen developers" or is that a recipe for disaster?

Thank you in advance, I truly appreciate any opinions or advice. I understand that this post doesn't give nearly enough answers to questions that would be raised while reading. I am more than willing to respond to additional questions in the comments below.

All the best,

Jason

[Stevieed]

1, you dont need a Application lifecycle management for small firm/ small project, it will just add another layer of complexity where your company doesnt have professional program developer to fill in. just make sure everything is backed up (back up take place every 30min on dev/qa/production environment, check power apps admin centre for setting.

2, Dataverse is the closest thing you can get to a professional highly scalable database in the low code regime. The only downside is that it's still very amature, for example you can't have more than 10 roll up column in a table, and roll up column cannot rolling up from roll up column in child table, and the roll up column does not auto refresh, it will only refresh every 30min minimum and you need to set it up. What is a roll up column? it just basically a =sum() column in excel. This brings a lot of limitation to many many business operation. To overcome this, you will need to write code or use power automate to write logic to sum the child table data into parent table, and it can be very complicated. 

3 you can store data in dataverse and related photos / document in sharepoint, check out topic about "dataverse sharepoint integration". however it may not be ideal as you think. If a company use sharepoint for a long time, they will most certainly have their own folder structure to store items, but dataverse has it's own sharepoint file structure, for example, it will name a folder as "u141ol-23o1iyg-321414-1234oh21" and store the related files there, which makes completely no sense to regular user, and I currently still don't see a way to change this. I decide to just pay for the dataverse file storage, as it's actually not too expansive, if you just storing documents. If you trying to store photos / video, then you are in trouble.

On another note, dataverse can interact with sharepoint very well with power automate, but to set this up you need to have knowledge in power automate. It will likely to take 1 month to learn it, as a lot of argument used in power automate dataverse connector are very implicit (such as Odata quary syntax, fetchXML syntax, etc, dataverse use different syntax all over the place, also some field names are completely different and not visible unless you use tool such as level up for dynamic 365, etc etc)

5, leaving dataverse / power app to non professionals is a disaster. they dont need full time developer, just get a part time dev to carry on your job when you left

[LinnZawWin · Accepted solution]

@JasonSealy 

1. If the company is small enough to start with ALM, I recommend you start with at least building the components (tables, apps, flows, etc.) in the solution. That's a good start that you're doing to start a basic app in a single solution. You can always set-up another environment when the project is mature enough to test in a separate environment before deploying to the live (production) environment. You might also want to explore the pipelines in Power Platform when they are ready to expand to multiple environments.

2. I think it would be easier to use some of the CDM entities (Contact, Account and Activities) in your system. You may re-build any of those tables as a custom table but there are some scenarios where the CDM entities are directly tied to other M365 products for integration (e.g. syncing Emails, Contacts, Tasks with Dynamics 365 App for Outlook)

3. As you mentioned, Dataverse file storage is expensive and it does not support features like versioning as in SharePoint (which is a useful collaboration tool for documents). As you planned, photo/file storage on SharePoint and table/number storage on Dataverse would be a good idea. I don't see any scalability issue but if you have a security concern, you can use third-party products like SharePoint Security Sync or CB Dynamics 365 to SharePoint Permissions Replicator to sync the permission between Dataverse and SharePoint.

4. If you set up the data validation in a Business Rule with an Entity scope, the validation will be done on the server side regardless of the source of data (entered from multiple apps). Dataverse plug-in can also be used for data validation which is complex and not achievable with Business Rules. In my opinion, if different developers are using the same table in their own apps, they should all agree on the basic data validation of the shared table rather than having different data validation for their own app. If there is any such requirement, they can have their own validation with Business Rules with a form scope for the form which is used in their own apps.

5. I think administrating the Dataverse system by a bunch of "citizen developers" without proper guidance from an IT could be a recipe for disaster. And I also don't think a normal software/database developer could be a perfect person for that role either. The company will need someone with Power Platform/Dynamics 365 administration/consultant experience to develop/provide proper guidance to the citizen developers. Depending on how much the company is going to invest in Power Platform as a low-code platform to build specific business apps, they may get part-time or full-time personnel as a skilled Power Platform admin/consultant probably a month or so before you leave (to have some transition and handover). Alternatively, the company can also engage with Microsoft partners (specialising in Power Platform) who offer services like empowering citizen developers, and advisory services (such as governance, app review/assessment, etc.).

[JasonSealy]

Thank you @LinnZawWin, I appreciate the help! Best of luck to you!