Our current database provider only backups our data once every 24-hours. We'd like to implement a process whereby we build a base spreadsheet (CSV or Excel) and only add new rows to that spreadsheet when new records are created, or update existing rows when records are modified. We would like to build a scheduled flow that runs every 4-hours, and the records it gets returned will only be those updated within the previous 4-hours. We'd like to do that instead of creating a new CSV or Excel table every 4-hours.
Some tables in our database have 1,000 fields. The database is accessible via API.
Sharepoint is extremely slow. Sharepoint lists crawl to a halt when they exceed 5000 rows. Sharepoint is the worst possible tool for what you are trying to do. Look at alternatives like Azure SQL, or datamarts etc.
What's wrong with your database being backed up every 24 hours? What's the business impact of losing a day of data? Can you compare that against the cost of increasing the backup frequency?
I was thinking more along the lines of creating a CSV file or an Excel file and storing that file in SharePoint. I agree that SharePoint lists are slow, but to import the backup file into our database I just need it to be in a CSV or Excel file.
The business impact of losing a day's worth of data could be quite large and ripple through our processes for several days if it were to happen. We're looking for ways to minimize that risk. So, IF there is a way perform the backup more frequently, we'd like to give it a try.
For what you want to do, I think Layer2 Cloud Connector would be your best bet. We've been using it for over a year to sync business system data including SQL databases to SharePoint. It is very fast and you can set it to synch in both directions or one way. You can set it to sync as often as you want. The biggest table I am currently synching has about 47,000 rows of data. Just make sure to index some fields in SharePoint before you do your first sync if it exceeds 5,000 rows.
You can get more information at Data Integration & File Sync for SharePoint and Office 365 | Layer2 Cloud Connector (layer2solutions...
Join us in Las Vegas to experience community, incredible learning opportunities, and connections that will help grow skills, know-how, and more.
Register today for two amazing days of learning, featuring intensive learning sessions across multiple tracks, led by engaging and dynamic experts.
The European SharePoint Conference returns live and in-person November 28-December 1 with 4 Microsoft Keynotes, 9 Tutorials, and 120 Sessions.