Unfortunately, there is no method to directly link pie colors to labels.
The charts in PowerApps are only for basic purposes. It's worth mentioning that you could look into Power BI for richer visualization capabilities. It's easier to customize pie charts. Power BI visualizations can be embedded inside PowerApps. It is a separate product from PowerApps though, and has different price plans.