Application Users, which are used for server to server authentication in Dynamics 365 online, have been designed without foresight into the full lifetime of how they will be used.
As it stands, the only way of preventing access yourself is to delete the app registration within Azure AD. This is insufficient because the CRM system administrator may not have access to Azure AD. Even if the app registration gets deleted, you are then left with an enabled application user that doesn't actually work.
For anyone facing this scenario, support will disable the application user if you open a support request, but they won't delete them.