I would create an Office365 group or and Active Directory security group - use Powershell to add in all user objects (or get someone else to if you aren't comfortable with Powershell), then go in and remove any that you don't want to have access, then share the App with that Security/O365 group. You will need to manage that group for any new joiners/leavers though..
Currently there's no 'deny' rule that can be set, which would be the way you would normally apply something like this:
1. Deny access to User1, User2 and User3
2. Grant access to all users
3. Deny rules override any access given so step 1 users still do not have access
I've submitted a request for this to be a new feature, so you can upvote that here if you'd like to see it implemented: