The Draft Orders (Standard) and Accelerated Methods of Custom Pricing do not create variants that could go over your store's variant limit. Visit Switch Discount Methods in Custom Pricing to learn how to switch.
If a customer is able to purchase a Custom Pricing variant when it’s out of stock, this is usually because the variant is still marked as in stock.
To fix this, you should manually sync your inventory within Custom Pricing. We generally suggest syncing the inventory when you create or edit a "Product Pricing Group".
Visit How does Custom Pricing handle inventory? for more information on how the app works with your store's inventory.