Can I hide the button for products I don't track inventory on?

Yes, there is a setting you can use to configure this.

1. Visit Campaigns, Product page button and choose the Behavior tab.
2. Under Behavior ensure  Hide the widget button when inventory is not tracked in the store catalog is checked.
3. Hit Save button settings.