Alpine.js is great for building interactive interfaces that are purely client-side (think, modals, accordions, menus, etc.). HTMX is more when you need client/server interaction (i.e. AJAX). So things like lazy loading, pagination, infinite scroll, asynchronous save buttons, etc. Most apps need both of these things, though Alpine has a smaller footprint and is probably faster so if you want to do them one at a time, you could start there?
This write up might be a good place to get started: Django, HTMX and Alpine.js: Modern websites, JavaScript optional