I’m referring to your JavaScript code that is making the AJAX calls.
When you’re addressing performance-related issues, the first step is always to figure out where the time is being spent. You mentioned that it takes “minutes” to render the map. Your first objective then is to figure out where those minutes are being spent, and attack that piece vigorously.