Location specific permissions

I have a Location model and users that are assigned to a specific location.
And I want to have permissions so that a user from location A can only change things that have to do with that location and not any else.

Is it even possible to do something like this?

Yes, you can add logic in your view to check before allowing a user to modify a location.