how can I prevent the user from filling the field in the serializer and still provide it with a value from the endpoint?
You can’t.
The basic principle is that you have no way to restrict what the user submits. You have no control over the user or the browser. There is nothing you can do to prevent the user from submitting whatever data they wish to submit.
That’s why rule #1 for accepting input is that you never trust what is received from the outside.
The best that can be done is to set things up in such a way that you can (hopefully) tell when something has been modified. But if you need to go that far, then you’re probably better off not even accepting that field as input in the first place.