I can't be certain here that the network configuration you've supplied will allow nginx to connect to audiobookshelf. If I were you I'd create a named docker network and connect both npm and audiobookshelf to it as described in their best practice documentation: https://nginxproxymanager.com/advanced-config/#best-practice-use-a-docker-network
A 503 indicates that NPM can't see the service you're trying to proxy to.
If the port isn't being explicitly exposed then it won't expose by default. I have my containers set up in a similar way where nginx has it's own network and I attach any containers I want to proxy to the same network. On most cases I'm able to remove the ports section from config and point nginx at the name of the container, and the container's default http port. The way OP originally configured this should be fine IF nginx and audiobookshelf are on the same docker network.
defaultin this example is the actual name of the network, in this case we're modifying the default network.external: truedeclares that the network has already been created,name: scoobydoois a way of customizing the name of the default network.