Selfhosted

Both your containers need share at least one network. It looks like have created a external network. Good job. The next step is to configure your container to join the network. Do something like this:

services:
  nginx-proxy-manager: 
     ....
     networks:
       - nginx-proxy-manager_default # or just nginx
      ...

networks:
  nginx:
    name: nginx-proxy-manager_default
    external: true  

Don't forget to also add the network in your audiobookshelf service.

Maybe this helps: https://docs.docker.com/compose/how-tos/networking/#use-an-existing-network

I don’t see an immediate issue but I do see some general issues.

Unless you own local.com, don’t use it.

While you could use .local as your tld, I wouldn’t do that either. You can buy a domain name for cheap and really that’s the way to go.

Also, reference your FQDN and not your hostname. Don’t expect hosts to fill in the blanks.

In the compose file for Audiobookshelf, you need to add the service itself to the network you defined at the bottom. Something like:

services:
  audiobookshelf:
    ...
    networks:
      - default
      - nginx

(The default network is to keep it in the network that is created on docker compose up).

Are you sure both containers (npm and audiobookshelf) are in the same network? I see you don't specify the network in the NPM compose file.

Check that like this:

$ docker inspect container-name| grep -A3 Networks
            "Networks": {
                "network_name": {
                    "IPAMConfig": null,
                    "Links": null,

See if both your containers are in the same network - they should be.

I'm pretty sure you have to specify a different port. Audiobookshelf uses port 80 by default - which is already taken by NPM - and since you're not mapping it to any other port those services will conflict.

Map the port to something easy to remember in the compose file. Not only is it required in your setup but it is good practice. If you have to manage a lot of services, it is hard to keep track of all the ports they use if you keep them on their default values. I personally chose a port range (like 12300 -> 12399) and map all of my services respectively (Komodo to 12300, Authentik to 12301, etc.).

Also remember to update the port in your NPM config.

TL;DR: Audiobookshelf's default port is 80, which conflicts with NPM. Change it.

Just to make sure, your issue is :

But I cannot access https://audiobookshelf.local.com/ or http://audiobookshelf.local.com/ (it automatically forwards to HTTPS).

If so, well, you're forcing SSL, which means it'll automatically transmit http request to https. If you don't want that, turn it off. Anyway, it's going to be used only locally, correct?

Based on you screenshot from the NPM Dashboard there seems to be something wrong. In the setup window you show that you forward the traffic with http and port 80, in the dashboard screenshot you forward the traffic with https and port 80.

Just skip http and self signed certificates all together. Modern Browsers make it a pain to use non https sites. A simple domain setup with dns acme challenge is a little bit of a hassle but worth the hour(s) of invested time. Especially with npm were it is a set and forget option.

Does pihole support wildcard dns entries yet? To my knowledge the gui only supports single entries so that you have to enter every subdomain manually in pihole that you want to have forwarded. Workaround would be to use a dnsmasq config file or use something else like addguard.

I can't be certain here that the network configuration you've supplied will allow nginx to connect to audiobookshelf. If I were you I'd create a named docker network and connect both npm and audiobookshelf to it as described in their best practice documentation: https://nginxproxymanager.com/advanced-config/#best-practice-use-a-docker-network

A 503 indicates that NPM can't see the service you're trying to proxy to.

Start a console on NPM. See if you can curl or wget the target.

Start with IPs and ports. Then move to DNS.

NPM's configuration never felt terribly comfortable to me.

Another common problem is to unintentionally set your container up where URLs require SSL. When you hit the container page the URL call backs can end up with HTTPS.

Are both containers connected to the same custom docker network? NPM might need to be connected to both a custom docker network and an bridge\external network.