this post was submitted on 15 Jan 2024
7 points (81.8% liked)

homelab

6646 readers
31 users here now

founded 4 years ago
MODERATORS
CMonster@lemmy.ml
7
[WORKAROUND] Nextcloud portforward stops working when it is moved to a bridged network (sh.itjust.works)
submitted 10 months ago* (last edited 10 months ago) by Kalcifer@sh.itjust.works to c/homelab@lemmy.ml
18 comments fedilink hide all child comments
 

cross-posted to: https://sh.itjust.works/post/12856684

I have the following topology:

The device running Nextcloud (snap) used to be connected to Router A, but I have recently added a bridge (Router B) and I moved Nextcloud's device to that bridged network; however, as soon as Nextcloud was moved to Router B, the portforward on Router A seemed to stop working -- as in I cannot connect to nexcloud from the public IP anymore. Bridges operate at layer 2, so this should make no difference whatsoever (this is reflected in the fact that other services (like SSH) still work perfectly fine portforwarded -- it's only Nextcloud that doesn't work), which leads me to think that it is a Layer 7 (i.e. Nextcloud) issue. What's going on here? How can Nextcloud even tell that it's been placed on a bridged network?

EDIT (2024-01-16T00:19Z):

I performed a network capture on the device running Nextcloud, and it appears that it's receiving the incoming request (SYN), and responds appropriately (SYN, ACK), but then Router B responds with Destination unreachable (Network unreachable), which is then, of course, followed by many requests for retransmission as the packets are being dropped. But what's causing the packets to be dropped? Why aren't they making it through the network?

EDIT (2024-01-25T08:37Z):

I’m not 100% sure what the previous problem was, but I think that it had to do with the bridge that I was using – not necessarily that it was broken, but perhaps it was jsut incompatible with the setup in some way. What I ended up doing was buying a different router that supported WDS, and then I created a WDS bridge between the two routers. The network seems to be working reliably, and as expected now.

top 18 comments
sorted by: hot top controversial new old
[–] Max_P@lemmy.max-p.me 5 points 10 months ago (2 children)

but then Router B responds with Destination unreachable (Network unreachable),

That's... interesting. Router B shouldn't be involved at all with this, it should be blindly forwarding the packets. That's a layer 3 error!

How's the bridge set up? Have you made sure router B doesn't do DHCP and doesn't take the IP of router A by accident?

[–] Kalcifer@sh.itjust.works 2 points 10 months ago (1 children)

That’s… interesting. Router B shouldn’t be involved at all with this, it should be blindly forwarding the packets. That’s a layer 3 error!

Indeed! I'm quite stumped.

How’s the bridge set up?

I set it up using this guide.

Have you made sure router B doesn’t do DHCP [...]?

Yup, it's disabled.

Have you made sure router B [...] doesn’t take the IP of router A by accident?

Yep, it's set to be static.

[–] Max_P@lemmy.max-p.me 2 points 10 months ago (1 children)

Hmm, I see, it's not a real L2 bridge, it's a hacky pretend one that relays.

I don't have a solution for this particular situation, but I do have a suggestion on how I would do it:

  • Make B have its own subnet, say, 192.168.1.0/24, assuming that A is on 192.168.0.0/24. Enable DHCP and everything, it's now it's own full network.
  • Make B a client of A with a static IP, like 192.168.0.2. That makes B present on A's network.
  • Add a route on A for B's network: 192.168.1.0/24 via 192.168.0.2.
  • Disable NAT on B, just set A as the default route. Since A can talk to any IP on B, B doesn't need to NAT, A can handle it for both networks.

Now, both routers should be able to exchange traffic while being responsible of their own subnet. The only thing missing would be to handle broadcasts so stuff like Bonjour/Avahi works correctly. But as a whole both layer 2 and 3 would behave a bit more cleanly with less surprises.

I think what's going on is B sorta pretends to be A in some way to do the relaying but something is going wrong.

[–] Kalcifer@sh.itjust.works 1 points 10 months ago (1 children)

Alright, I'll give your suggestion a go.

Make B have its own subnet, say, 192.168.1.0/24, assuming that A is on 192.168.0.0/24. Enable DHCP and everything, it’s now it’s own full network.

Done.

Make B a client of A with a static IP, like 192.168.0.2. That makes B present on A’s network.

Done.

Add a route on A for B’s network: 192.168.1.0/24 via 192.168.0.2.

I think I set this right: Network->Routing->Add->(Interface: wwan, Route type: unicast, Target: 192.168.0.1/24, Gateway: 192.168.1.1)

Disable NAT on B, just set A as the default route.

How would I go about doing this? I can't find any definitive information on how to disable NAT in OpenWRT.

The only thing missing would be to handle broadcasts so stuff like Bonjour/Avahi works correctly.

I do need this. I believe this would then require an mDNS reflector, right (it wasn't required before as relayd was bridging the networks)?

[–] Max_P@lemmy.max-p.me 1 points 10 months ago (1 children)

Sounds about right.

I think I set this right: Network->Routing->Add->(Interface: wwan, Route type: unicast, Target: 192.168.0.1/24, Gateway: 192.168.1.1)

That doesn't seem right. If you're using the exact same subnet numbers I've used for example: that's be target 192.168.1.0/24 (B's network) gateway 192.168.0.2 (B's IP on A's network as a WiFi client).

Router B is on two networks at the same time: its own (192.168.1.1/24) and A's network (192.168.0.2/24).

Router A is only on its own network (192.168.0.1/24) and talks to router B as just a client on its network (192.168.0.2). Whenever it has data to send to the 192.168.1.x network, it sends it to 192.168.0.2 which is on that network and will relay it.

How would I go about doing this? I can’t find any definitive information on how to disable NAT in OpenWRT.

Router B would wan configured as a WiFi client with a static IP of 192.168.0.2/24 and default gateway of 192.168.0.1 (router A). The regular default route will do just fine, as that will cover A's network as well. We'd only need to configure more if there was a third router involved. From there you just need to disable IP masquerading option in Network -> Firewall (you want it unchecked):

Firewall configuration for zone "want"

You don't need masquerade even though it's technically a "wan" because A knows how to send traffic to B's clients, so B itself doesn't have to pretend its clients come from itself.

I do need this. I believe this would then require an mDNS reflector, right (it wasn’t required before as relayd was bridging the networks)?

Correct. I found this: https://blog.christophersmart.com/2020/03/30/resolving-mdns-across-vlans-with-avahi-on-openwrt/

If that proves too complicated, I'd consider trying out the GRE tunnel method your original article suggests as an alternative to relayd. It's kind of like a super basic VPN that I think can be hardware offloaded so I wouldn't expect much of a performance hit, maybe even less than the relayd option.

[–] Kalcifer@sh.itjust.works 1 points 10 months ago (1 children)

Ok, so, I'm ending up with an issue where I can ping Router A from a device on Router B, but I get Destination Port Unreachable if I try to ping a device on Router A. Likewise, I can ping Router B from a device on Router A, but I get Destination Port Unreachable if I try to ping a device on Router B.

I have the route added to Router A (192.168.1.0/24 via 192.168.0.2), I have masquerading turned off for wan on Router B.

[–] Max_P@lemmy.max-p.me 1 points 10 months ago* (last edited 10 months ago) (2 children)

I think you also need to enable full forwarding to and from wan on Router B. I forgot it defaults to not doing that. Set input, output and forward to ACCEPT on Router B on the wan zone, and make sure you also allow forwarding to and from the lan zone. Router A should be fine, I assume A's WiFi and LAN is the same?

Basically now, Router A sends the traffic to B but B doesn't forward it to its LAN. But since we don't have NAT, A's devices addresses B's devices directly, not B itself, and there isn't any connection tracking happening, so it doesn't "remember" to allow the ping response back in. If you WireShark this, I bet B is successfully sending packets to A and A's devices, and A's packets make it all the way to B but B doesn't forward it to its own LAN, and it stops there.

Can you post the output of ip ro and ip a on both routers? (Feel free to redact your public IP/ISP stuff if it shows up)

[–] Kalcifer@sh.itjust.works 1 points 10 months ago* (last edited 10 months ago) (1 children)

Hrm, I still have the same issue. Here's the firewall settings:

lan zone:

  • Input: accept
  • Output: accept
  • Forward: accept
  • Masquerading: false (unchecked)
  • MSS clamping: false (unchecked)
  • Covered Networks: lan
  • Allow forward to destination zones: wan, wan6, wwan
  • Allow forward from source zones: unspecified

wan zone:

  • Input: accept
  • Output: accept
  • Forward: accept
  • Masquerading: false (unchecked)
  • MSS clamping: true (checked)
  • Covered Networks: wan, wan6, wwan
  • Allow forward to destination zones: unspecified
  • Allow forward from source zones: lan

EDIT: I didn't see your edit, as I hadn't refreshed the page.

[–] Max_P@lemmy.max-p.me 1 points 10 months ago* (last edited 10 months ago) (2 children)

~~Interesting, lan zone doesn't allow forward from wan but wan does allow both ways, maybe that's the one missing. I expect OpenWRT to wire it up both ways automatically... OpenWRT is a mystery sometimes.~~

Actually no, both show unspecified. You need both zones to allow both ways from the other zone.

[–] Kalcifer@sh.itjust.works 1 points 10 months ago (1 children)

I'm now encountering another issue where I can't ping any external IP's. I don't mean that DNS isn't resolving (I set that on Router B to use Router A as the DNS resolver), but the I can't ping, say, google.com, for example, from a device on Router B. I can see the ICMP requests in Wireshark, but they just say "no response".

[–] Max_P@lemmy.max-p.me 1 points 10 months ago (1 children)

Erm, okay that's not looking promising. It's starting to look like Router A doesn't like this setup at all. It's not routing B's traffic, possibly because it's not the subnet it expects to serve. Ugh. Check all the options you can in Router A if you can find something that will allow it to work.

You can fairly easily test that by enabling masquerading on B. It'll break most of what we just set up but it'll confirm that.

We still have some options on the OpenWRT side to make it masquerade only public traffic but now I'm wondering if A will even let you port forward to something on B. I would try that now and see if it works.

Is A able to ping B and devices on B, or only on A? A itself has a route for B's subnet right?

[–] Kalcifer@sh.itjust.works 1 points 10 months ago

I really appreciate all the help that you provided in this thread! To simplify the setup, I bought a different primary router, flashed OpenWRT to it, then set up a WDS bridge between it and the other router. So far, I've had no issues, and the setup has been greatly simplified. I'm, of course, still curious as to why the previous setup wasn't working, but at least everything is working now.

[–] Kalcifer@sh.itjust.works 1 points 10 months ago* (last edited 10 months ago)

Alright, I now am able to ping a device on Router B from a device on Router A, but I'm still not able to ping a device on Router A from a device on Router B.

Here's the firewall settings for Router B:

lan zone:

  • Input: accept
  • Output: accept
  • Forward: accept
  • Masquerading: false (unchecked)
  • MSS clamping: false (unchecked)
  • Covered Networks: lan
  • Allow forward to destination zones: wan, wan6, wwan
  • Allow forward from source zones: wan, wan6, wwan

wan zone:

  • Input: accept
  • Output: accept
  • Forward: accept
  • Masquerading: false (unchecked)
  • MSS clamping: true (checked)
  • Covered Networks: wan, wan6, wwan
  • Allow forward to destination zones: lan
  • Allow forward from source zones: lan

EDIT:

Scratch that! apparently it is working. I could've sworn that I checked the ping. Maybe I subconciously applied something else.

[–] Kalcifer@sh.itjust.works 1 points 10 months ago

If you WireShark this, I bet B is successfully sending packets to A and A’s devices, and A’s packets make it all the way to B but B doesn’t forward it to its own LAN, and it stops there.

Yep that's exactly what I see.

Can you post the output of ip ro and ip a on both routers? (Feel free to redact your public IP/ISP stuff if it shows up)

I would only be able to for one router. Router A is a tp-link AX73 which doesn't support OpenWRT. Router B, however, is a tp-link Archer C7 and is flashed with OpenWRT.

[–] BearOfaTime@lemm.ee 0 points 10 months ago* (last edited 10 months ago) (1 children)

Yea, B is still acting like a router if it's creating those messages.

I wonder if it's bridging mode is actually bridging, or if it's doing something weird.

OP - how is router B cabled? If it's a typical consumer device it has one uplink port, which you wouldn't use (usually) for a bridge setup. Because really a bridge today is just a switch (they were first called switching bridges, and we got lazy and just say switch). Use of an uplink port implies routing, not sure if the router changes its config for bridge mode. I just don't use that port when I need switching.

[–] Kalcifer@sh.itjust.works 1 points 10 months ago

OP - how is router B cabled?

For the bridge, it's set up over a wifi connection to Router A. For the Nextcloud server, it's just connected to one of the LAN ports on Router B.

[–] BearOfaTime@lemm.ee 0 points 10 months ago* (last edited 10 months ago) (1 children)

After looking at the docs you linked, I'm not sure why you'd need to bridge the wifi to the LAN - it already is via the inbuilt switch ports (it's been a while but when I tinkered with WRT it was for consumer wireless routers, which are a router, a switch, and a wireless access point rolled into one.).

Just disable DHCP, DNS, and connect one of the network ports (NOT the uplink) to the network that Router A is on, and Bob's your uncle. (

The switch part will provide all the bridging (since that's what switches do) and the wireless is already bridged to that switch.

So I don't even see a need for relayd software and it's config. The router, with DHCP and DNS off, the uplink port not used, is essentially just being a switching bridge. If you can disable the uplink port that would be even better, but so long as it's set to use DHCP for an address, it'll never get one. Or you could set it to a private class that you won't use: say your network is 192.168.x.x, set the uplink to use 10.0.0.x. You'll never have traffic wanting to use that network.

Edit: Ah, I see. You're using 2 wifi access points as a bridge to each other. Hmm, yea, that's not typically a thing (though I've seen it more recently) which is why you're using relayd.

Definitely looks like relayd is playing pretend at bridging somehow, but not perfectly. Is there a forum for relayd?

[–] Kalcifer@sh.itjust.works 1 points 10 months ago

Ah, I see. You’re using 2 wifi access points as a bridge to each other.

Yeah, this is a requirement for how I am trying to implement it.