45
I was exploring direct links between machines, and basically failed to break something.
I assigned IP address 192.168.0.1/24 to eth0 in two ways.
A. Adding 192.168.0.1/24 as usual
# ip addr add 192.168.0.1/24 dev eth0
# ping -c 1 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
64 bytes from 192.168.0.2: icmp_seq=1 ttl=64 time=0.051 ms
***
192.168.0.2 ping statistics
***
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.051/0.051/0.051/0.000 ms
#
B: Adding 192.168.0.1/32 and adding a /24 route
# ip addr add 192.168.0.1/32 dev eth0
# # 192.168.0.2 should not be reachable.
# ping -c 1 192.168.0.2
ping: connect: Network is unreachable
# # But after adding a route, it is.
# ip route add 192.168.0.0/24 dev eth0
# ping -c 1 192.168.0.2
PING 192.168.0.2 (192.168.0.2) 56(84) bytes of data.
64 bytes from 192.168.0.2: icmp_seq=1 ttl=64 time=0.053 ms
***
192.168.0.2 ping statistics
***
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.053/0.053/0.053/0.000 ms
#
Does this mean that adding an IP address with prefix is just a shorthand for adding the IP address with /32 prefix and adding a route afterwards? That is, does the prefix length has no meaning and the real work is done by the route entries?
Or is there any functional difference between the two methods?
Here is another case, these two nodes can reach each other via direct connection (no router in between) but don't share a subnet.
Node 1:
# ip addr add 192.168.0.1/24 dev eth0
# ip route add 192.168.1.0/24 dev eth0
# # Finish the config on Node B
# nc 192.168.1.1 8080 <<< "Message from 192.168.0.1"
Response from 192.168.1.1
Node 2:
# ip addr add 192.168.1.1/24 dev eth0
# ip route add 192.168.0.0/24 dev eth0
# # Finish the config on Node A
# nc -l 0.0.0.0 8080 <<< "Response from 192.168.1.1"
Message from 192.168.0.1
Can't test right now, but I have a strong suspicion you will have trouble getting IP broadcast to work. Normally broadcast address is calculated by setting all bits after the network prefix to 1, but your computer believes to be in a /32 "network". It won't broadcast over routes that are not part of its network.
And even if you calculate the broadcast address successfully (maybe the software you use has /24 hardcoded for whatever reason), no computer configured with a /32 address will receive it - 192.168.0.255 is not within the 192.168.0.1/32 network, so it will probably get forwarded according to your routes if you have forwarding enabled (except it shouldn't in this case with one network interface, because you never send packets back the way they came from)
Just did some basic testing on broadcast addresses using socat, broadcast is not working at all with /32 addresses. With /24 addresses, broadcast only reaches nodes that share a subnet. Nodes that don't share the subnet aren't reachable by broadcast even when they're reachable via unicast.
Edit1: Did more testing, it seems like broadcast traffic ignores routing tables.
On 192.168.0.2, I am running
socat -u udp-recv:8000,reuseaddr -to print UDP messages.Case 1: add
192.168.0.1/24Case 2: Same as above but delete 192.168.0.0/24 route
One of the functions of a router is splitting broadcast domains. You would not expect a broadcast to reach a different subnet.
IP and Routing is layer 3, broadcast is layer 2 with Mac addresses being shared within a broadcast domain (often a vlan/lan) and the only requirement for layer 2 is a switch you don't need routers. Devices on a lan talk only via switches which switch based on Mac address tables. You don't learn Mac addresses of devices past your broadcast domain, that's what a router handles.
So in network practice (nothing Linux related) if you are on a broadcast network that's a /24 subnet, what should happen is all devices within that subnet talk to each other without using a router, instead they learn a mac address and the associated ip from a broadcast from the device which owns it.
If you tell your device that it's only on a /32 then it should discard every arp it hears as invalid. Which means it won't learn any neighbouring lan devices.
While your network on your single device with the /32 probably works ok to get to other networks (routed networks like internet or other vlans), because other networks ask the router, and the router probably learned your mac and ip on whatever vlan/interface your device is connected via.
But unless you're trying to do something unconventional, devices on a lan should match the routers expected subnet. This way devices can trust their assumption that within their subnet they communicate to other local devices by learning other network devices network address via arp, and communicate directly in unicast via learned ips from that arp. If it's outside the subnet they then look to the gateway. They trust the gateway. The gateway should route to the right interface or next hop.
If you really wanted to make this work though, usually routers can proxy arp. So in this case, you tell the router to 'oroxy' and broadcast your arp to other devices. Those devices on your lan looking for your ip will find the routers Mac address, then using destination network address translation you can redirect the incoming connection from a lan device to your device via your router. Then your /32 ip can probably work. Usually this is done when someone has put a static ip on a device with a wrong subnet ip on a vlan with another subnet. So the device which arps is ignored by the router and the other network devices. If you use the router to proxy arp you can basically give the local lan devices an ip to hit that they expect, which then you can translate to the misconfigured device. This generally is considered a bandaid solution temporary until a vendor or technician can fix their misconfiguration. I do not recommend.
@biscuitswalrus @akash_rawal
Layer 3 decides if and where broadcasts go.
Layer 2 is mainly point to point.
Layer3 decides where broadcasts stop (at the boundary between two networks, i.e. a router)
Layer2 is where broadcasts go.