[Bug]: IP address exhaustion with container delete --force

[thomasjm]

I have done the following

• I have searched the existing issues
• If possible, I've reproduced the issue using the 'main' branch of this project

Steps to reproduce

If you create containers and then delete them with --force, it seems to leak IP addresses. Eventually you will reach a state where you can't create a new container, with the error "no free indices are available for allocation". Looking through the code, I found this comes from the IP address allocation logic.

I wrote a test script that will create a simple container and then container delete --force it in a loop. You can find it here:

https://gist.github.com/thomasjm/6a2d60e3233fa55bc7765c2426784b33#file-test_ip_exhaustion-sh

There's also another version that does a graceful shutdown, calling container stop and waiting for the container to fully stop before deleting it:

https://gist.github.com/thomasjm/6a2d60e3233fa55bc7765c2426784b33#file-test_ip_exhaustion_graceful-sh

Current behavior

Running the first script will eventually start generating failures after about ~255 iterations. (Because the default system subnet is a /24, so that's how many IP addresses are available.)

...
ip-leak-test-263
  [263] ip-leak-test-263 -> 192.168.64.180/24
ip-leak-test-264
  [264] ip-leak-test-264 -> 192.168.64.191/24
ip-leak-test-265
  [265] ip-leak-test-265 -> 192.168.64.242/24
FAILED at iteration 266: Error: failed to bootstrap container (cause: "unknown: "no free indices are available for allocation"")

Running the graceful shutdown version of the script does not result in failures.

Expected behavior

IP addresses shouldn't leak; creating a container should always succeed if there are <255 currently running containers. (Or whatever the size of the default subnet is.)

Environment

- OS: Version 26.3.1 (a) (25D771280a)
- Xcode: 
- Container: container CLI version 0.10.0 (build: release, commit: 6bdb647)

Relevant log output

N/A

Code of Conduct

• I agree to follow this project's Code of Conduct

[jglogan]

@thomasjm Thanks for filing the issue!

Originally this was because runtime plugin container-runtime-linux (SandboxService) did the allocate() call to container-network-vmnet, and we did a best effort cleanup at shutdown, but if the runtime plugin gets a SIGKILL the deallocation would never run.

Now the allocation happens in the ContainersService of container-apiserver at https://github.com/apple/container/blob/main/Sources/Services/ContainerAPIService/Server/Containers/ContainersService.swift#L435. We'd need to see why force quitting isn't hitting the deallocation path at https://github.com/apple/container/blob/main/Sources/Services/ContainerAPIService/Server/Containers/ContainersService.swift#L1000.

I'm not in love with the centralized approach to allocation, and would prefer that runtimes interact directly with the networks to which they attach instead of having the API service act as a central broker. What I'd like to test is allocating the address in its own long lived XPC connection, such that the connection determines the liveness of the container from the point of view of the network and vice versa.

This approach is more akin to what happens between the network plugin and vmnet, and the runtime plugins and vmnet, for tracking network interface attachments, so it seems that it would be inherently more reliable for IPAM as well.

The runtime plugin would terminate itself if any network plugin connection fails, and the network plugin would deallocate the runtime's IP/MAC for any failed runtime connection.

[jglogan]

@thomasjm I used your first script to test #1395:

  [298] ip-leak-test-298 -> 192.168.64.47/24
ip-leak-test-299
  [299] ip-leak-test-299 -> 192.168.64.48/24
ip-leak-test-300
  [300] ip-leak-test-300 -> 192.168.64.49/24
ip-leak-test-301
  [301] ip-leak-test-301 -> 192.168.64.50/24

=== RESULT: No exhaustion after 301 iterations. IPs are being reclaimed. ===
Done.

[thomasjm]

This sounds great, thanks!