Viktor Dukhovni
On Thu, Nov 26, 2020 at 02:12:49PM +0100, Magnus Therning wrote:
After some detours I found out that it's actually not a network issue, but rather that the process runs out of filedescriptors. Using =lsof= I can see that it doesn't seem to close /any/ sockets at all, instead they get stuck in a =CLOSE_WAIT= state:
#+begin_example COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME wd-stats 88674 magnus 23u IPv4 815196 0t0 TCP ip-192-168-0-9.eu-central-1.compute.internal:60624->52.119.188.213:https (CLOSE_WAIT) wd-stats 88674 magnus 24u IPv4 811362 0t0 TCP ip-192-168-0-9.eu-central-1.compute.internal:43482->52.119.189.184:https (CLOSE_WAIT) wd-stats 88674 magnus 25u IPv4 811386 0t0 TCP ip-192-168-0-9.eu-central-1.compute.internal:60628->52.119.188.213:https (CLOSE_WAIT) wd-stats 88674 magnus 26u IPv4 813527 0t0 TCP ip-192-168-0-9.eu-central-1.compute.internal:43486->52.119.189.184:https (CLOSE_WAIT) ... #+end_example
How many such still open file descriptors did you find?
Hundreds of them.
(If you run "lsof -n -P -i tcp -a -p $pid", it'll produce the output faster, reporting only sockets).
Contrary to other replies, indeed the sockets above are NOT closed in your process, otherwise they'd not be associated with a file descriptor and would just show up in "netstat", but not "lsof" output.
I don't know what happens inside Amazonka, but typically clients doing many concurrent HTTPS calls employ a TlsManager that maintains a connection pool, and would avoid opening too many concurrent connections, but would also keep a limited number of connections open for more requests.
After I reported it to the Amazonka project[1] I found out that it most likely is a known issue[2]. I have still to confirm that the fix for [2] solves the issue I'm seeing. [1]: https://github.com/brendanhay/amazonka/issues/608 [2]: https://github.com/brendanhay/amazonka/issues/490 /M -- Magnus Therning OpenPGP: 0x927912051716CE39 email: magnus@therning.org twitter: magthe http://magnus.therning.org/ I am always doing that which I cannot do, in order that I may learn how to do it. — Pablo Picasso