Tests occasionally hang in CI
Let's consolidate the various discussions about this issue here.
When running in CI, some regression tests occasionally stop making progress and sit there until the job times out (1 hour). If this happens, a single retry is typically enough to get the job to finish. This happens the most frequently with CCL:
- https://gitlab.common-lisp.net/asdf/asdf/-/jobs/11721
- https://gitlab.common-lisp.net/asdf/asdf/-/jobs/13812
- https://gitlab.common-lisp.net/asdf/asdf/-/jobs/13768
- https://gitlab.common-lisp.net/asdf/asdf/-/jobs/13987
It also happens occasionally with ECL:
- https://gitlab.common-lisp.net/asdf/asdf/-/jobs/13499
- https://gitlab.common-lisp.net/asdf/asdf/-/jobs/13441
- https://gitlab.common-lisp.net/asdf/asdf/-/jobs/15746
CCL seems to stop making progress at https://gitlab.common-lisp.net/asdf/asdf/-/blob/3e24bb1cda1a5de7349604841588d08681ebd6c6/test/test-program.script#L120 and ECL seems to stop making progress at https://gitlab.common-lisp.net/asdf/asdf/-/blob/3e24bb1cda1a5de7349604841588d08681ebd6c6/test/test-run-program-unix.script#L447
I've been unable to reproduce locally by running make td l=ccl t=test-program.script
over 400 times (after removing the --pull always
from the Makefile so I don't run afoul of Docker Hub rate limiting) under both light and heavy loads. This suggests it's something specific about the CI runner environment.
Both of them hang when child processes are started. One known issue about running in Docker is that the default configuration sets an extremely high ulimit -n
(file descriptors). We previously had issues with this and SBCL because SBCL iterates over all possible file descriptors and tries to close them. This resulted in new processes taking a very long time to spawn (see: c248f41d). It would not surprise me if we're running into a similar issue here with a combination of host characteristics and Docker defaults interacting badly with some implementations.
I propose we focus on CCL first since that happens the most frequently. Plus if we get it fixed by changing the test environment, it may fix ECL as well.