CLX can't connect to an X server via inet sockets since about 2018-02
Describe the bug
Between the 2018-01 and 2018-02 snapshots, CLX became unable to connect to an X server over inet sockets.
This prevents running CLX programs over networks, including through ssh's X11 forwarding.
The root cause is that XLIB::HOST-ADDRESS stopped being able to resolve hostnames, and now always errors.
This defect appears to be reproducible in each subsequent snapshot through 2021-07.
To Reproduce
There are multiple ways to reproduce.
- [SSH X11 forwarding]
- Assume all of
- an X server running on host-A,
- CMUCL with CLX installed on host-B,
- sshd running on host-B, configured to enable X11 forwarding,
- [Optional] some non-CLX X clients installed on host-B, for verifying that SSH's X11 forwarding works.
- [Note that host-B might be the same host as host-A.]
- On host-A, ssh to host-B, e.g., with
ssh -X name-for-host-B
- In the ssh session on host-B, inspect the value of DISPLAY. It ought to be something like
localhost:NN.0
, whereNN
is some decimal number. Take note ofNN
. - [Optional] Verify that SSH's X11 forwarding works for some non-CLX X client. Anything will do; we'll use
xdpyinfo
below. - Start CMUCL, e.g.,
./bin/lisp -nositeinit -noinit
in some directory containing a snapshot from 2018-02 or later. - At the REPL, load CLX and try to connect to the X server specified by the DISPLAY environment variable, using
NN
discovered above:(require :clx) (xlib:open-display "localhost" :display NN :protocol :internet)
- (Note that when
DISPLAY=localhost:NN.0
, XLIB:OPEN-DEFAULT-DISPLAY calls XLIB:OPEN-DISPLAY with such arguments. This reproduction calls OPEN-DISPLAY explicitly for clarity.)
- (Note that when
- This will produce an error. Note that the erroring function is XLIB::HOST-ADDRESS.
- Assume all of
- [Connecting to an X server without SSH's X11 forwarding]
- (Note that this is an insecure way for an X client to communicate with an X server in general. I describe it only because this reproduction doesn't involve configuring ssh, so might be easier to test.)
- Assume all of
- a reasonably complete X11 installation on some host
- CMUCL with CLX installed that same host.
- Identify a display number that's available for use on the host. Let's say that
5
is available. - Create a fresh Xauthority file to use solely for this testing
touch ~/Xauthority.cmucl-clx && mcookie | sed -e 's/^/add :5 . /'|xauth -q -f ~/Xauthority.cmucl-clx
- Start a fresh X server using the selected display number, the Xauthority file just created, and ensuring that the X server is listening for inet connections.
- If you're already in an X session of some kind, start a nested X server such as Xephyr
Xephyr :5 -listen inet -auth ~/Xauthority.cmucl-clx &
or XnestXnest :5 -listen inet -auth ~/Xauthority.cmucl-clx &
- [Untested] If you're at the OS console (i.e., not inside an X session), something like this might do
xinit -- :5 -listen inet -auth ~/Xauthority.cmucl-clx
- If you're already in an X session of some kind, start a nested X server such as Xephyr
- From some OS prompt (it doesn't need to be a descendant of xinit), start CMUCL, ensuring that the XAUTHORITY environment variable specifies the Xauthority file created for testing, e.g.,
XAUTHORITY=~/Xauthority.cmucl-clx ./bin/lisp -nositeinit -noinit
in some directory containing a snapshot from 2018-02 or later. - At the REPL, load CLX and try to connect to the X server you've just started
(require :clx) (xlib:open-display "localhost" :display 5 :protocol :internet)
- This will produce an error. Note that the erroring function is XLIB::HOST-ADDRESS.
Here's a transcript of the failure using the ssh X11 forwarding approach. This example starts on a Darwin host running Xquartz (that's host-A for the above), and "debian10" is a host running Debian 10 (that's host-B).
$ ssh -X debian10
Linux debian 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Aug 14 09:53:43 2021 from 10.0.2.2
$ echo $DISPLAY
localhost:10.0
$ xdpyinfo | sed -n '/^$/q; p' # we don't care about screen geometries
name of display: localhost:10.0
version number: 11.0
vendor string: The X.Org Foundation
vendor release number: 11907000
X.Org version: 1.19.7
maximum request size: 16777212 bytes
motion buffer size: 256
bitmap unit, bit order, padding: 32, LSBFirst, 32
image byte order: LSBFirst
number of supported pixmap formats: 7
supported pixmap formats:
depth 1, bits_per_pixel 1, scanline_pad 32
depth 4, bits_per_pixel 8, scanline_pad 32
depth 8, bits_per_pixel 8, scanline_pad 32
depth 15, bits_per_pixel 16, scanline_pad 32
depth 16, bits_per_pixel 16, scanline_pad 32
depth 24, bits_per_pixel 32, scanline_pad 32
depth 32, bits_per_pixel 32, scanline_pad 32
keycode range: minimum 8, maximum 255
focus: window 0xe00024, revert to None
number of extensions: 23
Apple-DRI
Apple-WM
BIG-REQUESTS
DAMAGE
DOUBLE-BUFFER
GLX
Generic Event Extension
MIT-SCREEN-SAVER
MIT-SHM
Present
RANDR
RENDER
SECURITY
SGI-GLX
SHAPE
SYNC
X-Resource
XC-MISC
XFIXES
XINERAMA
XInputExtension
XKEYBOARD
XVideo
default screen number: 0
number of screens: 1
$ cd ~/cmucl-snapshots/cmucl-2018-02
$ ./bin/lisp -nositeinit -noinit
CMU Common Lisp snapshot-2018-02 (21C Unicode), running on debian
With core: /net/_gateway/Users/kreuter/cmucl-snapshots/cmucl-2018-02/lib/cmucl/lib/lisp-sse2.core
Dumped on: Sat, 2018-02-10 18:54:52-05:00 on lorien3
See <http://www.cmucl.org/> for support information.
Loaded subsystems:
Unicode 1.29 with Unicode version 6.2.0
Python 1.1, target Intel x86/sse2
CLOS based on Gerd's PCL 2010/03/19 15:19:03
* (require :clx)
; Loading #P"/net/_gateway/Users/kreuter/cmucl-snapshots/cmucl-2018-02/lib/cmucl/lib/subsystems/clx-library.sse2f".
; [GC threshold exceeded with 12,004,720 bytes in use. Commencing GC.]
; [GC completed with 3,404,048 bytes retained and 8,600,672 bytes freed.]
; [GC will next occur when at least 15,404,048 bytes are in use.]
("CLX")
* (xlib:open-display "localhost" :display 10 :protocol :internet)
Error in function XLIB::HOST-ADDRESS: Unknown host "localhost"
[Condition of type SIMPLE-ERROR]
Restarts:
0: [ABORT] Return to Top-Level.
Debug (type H for help)
(XLIB::HOST-ADDRESS "localhost" #<unused-arg>)
Source: Error finding source:
Error in function DEBUG::GET-FILE-TOP-LEVEL-FORM: Source file no longer exists:
target:clx/dependent.lisp.
0] :back
0: (XLIB::HOST-ADDRESS "localhost" #<unused-arg>)
1: (XLIB::GET-BEST-AUTHORIZATION "localhost" 10 :INTERNET)
2: (XLIB:OPEN-DISPLAY "localhost" :DISPLAY 10 :PROTOCOL ...)
3: (INTERACTIVE-EVAL (XLIB:OPEN-DISPLAY "localhost" :DISPLAY 10 :PROTOCOL ...))
4: (LISP::%TOP-LEVEL)
5: ((LABELS LISP::RESTART-LISP SAVE-LISP))
Expected behavior
Here's a transcript of the behavior under the 2018-01 snapshot, representing a successful connection to the X server via an inet socket:
$ ssh -X debian10
Linux debian 4.19.0-17-amd64 #1 SMP Debian 4.19.194-3 (2021-07-18) x86_64
The programs included with the Debian GNU/Linux system are free software;
the exact distribution terms for each program are described in the
individual files in /usr/share/doc/*/copyright.
Debian GNU/Linux comes with ABSOLUTELY NO WARRANTY, to the extent
permitted by applicable law.
Last login: Sat Aug 14 09:53:58 2021 from 10.0.2.2
$ cd ~/cmucl-snapshots/cmucl-2018-01
$ ./bin/lisp -nositeinit -noinit
CMU Common Lisp snapshot-2018-01 (21C Unicode), running on debian
With core: /net/_gateway/Users/kreuter/cmucl-snapshots/cmucl-2018-01/lib/cmucl/lib/lisp-sse2.core
Dumped on: Sun, 2018-01-14 22:06:41-05:00 on lorien3
See <http://www.cmucl.org/> for support information.
Loaded subsystems:
Unicode 1.29 with Unicode version 6.2.0
Python 1.1, target Intel x86/sse2
CLOS based on Gerd's PCL 2010/03/19 15:19:03
* (require :clx)
; Loading #P"/net/_gateway/Users/kreuter/cmucl-snapshots/cmucl-2018-01/lib/cmucl/lib/subsystems/clx-library.sse2f".
; [GC threshold exceeded with 12,009,288 bytes in use. Commencing GC.]
; [GC completed with 3,458,904 bytes retained and 8,550,384 bytes freed.]
; [GC will next occur when at least 15,458,904 bytes are in use.]
("CLX")
* (xlib:open-display "localhost" :display 10 :protocol :internet)
#<XLIB:DISPLAY localhost:10 (The X.Org Foundation R11907000)>
Fix for this defect
By inspection, one way to fix this is to restore XLIB:HOST-ADDRESS in dependent.lisp to its definition prior to commit c8ebc741 (below). Note that when doing so interactively, it's necessary also to re-evaluate (DEFUN GET-BEST-AUTHORIZATION ...) in display.lisp, as the compiler will have previously compiled GET-BEST-AUTHORIZATION with knowledge that the faulty HOST-ADDRESS definition can't return anything.
#+CMU
(defun host-address (host &optional (family :internet))
;; Return a list whose car is the family keyword (:internet :DECnet :Chaos)
;; and cdr is a list of network address bytes.
(declare (type stringable host)
(type (or null (member :internet :decnet :chaos) card8) family))
(declare (clx-values list))
(labels ((no-host-error ()
(error "Unknown host ~S" host))
(no-address-error ()
(error "Host ~S has no ~S address" host family)))
(let ((hostent (ext:lookup-host-entry (string host))))
(when (not hostent)
(no-host-error))
(ecase family
((:internet nil 0)
(unless (= (ext::host-entry-addr-type hostent) 2)
(no-address-error))
(append (list :internet)
(let ((addr (first (ext::host-entry-addr-list hostent))))
(list (ldb (byte 8 24) addr)
(ldb (byte 8 16) addr)
(ldb (byte 8 8) addr)
(ldb (byte 8 0) addr)))))))))
Desktop (please complete the following information):
-
OS: All of the above is captured on Debian 10 running as a qemu guest of a Darwin qemu host. The X server I'm connecting to is Xquartz running on the Darwin qemu host.
-
Version: I believe the regression was introduced in January 2018, it appears to be present as recently as the 2021-07 snapshot.
Additional context
XLIB:OPEN-DISPLAY searches an Xauthority file for a key that matches the desired host and display number before opening any socket. Hosts are matched by network address, not by name. When OPEN-DISPLAY's PROTOCOL argument is :LOCAL, the network address of "127.0.0.1" is used unconditionally, so no host name resolution is involved. But when the protocol is :INTERNET, a host name argument to OPEN-DISPLAY must be resolved to a network address.
XLIB::HOST-ADDRESS is CLX's internal interface for hostname resolution, so it needs to work correctly in order for OPEN-DISPLAY to work with PROTOCOL :INTERNET.
The bug in HOST-ADDRESS is easy to identify: the import of telent-clx at commit c8ebc741 in January 2018 substantially changed the definition of HOST-ADDRESS in dependent.lisp. The telent-clx (and now sharplispers' clx) version of this function for CMUCL includes the names for 3 different interfaces to resolve a hostname, but hides each interface behind a different feature check (and doesn't include any read-time conditional for the case that none of the 3 features is present). I don't see anyplace where CMUCL might arrange for any of the 3 features when building CLX, and so HOST-ADDRESS always gets compiled to signal an error.
A git "blame" view of the current definition of HOST-ADDRESS in the sharplispers' CLX repo shows that the peculiar read-time conditionalization was present at the earliest commit of the telent-clx ancestor of sharplispers' clx; see line 1470 and following here:
https://github.com/sharplispers/clx/blame/master/dependent.lisp
Because the "Initial commit" annotation at line 1481 indicates that telent-clx always had some read-time conditionals in the definition of HOST-ADDRESS, it seems possible that telent-clx and its descendants may never have been able to connect to X servers over inet sockets on CMUCL.
As mentioned above, simply restoring HOST-ADDRESS to its definition prior to c8ebc741 fixes things. Please let me know if you'd prefer that this fix be incorporated into the sharplispers clx project upstream before addressing this in CMUCL's CLX distribution.