At least two problems with CLM's string interning
Describe the bug
In CLM, when strings are communicated between Lisp and motifd
, any string that's STRING=
to an element in a precomputed global string table is communicated by the index into the table, rather than by transmitting the characters. Lisp and motifd
have corresponding string tables. This "string interning" is implicit, and used for all strings communicated in either direction.
This approach creates at least the following problems:
-
On the Lisp side, any CLM API call that returns a string might return an element of the string table, but the API client won't know that in general. If the client program goes on to destructively modifies the string, Lisp and
motifd
will get out of sync. So, in effect, CLM interfaces that return strings have an undocumented constraint that strings returned from CLM interfaces should not be modified. I doubt this was intended. -
On the
motifd
side, one of the code paths that's supposed to construct and return anXmString
doesn't handle the CLM wire encoding for interned strings. CLM programs that trigger that code path can either evince unintended behavior or crashmotifd
.
To Reproduce
To demonstrate the first problem, we first obtain an element of XTI::*TOOLKIT-STRING-TABLE*
via some innocuous-looking CLM interface, and then we destructively modify the string. Because motifd
"tokenizes" all strings it sends to Lisp, an input field's input is one way to get motifd
to transmit a string_token
to Lisp. To trigger the unwanted behavior:
- Evaluate the following
DEFUN
. - Call the program, e.g.,
(input)
. - Type the string that's an element of the string table, e.g.,
"foreground"
, into the input field, without the quotation marks, and press enter. - Note that Lisp output indicates that TEXT-GET-STRING returned the string
"foreground"
, and that that string is a member of Lisp's string table. - Clear out the input field (e.g., with backspace).
- Type in the string
"foreground"
again, without quotation marks, and press enter. - Note that Lisp output indicates that TEXT-GET-STRING now returns the string
"%oreground"
, and that that string is present in Lisp's string table.
I believe the analogous consequences can occur using other CLM interfaces that return strings originating from motifd
. In short, modifying strings that CLM interfaces return can put Lisp into a state where transmissions from motifd
don't get represented correctly in Lisp.
(defun input ()
(labels ((create-input-field (parent)
(let ((text (create-text-field parent "input")))
(manage-child text)
text))
(enter-callback (widget call-data &rest client-data)
(declare (ignore call-data client-data))
(let* ((string (text-get-string widget))
(initial (copy-seq string))
(position (position string xti::*toolkit-string-table*)))
(setf (char string 0) #\%)
(format t "~&initial: ~S, position: ~A; final: ~S"
initial position string)
(finish-output)))
(main ()
(let* ((shell (create-application-shell))
(text (create-input-field shell)))
(add-callback text :activate-callback #'enter-callback)
(realize-widget shell))))
(run-motif-application
#'main :application-name "input" :application-class "Input")))
For the second problem, we want Lisp to communicate a string that it will tokenize in a context where motifd
will build an XmString
using inputs from the wire. An XmLabel
's labelString
resource is required to be an XmString
, so supplying a string that Lisp will tokenize for the :LABEL-STRING
argument will trigger this. To reproduce unwanted behavior:
- Evaluate the following
DEFUN
. - Call the program with a string that's
STRING=
to the first element in the string table, i.e.,(memo "accelerator")
. - When I call the program this way, I get a window containing the string
"message"
. (I believe it would be possible for the invocation to crashmotifd
instead of showing the wrong string, as well.)
Next,
- Call the program with some extra widget initargs after the label string
(memo "accelerator" :foreground #xff0000)
. - When I call the program this way,
motifd
always segfaults. (In principle I think it'd be possible formotifd
not to segfault, though ISTM unlikely.)
Finally,
- If desired, run the program with any string that's not in the string table to verify that the program isn't itself at fault, e.g.,
(memo "foo" :foreground #xff0000)
.
What's going on: the string, "accelerator"
gets transmitted as a tag plus a string_token
. toolkit_read_value
decomposes the tag and string_token
, then calls message_read_xm_string
, with the tag and the string_token
. message_read_xm_string
doesn't have a case for string_token_tag
, so it tries to read an OID
off the wire using message_read_oid
.
In the first case (no arguments after the "labelString"
, message_read_oid
will read four (possibly uninitialized) bytes from a buffer, and will likely return NULL
. NULL
isn't a valid XmString
, and in this case the widget behaves as if no "labelString"
was supplied. (I don't know whether Xt or Motif guarantees that behavior, though.)
In the second case, after the tagged string_token
representing "accelerator"
, message_read_xm_string
reads the tagged string_token
representing "foreground"
; this will return NULL
. Evidently, NULL
is an invalid resource name for XtCreateManagedWidget
.
(defun memo (&optional (message (error "Need a message to display.")) &rest label-initargs)
(labels ((main (message)
(let* ((shell (create-application-shell))
(msg (apply
#'create-managed-widget
"message" :label shell :label-string message
label-initargs)))
(realize-widget shell))))
(run-motif-application
#'main :init-args (list message)
:application-name "memo" :application-class "Memo")))
Expected behavior
For the first problem, general I'd expect Lisp interfaces that return strings either to allow clients to destructively modify those strings freely, or else to document any restrictions on destructive modification. Further, IMO, there must be good justifications for any such restrictions; and I don't think there are in this case.
The second problem is just a common-or-garden bug, missing coverage for a wire protocol state. However, the way I propose addressing the first problem means that the offending protocol state can't happen.
Fix for this defect
Working on it. (I'd actually already done some of the work, as I'd noticed a peculiar inefficiency in this aspect of CLM some time ago, though I hadn't noticed these problems.) ETA later today or tomorrow.
Desktop (please complete the following information):
A Debian 10 virtual machine.
Additional context
I observe that all the strings in the string table are resource names (analogous to slot names & initargs in CLOS). If that's all that the string table is for, then I don't believe the interning needs to be intertwined with string I/O. More results shortly, I hope.