README 6.15 KB
Newer Older
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
######################################################################
## 
##    Copyright (C) 2001,2000, 2003
##    Department of Computer Science, University of Tromsø, Norway
## 
## Filename:      README
## Author:        Frode Vatvedt Fjeld <frodef@acm.org>
## Created at:    Wed Dec  8 15:35:53 1999
## Distribution:  See the accompanying file COPYING.
##                
## $Id: README,v 1.21 2003/12/11 11:28:21 frodef Exp $
##                
######################################################################

Binary-types is a Common Lisp package for reading and writing binary
files. Binary-types provides macros that are used to declare the
mapping between lisp objects and some binary (i.e. octet-based)
representation.

Supported kinds of binary types include:

 * Signed and unsigned integers of any octet-size, big-endian or
   little-endian. Maps to lisp integers.

 * Enumerated types based on any integer type. Maps to lisp symbols.

 * Complex bit-field types based on any integer type. Sub-fields can
   be numeric, enumerated, or bit-flags. Maps to lisp lists of symbols
   and integers.

 * Fixed-length and null-terminated strings. Maps to lisp strings.

 * Compound records of other binary types. Maps to lisp DEFCLASS
   classes or, when you prefer, DEFSTRUCT structs.

Typically, a complete binary record format/type can be specified in a
single (nested) declaration statement. Such compound records may then
be read and written with READ-BINARY and WRITE-BINARY.

Binary-types is *not* helpful in reading files with variable
bit-length code-words, such as most compressed file formats. It will
basically only work with file-formats based on 8-bit bytes
(octets). Also, at this time no floating-point types are supported out
of the box.

Binary types may now be declared with the DEFINE-BINARY-CLASS macro,
which has the same syntax (and semantics) as DEFCLASS, only there is
an additional slot-option (named :BINARY-TYPE) that declares that
slot's binary type. Note that the binary aspects of slots are *not*
inherited (the semantics of inheriting binary slots is unclear to me).

Another slot-option added by binary-types is :MAP-BINARY-WRITE, which
names a function (of two arguments) that is applied to the slot's
value and the name of the slot's binary-type in order to obtain the
value that is actually passed to WRITE-BINARY. Similarly,
:MAP-BINARY-READ takes a function that is to be applied to the binary
data and type-name when a record of that type is being read.  A
slightly modified version of :map-binary-read is
:MAP-BINARY-READ-DELAYED, which will do essentially the same thing as
:map-binary-read, only the mapping will be "on-demand": A slot-unbound
method will be created for this purpose.

A variation of the :BINARY-TYPE slot-option is :BINARY-LISP-TYPE,
which does everything :BINARY-TYPE does, but also passes on a :TYPE
slot-option to DEFCLASS (or DEFSTRUCT).  The type-spec is inferred
from the binary-type declaration. When using this mechanism, you
should be careful to always provide a legal value in the slot (as you
must always do when declaring slots' types). If you find this
confusing, just use :BINARY-TYPE.

Performance has not really been a concern for me while designing this
package. There's no obvious performance bottlenecks that I know of,
but keep in mind that all "binary" reads and writes are reduced to
individual 8-bit READ-BYTEs and WRITE-BYTEs. If you do identify
particular performance bottlenecks, let me know.

The included file "example.lisp" demonstrates how to use this
package. To give you a taste of what it looks like, the following
declarations are enough to read the header of an ELF executable file
with the form

   (let ((*endian* :big-endian))
     (read-binary 'elf-header stream)


;;; ELF basic type declarations
(define-unsigned word 4)
(define-signed sword  4)
(define-unsigned addr 4)
(define-unsigned off  4)
(define-unsigned half 2)

;;; ELF file header structure
(define-binary-class elf-header ()
  ((e-ident
    :binary-type (define-binary-struct e-ident ()
		   (ei-magic nil :binary-type
			     (define-binary-struct ei-magic ()
			       (ei-mag0 0 :binary-type u8)
			       (ei-mag1 #\null :binary-type char8)
			       (ei-mag2 #\null :binary-type char8)
			       (ei-mag3 #\null :binary-type char8)))
		   (ei-class nil :binary-type
			     (define-enum ei-class (u8)
			       elf-class-none 0
			       elf-class-32   1
			       elf-class-64   2))
		   (ei-data nil :binary-type
			    (define-enum ei-data (u8)
			      elf-data-none 0
			      elf-data-2lsb 1
			      elf-data-2msb 2))
		   (ei-version 0 :binary-type u8)
		   (padding nil :binary-type 1)
		   (ei-name "" :binary-type
			    (define-null-terminated-string ei-name 8))))
   (e-type
    :binary-type (define-enum e-type (half)
		   et-none 0
		   et-rel  1
		   et-exec 2
		   et-dyn  3
		   et-core 4
		   et-loproc #xff00
		   et-hiproc #xffff))
   (e-machine
    :binary-type (define-enum e-machine (half)
		   em-none  0
		   em-m32   1
		   em-sparc 2
		   em-386   3
		   em-68k   4
		   em-88k   5
		   em-860   7
		   em-mips  8))
   (e-version   :binary-type word)
   (e-entry     :binary-type addr)
   (e-phoff     :binary-type off)
   (e-shoff     :binary-type off)
   (e-flags     :binary-type word)
   (e-ehsize    :binary-type half)
   (e-phentsize :binary-type half)
   (e-phnum     :binary-type half)
   (e-shentsize :binary-type half)
   (e-shnum     :binary-type half)
   (e-shstrndx  :binary-type half)))


For a second example, here's an approach to supporting floats:

  (define-bitfield ieee754-single-float (u32)
    (((:enum :byte (1 31))
       positive 0
       negative 1)
      ((:numeric exponent 8 23))
      ((:numeric significand 23 0))))




The postscript file "type-hierarchy.ps" shows the binary types
hierarchy.  It is generated using psgraph from the CMU lisp
repository:

  (with-open-file (*standard-output* "type-hierarchy.ps"
                   :direction :output
                   :if-exists :supersede)
    (psgraph:psgraph 'binary-type
	             #'(lambda (p)
	                 (mapcar #'class-name
				 (aclmop:class-direct-subclasses
                                   (find-class p))))
                     #'(lambda (s) (list (symbol-name s)))
	             t))