cl-protobufs.rst 37.8 KB
Newer Older
Scott McKay's avatar
Scott McKay committed
1
2
3
4
5
6
7
8
9
10
11
12
.. raw:: LaTeX

   \setlength{\parindent}{0pt}
   \setlength{\parskip}{6pt plus 2pt minus 1pt}


=========================
Protobufs for Common Lisp
=========================


:Description: Protobufs for Common Lisp
Scott McKay's avatar
Scott McKay committed
13
14
:Author: Scott McKay <swmckay@gmail.com>
:Date: $Date: 2012-08-31 11:13 -0500 (Fri, 31 Aug 2012) $
Scott McKay's avatar
Scott McKay committed
15
16
17
18
19
20
21
22
23
24

.. contents::
..
    1  Introduction
      1.1  Implementation notes
      1.2  Model classes
    2  Defining a Protobufs schema
      2.1  .proto file to Lisp conversion
      2.2  CLOS classes to .proto conversion
      2.3  Using .proto files directly
Scott McKay's avatar
Scott McKay committed
25
        2.3.1  A note on Lisp packages
Scott McKay's avatar
Scott McKay committed
26
      2.4  Using the Protobufs macros
27
        2.4.1  Protobufs types
28
        2.4.2  Protobufs service stubs
Scott McKay's avatar
Scott McKay committed
29
30
    3  Serializing and deserializing
      3.1  Wire format
31
32
      3.2  Text format
    4  Other API functions
33
34
35
      4.1 Extensions functions
      4.2 Initialization functions
      4.3 Python compatibility functions
Scott McKay's avatar
Scott McKay committed
36
37
    5  Lisp-only extensions
      5.1 Type aliases
Scott McKay's avatar
Scott McKay committed
38
39
40
41
42


Introduction
============

Scott McKay's avatar
Scott McKay committed
43
44
45
46
The Common Lisp Protobufs library provides a fully in-Lisp way for
Common Lisp programs to use existing, or define new Protobufs
"schemas", and serialize and deserialize objects to and from the
Protobufs wire and text formats.
Scott McKay's avatar
Scott McKay committed
47

Scott McKay's avatar
Scott McKay committed
48
49
To use it, first load the ASDF declaration file ``protobufs/cl-protobufs.asd``
and then use ASDF to load the library named ``:cl-protobufs``.
Scott McKay's avatar
Scott McKay committed
50
51
52
53
54
55


Implementation notes
--------------------

The Protobufs library defines a set of model classes that describes a
56
57
58
protobufs "schema" (i.e., one .proto file). These classes describe a
schema, its options, enums and enum values, messages and fields, and
services and methods.
Scott McKay's avatar
Scott McKay committed
59

Scott McKay's avatar
Scott McKay committed
60
61
62
63
64
65
Unlike the 'protobuf' library described at ``http://common-lisp.net/project/protobuf``,
this implementation is entirely written in Common Lisp. It provides
some things that the 'protobuf' library does not, in particular,
the ability to define Protobufs schemas entirely within Lisp and
the ability to "export" a set of CLOS classes as a Protobufs schema.

Scott McKay's avatar
Scott McKay committed
66
67
68
69
70
The library provides the means to convert several kinds of inputs into
the Protobufs models, including:

 - Parse an existing .proto file into a set of model objects.
 - Convert a set of related CLOS classes into a set of model objects.
71
 - Compile a ``proto:define-schema`` macro call into a set of model objects.
Scott McKay's avatar
Scott McKay committed
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93

It also provides two ways to convert the model objects into outputs:

 - Print a set of model objects using the standard Protobufs v2 syntax.
 - Print a set of model objects using the Common Lisp syntax, defined below.

There are two formats for serialization and deserialization:

 - The wire format, which is compact and fast.
 - The text format, which is human readable.

Finally, there is a validator that takes an old version of a Protobufs
schema and a new version of the same schema and generates a set of
warnings that describes any incompatibilities between the old schema and
the new one.


Model classes
-------------

::

94
  proto:protobuf-schema                                         [Class]
Scott McKay's avatar
Scott McKay committed
95
96

The class the represents a Protobufs schema, i.e., one .proto file.
97
It has slots for the name, options, enums, messages and services. The
98
name is equal to the name of the .proto file, without the file type.
Scott McKay's avatar
Scott McKay committed
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139

::

  proto:protobuf-option                                         [Class]

The class that represents a Protobufs option.
It has slots for a key and a value.

::

  proto:protobuf-enum                                           [Class]

The class that represents a Protobufs enum type.
It has slots for the enum name and its values.

::

  proto:protobuf-enum-value                                     [Class]

The class that represents one value in a Protobufs enum type.
It has slots for the value name and the value index.

::

  proto:protobuf-message                                        [Class]

The class that represents a Protobufs message.
It has slots for the name, options, nested enums and messages, and fields.

::

  proto:protobuf-field                                          [Class]

The class that represents one field in a Protobufs message.
It has slots for the name, type, index and options.

::

  proto:protobuf-service                                        [Class]

The class that represents a Protobufs service.
140
It has slots for the name, options and methods.
Scott McKay's avatar
Scott McKay committed
141
142
143

::

144
  proto:protobuf-method                                         [Class]
Scott McKay's avatar
Scott McKay committed
145

146
The class that represents one method description in a Protobufs service.
Scott McKay's avatar
Scott McKay committed
147
148
149
150
151
152
153
It has slots for the name, input type, output type and options.


Defining a Protobufs schema
===========================

There are several ways to define a Protobufs schema: convert a .proto
154
file to a .lisp source file and then use the Lisp file; convert a set
Scott McKay's avatar
Scott McKay committed
155
of Lisp classes to a Protobufs model, and then use either the .lisp or
156
157
the .proto representation of the model; use a .proto file directly in
an ASDF system; or use the Protobufs macros in a Lisp source file.
Scott McKay's avatar
Scott McKay committed
158
159
160
161
162
163
164


.proto file to Lisp conversion
------------------------------

If you have an existing .proto source file that you would like to
convert to Lisp classes (more precisely, to the macros defined by the
165
166
Protobufs library), you can use ``proto:parse-schema-from-file`` to
read the .proto file and then use ``proto:write-schema`` to write a
167
168
new .lisp file. (This is what that ASDF module type ``:protobuf-file``
does.)
Scott McKay's avatar
Scott McKay committed
169
170
171

::

172
173
  proto:parse-schema-from-file (pathname                        [Function]
                                &key name class conc-name)
Scott McKay's avatar
Scott McKay committed
174

175
Parses the contents of the file given by *pathname*, and returns the
176
Protobufs model (a set object objects rooted at ``proto:protobuf-schema``)
Scott McKay's avatar
Scott McKay committed
177
178
179
corresponding to the parsed file. The name of the Protobufs schema is
generated automatically from the file name.

180
181
182
183
*name*, *class* and *conc-name* are as for ``proto:parse-schema-from-stream``.
The defaults for *name* and *class* are produced by taking the name of the
file and generating a name string and a class name symbol.

Scott McKay's avatar
Scott McKay committed
184
185
::

186
187
  proto:parse-schema-from-stream (stream                        [Function]
                                  &key name class conc-name)
Scott McKay's avatar
Scott McKay committed
188
189

Parses the contents of the stream *stream*, and returns the Protobufs
190
191
192
193
194
195
196
197
198
schema corresponding to the parsed file.

If *name* is supplied, it gives the Protobufs name (a string) for the
schema. If *class* is supplied, it gives the Lisp name (a symbol). These
are only used for display purposes.

*conc-name* is the default "conc name" to use for all of the messages
in the file. The default is "", which has the effect of giving eponymous
slot accessors to all of the classes generating during the import process.
Scott McKay's avatar
Scott McKay committed
199
200
201

::

202
  proto:write-schema (schema &key stream type)                  [Function]
Scott McKay's avatar
Scott McKay committed
203

204
Pretty-prints the Protobufs schema *schema* onto the stream *stream*,
Scott McKay's avatar
Scott McKay committed
205
206
which defaults to ``*standard-output*``.

Scott McKay's avatar
Scott McKay committed
207
*type* can be either ``:proto`` or ``:lisp``.
Scott McKay's avatar
Scott McKay committed
208
209
210
211
212
213


CLOS classes to .proto conversion
---------------------------------

If you have an existing set of CLOS classes that you would like to
214
convert to a Protobufs schema, you can use ``proto:generate-schema-from-classes``.
Scott McKay's avatar
Scott McKay committed
215
216
217
218
219
220
221

Note that the Protobufs schema is an *approximation* of a good schema.
You should review it and, if necessary, change it (and probably the Lisp
classes as well) until you have a good Protobufs schema definition.

::

222
223
224
225
  proto:generate-schema-for-classes (classes                    [Function]
                                     &key name package lisp-package
                                          slot-filter type-filter enum-filter value-filter
                                          alias-existing-classes)
Scott McKay's avatar
Scott McKay committed
226
227
228

Given a list of class names *classes*, this generates a Protobufs schema
for the classes, generating any necessary enum types that correspond to
Scott McKay's avatar
Scott McKay committed
229
Lisp ``member`` types. The return value is the model, rooted at an instance
230
of ``proto:protobuf-schema``.
Scott McKay's avatar
Scott McKay committed
231

232
*name* and *package* can be supplied to give the Protobufs name and
233
234
235
236
package. *lisp-package* can be supplied to give the name of the Lisp
package, if it is different from *package*. (Note that you should
still use ``in-package`` at the top of .lisp files, and it should
match the value of *lisp-package*.)
Scott McKay's avatar
Scott McKay committed
237

238
239
240
241
242
243
*slot-filter*, *type-filter*, *enum-filter* and *value-filter* are
filtering functions that can be used to weed out things from the Lisp
classes that should not be included in the Protobufs schema.

*slot-filter* is a function of two arguments, a list of all the slots
in the class and the slot currently being processed, and should return
244
true if the slot is to be kept or ``nil`` if it to be discarded. For
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
example, if there are internal implementation slots in a class that
need not appear in the Protobufs description, it can be used to filter
them out.

*type-filter* is a function of one argument, the type (of a slot); it
should return a "transformed" type if any is required. For example,
complex ``and`` and ``or`` types can't be directly represented in
Protobufs; this can be used to substitute something workable.

*enum-filter* is a function of one argument, a list of all the values
of a ``member`` type; it should return the transformed values. For
example, there maybe be some enumeration values that don't make sense;
they can be discarded by the filter.

*value-filter* is a function of one argument, the value of a slot
initform. It should transform the value into a scalar value suitable
for Protobufs.

263
If *alias-existing-classes* is true (the default), the generated
Scott McKay's avatar
Scott McKay committed
264
Lisp code will include ``:alias-for`` so that there will be no clash
265
with the existing Lisp class.
Scott McKay's avatar
Scott McKay committed
266
267
268

::

269
270
271
272
  proto:write-schema-for-classes (classes                       [Function]
                                  &key stream type name package lisp-package
                                       slot-filter type-filter enum-filter value-filter
                                       alias-existing-classes)
Scott McKay's avatar
Scott McKay committed
273
274
275
276

Given a list of class names *classes*, this generates a Protobufs schema
for the classes, generating enum types as necessary, and then
pretty-prints the result onto *stream*. *type* can be either ``:proto``
277
278
(the default) or ``:lisp``; it controls which format the generated
code will be printed in. The return value is the model, rooted at an
279
instance of ``proto:protobuf-schema``.
Scott McKay's avatar
Scott McKay committed
280

281
*name* and *package* can be supplied to give the Protobufs name and
282
283
package. *lisp-package* can be supplied to give the name of the Lisp
package, if it is different from *package*.
Scott McKay's avatar
Scott McKay committed
284

285
*slot-filter*, *type-filter*, *enum-filter* and *value-filter* are
286
as for ``proto:generate-schema-for-classes``.
287

288
*alias-existing-classes* is as for ``proto:generate-schema-for-classes``.
289

Scott McKay's avatar
Scott McKay committed
290
291
292
293

Using .proto files directly
---------------------------

294
295
296
297
298
299
300
301
In addition to using the tools described above to convert between
.proto files and .lisp files, you can also use .proto files directly
in ASDF systems. Just use the ASDF module type ``:protobuf-file`` in
your system, and compile and load the system in the usual way. This
will create both the Protobufs model and the Lisp classes that
correspond to the Protobufs messages. (Note that it will also leave a
.lisp file having the same name as the .proto file in the file
system.)
Scott McKay's avatar
Scott McKay committed
302
303


Scott McKay's avatar
Scott McKay committed
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
A note on Lisp packages
~~~~~~~~~~~~~~~~~~~~~~~

When using an existing .proto file directly, it will likely contain a
``package`` line, but not a ``lisp_package`` line. CL-Protobufs needs
to choose some package to use. Here is what it does:

 - The package name from the ``package`` line is converted to a more
   Lisp-like name, e.g., ``fortune_teller`` becomes ``fortune-teller``.
 - If the Lisp package exists (i.e., you have previously used
   ``defpackage`` to define the packaged), then CL-Protobufs just
   uses it.
 - If the Lisp package does not exist, CL-Protobufs creates a new
   package of the given name that uses no other packages, not even
   the ``common-lisp`` package. In addition, the symbols naming all
   of the enum types, message types, field name and service method
   names are exported from the new package.


Scott McKay's avatar
Scott McKay committed
323
324
325
326
327
328
Using the Protobufs macros
--------------------------

You can define a Protobufs schema entirely within Lisp by using the
following macros. For example::

329
  (proto:define-schema color-wheel
330
331
      (:package com.google.colorwheel
       :lisp-package color-wheel)
Scott McKay's avatar
Scott McKay committed
332
333
334
335
336
    (proto:define-message color-wheel
        (:conc-name color-wheel-)
      (name   :type string)
      (colors :type (proto:list-of color) :default ()))
    (proto:define-message color
Scott McKay's avatar
Scott McKay committed
337
        (:conc-name color-)
Scott McKay's avatar
Scott McKay committed
338
      (name    :type (or string null))
Scott McKay's avatar
Scott McKay committed
339
340
      (r-value :type integer)
      (g-value :type integer)
Scott McKay's avatar
Scott McKay committed
341
342
      (b-value :type integer)
      (proto:define-extension 1000 max))
Scott McKay's avatar
Scott McKay committed
343
    (proto:define-extend color ()
Scott McKay's avatar
Scott McKay committed
344
      ((opacity 1000) :type (or null integer)))
Scott McKay's avatar
Scott McKay committed
345
    (proto:define-message get-color-request ()
Scott McKay's avatar
Scott McKay committed
346
347
      (wheel :type color-wheel)
      (name  :type string))
Scott McKay's avatar
Scott McKay committed
348
    (proto:define-message add-color-request ()
Scott McKay's avatar
Scott McKay committed
349
350
      (wheel :type color-wheel)
      (color :type color))
Scott McKay's avatar
Scott McKay committed
351
352
    (proto:define-service color-wheel ()
      (get-color (get-color-request color)
353
        :options (:deadline 1.0))
Scott McKay's avatar
Scott McKay committed
354
      (add-color (add-color-request color)
355
        :options (:deadline 1.0))))
Scott McKay's avatar
Scott McKay committed
356

Scott McKay's avatar
Scott McKay committed
357
358
This will create the Protobufs model objects, Lisp classes and enum
types that correspond to the model. The .proto file of the same schema
359
looks something like this::
Scott McKay's avatar
Scott McKay committed
360
361
362

  syntax = "proto2";

363
  package com.google.colorwheel;
Scott McKay's avatar
Scott McKay committed
364

365
366
  option (lisp_package) = "color-wheel";

Scott McKay's avatar
Scott McKay committed
367
368
369
370
371
372
373
374
375
376
  message ColorWheel {
    required string name = 1;
    repeated Color colors = 2;
  }

  message Color {
    optional string name = 1;
    required int64 rValue = 2;
    required int64 gValue = 3;
    required int64 bValue = 4;
Scott McKay's avatar
Scott McKay committed
377
378
379
    extensions 1000 to max;
  }

Scott McKay's avatar
Scott McKay committed
380
  extend Color {
Scott McKay's avatar
Scott McKay committed
381
    optional int64 opacity = 1000;
Scott McKay's avatar
Scott McKay committed
382
383
384
385
386
387
388
389
390
391
392
393
394
395
  }

  message GetColorRequest {
    required ColorWheel wheel = 1;
    required string name = 2;
  }

  message AddColorRequest {
    required ColorWheel wheel = 1;
    required Color color = 2;
  }

  service ColorWheel {
    rpc GetColor (GetColorRequest) returns (Color) {
396
      option deadline = 1.0;
Scott McKay's avatar
Scott McKay committed
397
398
    }
    rpc AddColor (AddColorRequest) returns (Color) {
399
      option deadline = 1.0;
Scott McKay's avatar
Scott McKay committed
400
401
    }
  }
Scott McKay's avatar
Scott McKay committed
402

403
Note that Lisp types ``(or null <T>)`` turn into optional fields,
404
and Lisp types ``(proto:list-of <T>)`` and ``(proto:vector-of <T>)``
405
406
turn into repeated fields representing by lists or vectors,
respectively.
407

Scott McKay's avatar
Scott McKay committed
408
409
410
411
412
413
Note also that the macros have assigned indexes to the fields for each
method; similarly, they will assign values to enumerations as well.
*This is not stable*, that is, if you add new fields or enum values,
the indexes could change, which would result in an incompatible
Protobufs schema. 

Scott McKay's avatar
Scott McKay committed
414
415
::

416
417
418
419
  proto:define-schema (type (&key name syntax import            [Macro]
                                  package lisp-package
                                  optimize options documentation)
                       &body messages)
Scott McKay's avatar
Scott McKay committed
420

421
422
423
424
Defines a Protobufs "schema" whose name is given by the symbol *type*,
corresponding to a .proto file of that name. By a "schema", we mean an
object that corresponds to the contents of one .proto file. If *name*
is not supplied, the Protobufs name of the schema is the camel-cased
Scott McKay's avatar
Scott McKay committed
425
426
427
rendition of *type* (e.g., the schema named ``color-wheel``, by
default, becomes ``ColorWheel``); otherwise the Protobufs name is the
string *name*.
Scott McKay's avatar
Scott McKay committed
428

Scott McKay's avatar
Scott McKay committed
429
*imports* is a list of pathname strings to be imported. This corresponds
430
to ``import`` in a .proto file. Note that ``proto:define-schema`` can
431
432
433
import both .proto files and .lisp files containing Protobufs macros,
but the generated .proto code will convert all of these to imports of
.proto files.
434
435

*syntax* and *package* are strings that give the Protobufs syntax and
Scott McKay's avatar
Scott McKay committed
436
437
438
package name. *lisp-package* can be supplied to give the name of the
Lisp package, if it is different from *package*. *package* corresponds
to ``package`` in a .proto file. If you want to specify a Lisp package
439
in a .proto file, you can use ``option (lisp_package)``.
Scott McKay's avatar
Scott McKay committed
440
441

*optimize* can be either ``:space`` (the default) or ``:speed``. When it
Scott McKay's avatar
Scott McKay committed
442
443
444
is ``:space`` the serialization methods generated for each message are
compact, but slower; when it is ``:speed``, the serialization methods
will be much faster, but will take more space. This corresponds to
445
``option optimize_for = CODE_SIZE|SPEED`` in a .proto file.
Scott McKay's avatar
Scott McKay committed
446
447
448

*options* is a property list whose keys and values are both strings,
for example, ``:option ("java_package" "com.yoyodyne.overthruster")``.
449
They are passed along unchanged to the generated .proto file.
Scott McKay's avatar
Scott McKay committed
450
451

*documentation* is a documentation string that is preserved as a comment
Scott McKay's avatar
Scott McKay committed
452
in the .proto file.
Scott McKay's avatar
Scott McKay committed
453
454

*body* consists of any number of calls to ``proto:define-enum``,
Scott McKay's avatar
Scott McKay committed
455
``proto:define-message``, ``proto:define-extend`` or ``proto:define-service``.
Scott McKay's avatar
Scott McKay committed
456
457
458
459
460
461
462
463
464
465
466

::

  proto:define-enum (type (&key name conc-name alias-for        [Macro]
                                options documentation)
                     &body values)

Defines a Protobufs enum type and a corresponding Lisp deftype whose name
is given by the symbol *type*. If *name* is not supplied, the Protobufs
name of the enum is the camel-cased rendition of *type*; otherwise the
Protobufs name is the string *name*. If *conc-name* is given, it will
Scott McKay's avatar
Scott McKay committed
467
be used as the prefix for all of the enum value names. In a .proto file,
468
you can use ``option (lisp_name)`` to override the default name for the
Scott McKay's avatar
Scott McKay committed
469
enum type in Lisp.
Scott McKay's avatar
Scott McKay committed
470
471

If *alias-for* is given, no Lisp deftype is defined. Instead, the enum
Scott McKay's avatar
Scott McKay committed
472
will be used as an alias for a ``member`` type that already exists in Lisp.
473
You can use ``option (lisp_alias)`` in a .proto file to give the Lisp
Scott McKay's avatar
Scott McKay committed
474
alias for an enum type.
Scott McKay's avatar
Scott McKay committed
475
476
477
478

*options*  is a property list whose keys and values are both strings.

*documentation* is a documentation string that is preserved as a comment
Scott McKay's avatar
Scott McKay committed
479
in the .proto file.
Scott McKay's avatar
Scott McKay committed
480

Scott McKay's avatar
Scott McKay committed
481
*body* consists of the enum values, each of which is either a symbol
482
483
484
485
486
or a list either of the form ``(name index)`` or ``(name &key index)``.
By default, and if you have not explicitly given an index, the indexes
start at 0 and are incremented by 1 for each new enum value. For
schema forward and backward compatibility, you should always use the
explicit form, either ``(name index)`` or ``(name &key index)``.
Scott McKay's avatar
Scott McKay committed
487

488
``proto:define-enum`` can be used only within ``proto:define-schema``
Scott McKay's avatar
Scott McKay committed
489
or ``proto:define-message``.
Scott McKay's avatar
Scott McKay committed
490
491
492
493
494
495
496

::

  proto:define-message (type (&key name conc-name alias-for     [Macro]
                                   options documentation)
                        &body fields)

Scott McKay's avatar
Scott McKay committed
497
498
499
500
501
502
503
504
Defines a Protobuf message and a corresponding Lisp defclass whose
name is given by the symbol *type*. If *name* is not supplied, the
Protobufs name of the class is the camel-cased rendition of *type*
(e.g., the class named ``color-wheel``, by default, becomes
``ColorWheel``); otherwise the Protobufs name is the string *name*. If
*conc-name* is given, it will be used as the prefix for all of the
slot accessor names. In a .proto file, you can use ``option (lisp_name)``
to override the default name for the class in Lisp.
Scott McKay's avatar
Scott McKay committed
505
506
507
508

If *alias-for* is given, no Lisp defclass is defined. Instead, the
message will be used as an alias for a class that already exists in
Lisp. This feature is intended to be used to define messages that will
509
510
511
512
be serialized from existing Lisp classes; unless you get the slot names,
readers and writers exactly right for each field, it will be the case
that trying to (de)serialize into a(n aliased) Lisp object won't work.
You can use ``option (lisp_alias)`` in a .proto file to give the Lisp
Scott McKay's avatar
Scott McKay committed
513
alias for the class corresponding to a message.
Scott McKay's avatar
Scott McKay committed
514
515
516
517

*options*  is a property list whose keys and values are both strings.

*documentation* is a documentation string that is preserved as a comment
Scott McKay's avatar
Scott McKay committed
518
in the .proto file.
Scott McKay's avatar
Scott McKay committed
519

Scott McKay's avatar
Scott McKay committed
520
The body *fields* consists of fields, ``proto:define-enum``,
Scott McKay's avatar
Scott McKay committed
521
522
``proto:define-message`` or ``proto:define-extension`` forms.

523
Fields take the form ``(slot &key index type name default reader writer)``.
524
525
*slot* can be either a symbol giving the slot name or a list of the
form ``(slot index)``. By default, the field indexes start at 1 and
526
are incremented by 1 for each new field value. *type* is the type of
527
the slot. For schema forward and backward compatibility, you should
528
always use either the ``(slot index)`` form or supply ``:index``.
Scott McKay's avatar
Scott McKay committed
529
530
531
532
533
534
535
536
537

*name* can be used to override the defaultly generated Protobufs field
name (for example, a Lisp field called ``color-name``, by default,
becomes ``color_name``). *default* is the default value for the
slot. *reader* is the name of a Lisp slot reader function to use to
get the value during serialization, as opposed to using
``slot-value``; this is meant to be used when aliasing an existing
class. *writer* can be similarly used to name a Lisp slot writer
function.
538

Scott McKay's avatar
Scott McKay committed
539
540
541
542
Note that the Protobufs does not support full Lisp type expressions in
the types of fields. The following type expressions are supported:

 - ``integer``, optionally with upper and lower bounds
543
544
545
 - ``signed-byte``, which correspond to ``proto:int32`` or ``proto:int64``
 - ``unsigned-byte``, which correspond to ``proto:uint32`` or ``proto:uint64``
 - ``float`` and ``double-float``
Scott McKay's avatar
Scott McKay committed
546
 - ``string``and ``character``
547
 - ``(array (unsigned-byte 8))``, which corresponds to ``proto:byte-vector``
Scott McKay's avatar
Scott McKay committed
548
549
550
551
 - ``boolean``
 - ``(member ...)``, where all the members are symbols or keywords or ``nil``
 - the name of a class that corresponds to another Protobufs message
 - ``(proto:list-of <T>)``, where ``<T>`` is any of the above types
552
 - ``(proto:vector-of <T>)``, where ``<T>`` is any of the above types
Scott McKay's avatar
Scott McKay committed
553
554
 - ``(or <T> null)``, where ``<T>`` is any of the above types

555
556
557
558
559
560
``member`` corresponds to a Protobufs ``enum``. ``(or <T> null)``
corresponds to an optional field. ``proto:list-of`` corresponds to a
repeated field, and the Lisp slot will be typed as a list. ``proto:vector-of``
corresponds to a repeated field, and the Lisp slot will be typed as an
adjustable array with a fill pointer. The other types correspond to
the various Protobufs scalar field types.
561

562
``proto:define-message`` can be used only within ``proto:define-schema``
Scott McKay's avatar
Scott McKay committed
563
or ``proto:define-message``.
564

565
566
567
568
569
570
571
572
573
574
575
576
577
578
579
580
581
582
583
584
585
586
587
588
589
590
591
::

  proto:define-extension (from to)                              [Macro]

Defines a field extension for the indexes from *from* to *to*.
*from* and *to* are positive integers ranging from 1 to 2^29 - 1.
*to* can also be the token ``max``, i.e., 2^29 - 1.

Once an extension to a message has been defined, you can use
``proto:define-extends`` to add new fields.

``proto:define-extension`` can be used only within ``proto:define-message``.

In non-Lisp implementations of Protobufs, you set and get the value
of an extension using functions like ``SetExtension()`` and
``GetExtension()``. For example, if you extended a ``Color`` message
to have an ``opacity`` field, you would set the field using something
like this::

  Color color;
  color.SetExtension(opacity, 0.5);

In Common Lisp Protobufs, you can just use an ordinary slot accessor::

  (let ((color (make-instance 'color)))
    (setf (color-opacity color) 0.5))

592
593
::

594
  proto:define-extend (type (&key name conc-name                [Macro]
Scott McKay's avatar
Scott McKay committed
595
596
                                  options documentation)
                       &body fields)
597

Scott McKay's avatar
Scott McKay committed
598
Defines a Protobuf ``extend``, that is, an extension to an existing
599
600
601
602
603
message (and corresponding Lisp class) that has additional fields that
were reserved by ``proto:define-extension``. *type* and *name* are as
for ``proto:define-message``. Note that no new Lisp class is defined;
the additional slots are implemented as getter and setter methods on
a closed-over variable. The other options, such as *conc-name* and
604
605
606
607
608
609
610
611
612
*alias-for* are take from the extended message.

*options*  is a property list whose keys and values are both strings.

*documentation* is a documentation string that is preserved as a comment
in the .proto file.

The body *fields* consists only of fields, which take the same form as
they do for ``proto:define-message``.
Scott McKay's avatar
Scott McKay committed
613

614
``proto:define-extend`` can be used only within ``proto:define-schema``
Scott McKay's avatar
Scott McKay committed
615
or ``proto:define-message``.
Scott McKay's avatar
Scott McKay committed
616
617
618
619
620

::

  proto:define-service (type (&key name                         [Macro]
                                   options documentation)
621
                        &body method-specs)
Scott McKay's avatar
Scott McKay committed
622

623
624
625
626
Defines a Protobufs service named *type* and corresponding Lisp generic
functions for all its methods. If *name* is not supplied, the Protobufs
name of the service is the camel-cased rendition of *type*; otherwise
the Protobufs name is the string *name*.
Scott McKay's avatar
Scott McKay committed
627
628
629
630

*options*  is a property list whose keys and values are both strings.

*documentation* is a documentation string that is preserved as a comment
Scott McKay's avatar
Scott McKay committed
631
in the .proto file.
Scott McKay's avatar
Scott McKay committed
632

633
The body is a set of method specs of the form
634
635
636
637
638
639
640
641
642
643
``(name (input-type [=>] output-type &key streams) &key options documentation)``.

For each method spec, *name* is a symbol naming the RPC method.
*input-type* and *output-type* give the input and output types of the method;
they may either be symbols or a list of the form ``(type &key name)``.
You can optionally include the symbol ``=>`` between the input and
output types; this seems to improve readability.

*streams* is also the name of a type, and provides a hook to RPC
implementations that implement "streaming".
Scott McKay's avatar
Scott McKay committed
644

645
``proto:define-service`` can only be used within ``proto:define-schema``.
646

Scott McKay's avatar
Scott McKay committed
647

648
Protobufs types
Scott McKay's avatar
Scott McKay committed
649
~~~~~~~~~~~~~~~
650
651
652
653
654
655
656
657
658
659
660
661
662
663
664

The following types are defined in the ``protobufs`` package:

 - ``proto:int32``, which corresponds to the Protobufs ``int32`` type
 - ``proto:int64``, which corresponds to the Protobufs ``int64`` type
 - ``proto:uint32``, which corresponds to the Protobufs ``uint32`` type
 - ``proto:uint64``, which corresponds to the Protobufs ``uint64`` type
 - ``proto:sint32``, which corresponds to the Protobufs ``sint32`` type
 - ``proto:sint64``, which corresponds to the Protobufs ``sint64`` type
 - ``proto:fixed32``, which corresponds to the Protobufs ``fixed32`` type
 - ``proto:fixed64``, which corresponds to the Protobufs ``fixed64`` type
 - ``proto:sfixed32``, which corresponds to the Protobufs ``sfixed32`` type
 - ``proto:sfixed64``, which corresponds to the Protobufs ``sfixed32`` type
 - ``proto:byte-vector``, which corresponds to the Protobufs ``bytes`` type
 - ``proto:list-of``, which corresponds to a repeated field
665
 - ``proto:vector-of``, which corresponds to a repeated field
666
667
668
669
670
671
672
673
674
675
676
677

The following existing Lisp type correspond to other Protobufs types:

 - ``string`` is the Protobufs UTF-8 encoded ``string`` type
 - ``boolean``  is the Protobufs ``bool`` type
 - ``float``  is the Protobufs ``float`` type
 - ``double-float``  is the Protobufs ``double`` type
 - ``member`` of a set of keywords generates a Protobufs ``enum`` type

Note that ``(or <T> null)`` corresponds to an optional field.


678
Protobufs service stubs
Scott McKay's avatar
Scott McKay committed
679
~~~~~~~~~~~~~~~~~~~~~~~
680
681
682

When you use the ``proto:define-service`` macro to define a service
with some methods, the macro defines "stubs" (CLOS generic functions)
683
684
for each of the methods in the service. Each method named ``foo`` gets
a client stub and a server stub whose signatures are, respectively::
685

Alejandro R Sedeño's avatar
Alejandro R Sedeño committed
686
687
688
689
690
691
692
  call-foo  (rpc-channel request &key callback) => response
  foo-impl  (rpc-channel request) => response

These methods are interned in a different lisp package, ``XXX-RPC``,
where ``XXX`` is the name of the lisp package into which the rest of
the schema's symbols are interned. This is done so that message field
accessors methods can't collide with the stubs.
693
694

The type of *rpc-channel* is unspecified, but is meant to be a
695
696
"channel" over which the RPC call will be done. The types of *request*
and *response* are message classes that were defined via
697
Protobufs. *callback* is a function of two arguments, the RPC channel
698
and the response; it is intended for use by asynchronous RPC calls.
699
700
701
702
703
704
705

For example, this fragment defines four stubs::

  (proto:define-service color-wheel ()
    (get-color (get-color-request color))
    (add-color (add-color-request color)))

Alejandro R Sedeño's avatar
Alejandro R Sedeño committed
706
707
708
709
The client stubs are ``call-get-color`` and ``call-add-color``, the
server stubs are ``get-color-impl`` and ``add-color-impl``. An RPC
library will implement a method for the client stub. You must fill in
the server stub yourself; it will implement the desired functionality.
710

711
712
713
The client stub also gets a single method defined for it that looks like
something like this::

Alejandro R Sedeño's avatar
Alejandro R Sedeño committed
714
  (defmethod call-foo (rpc-channel (request input-type) &key callback)
715
    (let ((call (and *rpc-package* *rpc-call-function*)))
716
      (funcall call rpc-channel method request :callback callback)))
717
718
719
720
721
722
723
724

where *rpc-channel*, *request* and *callback* are as above.
The special variables ``*rpc-package*`` and ``*rpc-call-function*``
are filled in when the RPC package is loaded. *method* is the
``proto:protobuf-method`` that describes the method; this is
included so that the RPC implementation can determine what type
of response object to create, what timeout to use, etc.

725
726
727
728
It is beyond the scope of this Protobufs library to provide the RPC
service; that is the domain of another library.


Scott McKay's avatar
Scott McKay committed
729
730
731
732
733
734
735
736
737
738
739
740
741
742
743
744
745
746
747
748
749
Serializing and deserializing
=============================

You can serialize from Lisp objects or deserialize into Lisp objects
using either the fast and compact Protobufs wire format, or the
human-readable text format.


Wire format
-----------

::

  proto:serialize-object-to-stream (object type                 [Function]
                                    &key stream visited)

Serializes the object *object* of type *type* onto the stream *stream*
using the wire format. *type* is the Lisp name of a Protobufs message
(often the name of a Lisp class) or a ``proto:protobuf-message`` object.
*type* defaults to the class of *object*

750
The element type of *stream* must be ``(unsigned-byte 8)``.
Scott McKay's avatar
Scott McKay committed
751

Scott McKay's avatar
Scott McKay committed
752
*visited* is an ``eql`` hash table used to cache object sizes. If it is
753
754
755
756
757
supplied, it will be cleared before it is used; otherwise, a fresh table
will be created.

The returned value is a byte vector containing the serialized object.
If the stream is ``nil``, the buffer is not actually written anywhere.
Scott McKay's avatar
Scott McKay committed
758
759
760
761
762
763
764
765
766
767
768

::

  proto:serialize-object (object type buffer                    [Generic function]
                          &optional start visited)

Serializes the object *object* of type *type* into the byte array
*buffer* using the wire format. *type* is the Lisp name of a Protobufs
message (often the name of a Lisp class) or a ``proto:protobuf-message``
object. *type* defaults to the class of *object*. The buffer is assumed
to be large enough to hold the serialized object; if it is not, an
Scott McKay's avatar
Scott McKay committed
769
out-of-bounds condition may be signaled.
Scott McKay's avatar
Scott McKay committed
770

771
772
773
The object is serialized using the wire format into the byte array
(i.e., a vector whose type is ``(unsigned-byte 8)``) given by *buffer*,
starting at the fixnum index *start* .
Scott McKay's avatar
Scott McKay committed
774

Scott McKay's avatar
Scott McKay committed
775
*visited* is an ``eql`` hash table used to cache object sizes.
Scott McKay's avatar
Scott McKay committed
776
777

The returned values are the modified buffer containing the serialized
778
779
780
object and the index that points one past the last serialized byte in
the buffer, which will be the number of bytes required to serialize the
object if *start* was 0.
Scott McKay's avatar
Scott McKay committed
781

Scott McKay's avatar
Scott McKay committed
782
783
Note that ``proto:serialize-object`` will not correctly serialize a
set of objects that has cycles. You must resolve these yourself.
Scott McKay's avatar
Scott McKay committed
784
785
786
787
788
789
790
791
792

::

  proto:deserialize-object-from-stream (type &key stream)       [Function]

Deserializes an object of the given type *type* as a Protobuf object.
*type* is the Lisp name of a Protobufs message (usually the name of a
Lisp class) or a ``proto:protobuf-message``.

793
794
The element type of *stream* must be ``(unsigned-byte 8)``.

Scott McKay's avatar
Scott McKay committed
795
796
797
798
799
800
801
802
803
804
805
806
The returned value is the deserialized object.

::

  proto:deserialize-object (type buffer &optional start end)    [Generic function]

Deserializes an object of the given type *type* as a Protobufs object.
*type* is the Lisp name of a Protobufs message (usually the name of a
Lisp class) or a ``proto:protobuf-message``.

The encoded bytes come from the byte array given by *buffer*, starting
at the fixnum index *start* up to the end of the buffer, given by *end*.
807
808
*start* defaults to 0, *end*' defaults to the length of the buffer.

Scott McKay's avatar
Scott McKay committed
809
If a zero byte is encountered in in the "tag position" during
810
811
deserialization, this is interpreted as an "end of object" marker
and deserialization stops.
Scott McKay's avatar
Scott McKay committed
812
813
814
815
816
817
818
819
820
821
822
823
824

The returned values are the deserialized object and the index into the
buffer at which the deserialization ended.

::

  proto:object-size (object type &optional visited)             [Generic function]

Computes the size in bytes of the object *object* of type *type*.
*type* is the Lisp name of a Protobufs message (usually the name of a
Lisp class) or a ``proto:protobuf-message``. *type* defaults to the
class of *object*

Scott McKay's avatar
Scott McKay committed
825
*visited* is an ``eql`` hash table used to cache object sizes.
Scott McKay's avatar
Scott McKay committed
826
827
828
829
830
831
832
833
834
835
836
837
838
839
840
841
842
843
844
845
846
847
848
849
850
851
852
853
854

The returned value is the size of the serialized object in bytes.


Text format
-----------

::

  proto:print-text-format (object &optional type                [Function]
                           &key stream suppress-line-breaks)

Prints the object *object* of type *type* onto the stream *stream* using
the textual format. *type* defaults to the class of *object*.

If *suppress-line-breaks* is true, all the output is put on a single line.

::

  proto:parse-text-format (type &key stream)                    [Function]

Parses the textual format of an object of the given type *type*. *type*
is the Lisp name of a Protobufs message (usually the name of a Lisp
class) or a ``proto:protobuf-message``. The input is read from the
stream *stream*.

The returned value is the object.


855
856
Other API functions
===================
Scott McKay's avatar
Scott McKay committed
857

858
859
860
861
862
863
864
865
866
867
868
869
870
871
872
873
874
875
876
877
878
879
880
881
882
883
884
885
886
887
888
889
890
891
892
893
894
895
896
897
898
Extensions functions
--------------------

::

proto:get-extension (object slot)                               [Generic function]

Returns the value of the extended slot *slot* in the object *object*.

Since you can just use the ordinary slot reader function, you should
not need to call ``proto:get-extension``. It is included for compatibility
with other Protobufs APIs.

::

proto:set-extension (object slot value)                         [Generic function]

Sets the value of the extended slot *slot* in the object *object*
to *value*.

Since you can just use the ordinary slot writer function, you should
not need to call ``proto:set-extension``. It is included for compatibility
with other Protobufs APIs.

::

proto:has-extension (object slot)                               [Generic function]

Returns true iff the object *object* has any value for the extended
slot *slot*.

::

proto:clear-extension (object slot)                             [Generic function]

Removes the value for the extended slot *slot* in the object *object*.


Initialization functions
------------------------

899
900
901
902
903
904
::

  proto:object-initialized-p (object type)                      [Generic function]

Returns true iff all of the fields of *object* of type *type* are
initialized, i.e., there are no fields whose value is unbound.
Scott McKay's avatar
Scott McKay committed
905
906
907

::

908
  proto:slot-initialized-p (object type slot)                   [Generic function]
Scott McKay's avatar
Scott McKay committed
909

910
911
912
Returns true iff the field *slot* of *object* of type *type* is
initialized, i.e., there are no fields whose value is unbound.

913
914
::

915
916
917
918
919
  proto:reinitialize-object (object type)                       [Generic function]

Initializes all of the fields of *object* of type *type* to their
default values.

920
921
922
923
924
925

Python compatibility functions
------------------------------

By popular demand, the Protobufs library provides an API that is very
similar to the API of the Python Protobufs library.
Scott McKay's avatar
Scott McKay committed
926
927
928
929
930
931
932
933

::

  proto:is-initialized (object)                                 [Generic function]

Returns true iff all of the fields of *object* are initialized, i.e.,
there are no fields whose value is unbound.

934
935
936
937
938
939
940
941
942
943
944
945
::

  proto:has-field (object slot)                                 [Generic function]

Returns true iff the field *slot* is initialized in *object*.

::

  proto:clear (object)                                          [Generic function]

Initializes all of the fields of *object* to their default values.

Scott McKay's avatar
Scott McKay committed
946
947
948
949
950
951
952
953
954
955
956
957
958
959
960
961
962
963
964
965
966
967
968
::

  proto:serialize (object &optional buffer start end)           [Generic function]

Serializes *object* into *buffer* using the wire format, starting at the
index *start* and going no further than *end*. *object* is an object
whose Lisp class corresponds to a Protobufs message.

::

  proto:merge-from-array (object buffer &optional start end)    [Generic function]

Deserializes the object encoded in *buffer* into *object*, starting at
the index *start* and ending at *end*. *object* is an object whose Lisp
class corresponds to a Protobufs message.

::

  proto:octet-size (object)                                     [Generic function]

Returns the number of bytes required to serialize *object* using the
wire format. *object* is an object whose Lisp class corresponds to a
Protobufs message.
Scott McKay's avatar
Scott McKay committed
969
970
971
972
973
974
975
976
977


Lisp-only extensions
====================

CL-Protobufs includes some Lisp-only extensions that have no
counterpart in Protobufs, but which "ground out" to compatible
Protobufs code.

Scott McKay's avatar
Scott McKay committed
978

Scott McKay's avatar
Scott McKay committed
979
980
981
982
983
984
985
986
987
988
989
990
991
992
993
994
995
996
997
998
999
1000
1001
1002
1003
1004
1005
1006
1007
1008
1009
1010
1011
1012
1013
1014
1015
1016
1017
1018
1019
1020
1021
1022
1023
1024
1025
1026
1027
1028
1029
1030
1031
1032
1033
1034
1035
1036
Type aliases
------------

::

  proto:define-type-alias (type (&key name alias-for            [Macro]
                                      documentation)
                           &key lisp-type proto-type
                                serializer deserializer)

Defines a Lisp type alias named *type* whose Lisp type is *lisp-type*
and whose Protobufs type is *proto-type*.  *lisp-type* must be a valid
Lisp type expression; *proto-type* myst be a Protobufs primitive type
(e.g., ``int32``, ``string``).

*serializer* is a function of one argument that takes an object of
type *lisp-type* and returns an object having the Protobufs primitive
type *proto-type*. *deserializer* is a function of one argument that
takes an object of type *proto-type* and returns an object having the
type *lisp-type*.

If *name* is not supplied, the Protobufs name of the type alias is the
camel-cased rendition of *type*; otherwise the Protobufs name is the
string *name*.

If *alias-for* is given, no Lisp deftype for ``type`` is
defined. Instead, the type alias is assumed to refer to a
previously-defined Lisp type.

For example, this Lisp schema::

  (proto:define-schema revision-history
      (:package revision-history)
    (proto:define-type-alias date ()
      :lisp-type integer
      :proto-type string
      :serializer integer-to-date
      :deserializer date-to-integer)
    (proto:define-message revision ()
      (proto:define-message metadata ()
        (author :type (or null string))
        (revision :type (or null string))
        (date :type (or null date)))
      (name :type string)
      (description :type string)))

will generate this Protobufs schema::

  message Revision {
    message Metadata {
      optional string author = 1;
      optional string revision = 2;
      // alias maps Lisp integer to Protobufs string
      optional string date = 3;
    }
    required string name = 1;
    required string description = 2;
  }