README.md 6.44 KB
Newer Older
Eric Timmons's avatar
Eric Timmons committed
1
2
3
4
This project provides a high level interface for interacting with tar
archives. It consists of several systems that provide different levels of
functionality.

Eric Timmons's avatar
Eric Timmons committed
5
6
**NOTE**: In order to load `tar/create` on Windows you need a version of osicat
with commits from [this PR](https://github.com/osicat/osicat/pull/57).
7

Eric Timmons's avatar
Eric Timmons committed
8
9
10
11
12
13
## Quickstart

If you want to extract a tar archive, without caring about preserving all the
metadata, run the following:

```common-lisp
Eric Timmons's avatar
Eric Timmons committed
14
(asdf:load-system "tar/simple-extract")
Eric Timmons's avatar
Eric Timmons committed
15
16
17
18
19
20
21
22
23

(tar:with-open-archive (a "/path/to/file.tar")
  (tar-simple-extract:simple-extract-archive a :directory "/path/to/extraction/point/"))
```

If you want to extract a tar archive, attempting to preserve symbolic links and
as much metadata as possible, evaluate the following:

```common-lisp
Eric Timmons's avatar
Eric Timmons committed
24
(asdf:load-system "tar/extract")
Eric Timmons's avatar
Eric Timmons committed
25
26
27
28
29

(tar:with-open-archive (a "/path/to/file.tar")
  (tar-extract:extract-archive a :directory "/path/to/extraction/point/"))
```

Eric Timmons's avatar
Eric Timmons committed
30
31
32
33
34
35
36
37
38
39
40
41
42
If you want to create a tar archive, attempting to preserve symbolic links and
as much metadata as possible, evaluate the following:

```common-lisp
(asdf:load-system "tar/create")

(let ((*default-pathname-defaults* #p"/path/to/parent/"))
  (tar:with-open-archive (a "/path/to/file.tar" :direction :output)
    (tar-create:create-archive a '("directory/") :recursep t)))
```

## Systems
### tar
Eric Timmons's avatar
Eric Timmons committed
43
44
45

The `tar` system is a thin layer on top of
the [`tar-file`](https://gitlab.common-lisp.net/cl-tar/cl-tar-file)
Eric Timmons's avatar
Typo    
Eric Timmons committed
46
system. While `tar-file` is focused on reading and writing physical entries,
Eric Timmons's avatar
Eric Timmons committed
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
`tar` places an emphasis on reading and writing logical entries.

The practical effect of this is that when using `tar`, any single object stored
in a tar archive (regular file, symlink, etc.) is always represented as a
single entry (`tar:file-entry`, `tar:symbolic-link-entry`, etc.). This is in
contrast to `tar-file` where a regular file with a long name could be either
unrepresentable, a single `tar-file:file-entry` with a ustar header, a
`tar-file:pax-extended-attributes-entry` followed by a `tar-file:file-entry`,
or a `tar-file:gnu-long-name-entry` followed by a `tar-file:file-entry`. `tar`
takes care of generating and interpreting the correct sequence of physical
entries, depending on the type of the archive being used.

While this system is useful for reading and writing tar archives, its primary
purpose is for the inspection of archives and creating archives whose content
does not come directly from the file system. For file system integration, see
the remaining systems.

Eric Timmons's avatar
Eric Timmons committed
64
### tar/simple-extract
Eric Timmons's avatar
Eric Timmons committed
65

Eric Timmons's avatar
Eric Timmons committed
66
The `tar/simple-extract` system provides functionality to extract a tar archive
Eric Timmons's avatar
Eric Timmons committed
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
to your filesystem. Unfortunately, faithfully extracting tar files onto the
filesystem is a complex task and is impossible to perform using only the
functionality mandated by the Common Lisp specification.

Therefore, this system does not try to faithfully reproduce the contents of the
tar file. This means that it should work on any CL implementation that `tar`
and `tar-file` does, but the cost is there is information loss.

This system does not support extracting the following entry types:

+ Symbolic links. They can be skipped or dereferenced.
+ Hard links. They can be skipped or dereferenced.
+ Character devices. They can be skipped.
+ Block devices. They can be skipped.
+ FIFO. They can be skipped.

Additionally, metadata such as file owner, permissions, and modification time
are not set.

It is recommended that this extraction method is used only to extract an
archive to an empty folder. If that is done with default settings, the
extraction process should be fairly safe and predictable. Otherwise, you run
the risk of existing symlinks being followed and overwriting arbitrary files on
your machine.

Eric Timmons's avatar
Eric Timmons committed
92
### tar/extract
Eric Timmons's avatar
Eric Timmons committed
93

Eric Timmons's avatar
Eric Timmons committed
94
The `tar/extract` system attempts to provide full extraction functionality. As
Eric Timmons's avatar
Eric Timmons committed
95
96
97
such, it is a much more complex beast and likely does not work on all
implementation/OS combinations. Patches are always welcome to make it more
portable.
Eric Timmons's avatar
Eric Timmons committed
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133

This system does not support extracting the following entry types on Windows:

+ Symbolic links. They can be skipped or dereferenced.
+ Hard links. They can be skipped or dereferenced.
+ Character devices. They can be skipped.
+ Block devices. They can be skipped.
+ FIFO. They can be skipped.

While it is *possible* to create symbolic links on Windows, it requires special
user permissions and many Windows applications are not designed with symlinks
in mind. This makes it both very unlikely that an arbitrary user can create
symlinks and very likely that the creation of symlinks would pose a risk to
other applications running on the same machine. Hard link support is also
possible, but I've rarely seen it and need to do more research before
attempting to support it.

While there is much less information loss when extracting an archive using this
system, that comes at the cost of increased security concerns when given an
untrusted archive as input. For example, a typical attack vector is to write a
symlink that points to an arbitrary file on your system and then using that
symlink to modify the target.

The goal of this system is to provide default extraction options so that it is
safe to extract an untrusted archive into an empty directory and have no way
for the extraction process to modify any file that exists outside of that
directory. Bug reports and patches are always welcome if you find this goal is
not met.

The safety goal is paramount, and as such, the performance of this system is
likely not the best. A primary reason is that we want to be robust to the
different file systems that are out there (e.g., case insensitive or unicode
normalizing), so we (currently) do not rely on caches to determine if a path is
safe to write to. The result is that there are many syscalls that happen during
extraction. There are probably ways this can be improved and patches are always
welcome.
Eric Timmons's avatar
Eric Timmons committed
134
135
136
137
138
139
140
141
142
143
144
145
146
147

### tar/create

The `tar/create` system produces a tar file from your file system that
faithfully reproduces all the metadata. It is a fairly complex beast and might
not work on all implementation/OS combinations. Patches are always welcome to
make it more portable.

See the docstring for `tar-create:create-archive` to understand its options.

## FAQ

### Can I create/extract compressed tar files?

148
149
150
151
Yes! It supports transparent gzip comporession. On input streams, the first few
bytes are examined for magic bytes. On output streams, the pathname is used as
a hint. You can override this behavior using the `:compression` argument to
`with-open-archive` or `open-archive`.