term% cat index.txt VENTI(8) System Manager's Manual VENTI(8)
NAME
venti - archival storage server
SYNOPSIS
venti/venti [ -Ldrs ] [ -a address ] [ -B blockcachesize ] [ -c config
] [ -C lumpcachesize ] [ -h httpaddress ] [ -I indexcachesize ] [ -m
free-memory-percent ] [ -W webroot ]
DESCRIPTION
Venti is a SHA1-addressed archival storage server. See venti(6) for a
full introduction to the system. This page documents the structure and
operation of the server.
A venti server requires multiple disks or disk partitions, each of
which must be properly formatted before the server can be run.
Disk
The venti server maintains three disk structures, typically stored on
raw disk partitions: the append-only data log, which holds, in sequen‐
tial order, the contents of every block written to the server; the in‐
dex, which helps locate a block in the data log given its score; and
optionally the bloom filter, a concise summary of which scores are
present in the index. The data log is the primary storage. To improve
the robustness, it should be stored on a device that provides RAID
functionality. The index and the bloom filter are optimizations em‐
ployed to access the data log efficiently and can be rebuilt if lost or
damaged.
The data log is logically split into sections called arenas, typically
sized for easy offline backup (e.g., 500MB). A data log may comprise
many disks, each storing one or more arenas. Such disks are called
arena partitions. Arena partitions are filled in the order given in
the configuration.
The index is logically split into block-sized pieces called buckets,
each of which is responsible for a particular range of scores. An in‐
dex may be split across many disks, each storing many buckets. Such
disks are called index sections.
The index must be sized so that no bucket is full. When a bucket
fills, the server must be shut down and the index made larger. Since
scores appear random, each bucket will contain approximately the same
number of entries. Index entries are 40 bytes long. Assuming that a
typical block being written to the server is 8192 bytes and compresses
to 4096 bytes, the active index is expected to be about 1% of the ac‐
tive data log. Storing smaller blocks increases the relative index
footprint; storing larger blocks decreases it. To allow variation in
both block size and the random distribution of scores to buckets, the
suggested index size is 5% of the active data log.
The (optional) bloom filter is a large bitmap that is stored on disk
but also kept completely in memory while the venti server runs. It
helps the venti server efficiently detect scores that are not already
stored in the index. The bloom filter starts out zeroed. Each score
recorded in the bloom filter is hashed to choose nhash bits to set in
the bloom filter. A score is definitely not stored in the index of any
of its nhash bits are not set. The bloom filter thus has two parame‐
ters: nhash (maximum 32) and the total bitmap size (maximum 512MB, 232
bits).
The bloom filter should be sized so that nhash × nblock ≤ 0.7 × b,
where nblock is the expected number of blocks stored on the server and
b is the bitmap size in bits. The false positive rate of the bloom
filter when sized this way is approximately 2-nblock. Nhash less than
10 are not very useful; nhash greater than 24 are probably a waste of
memory. Fmtbloom (see venti-fmt(8)) can be given either nhash or
nblock; if given nblock, it will derive an appropriate nhash.
Memory
Venti can make effective use of large amounts of memory for various
caches.
The lump cache holds recently-accessed venti data blocks, which the
server refers to as lumps. The lump cache should be at least 1MB but
can profitably be much larger. The lump cache can be thought of as the
level-1 cache: read requests handled by the lump cache can be served
instantly.
The block cache holds recently-accessed disk blocks from the arena par‐
titions. The block cache needs to be able to simultaneously hold two
blocks from each arena plus four blocks for the currently-filling
arena. The block cache can be thought of as the level-2 cache: read
requests handled by the block cache are slower than those handled by
the lump cache, since the lump data must be extracted from the raw disk
blocks and possibly decompressed, but no disk accesses are necessary.
The index cache holds recently-accessed or prefetched index entries.
The index cache needs to be able to hold index entries for three or
four arenas, at least, in order for prefetching to work properly. Each
index entry is 50 bytes. Assuming 500MB arenas of 128,000 blocks that
are 4096 bytes each after compression, the minimum index cache size is
about 6MB. The index cache can be thought of as the level-3 cache:
read requests handled by the index cache must still go to disk to fetch
the arena blocks, but the costly random access to the index is avoided.
The size of the index cache determines how long venti can sustain its
`burst' write throughput, during which time the only disk accesses on
the critical path are sequential writes to the arena partitions. For
example, if you want to be able to sustain 10MB/s for an hour, you need
enough index cache to hold entries for 36GB of blocks. Assuming
8192-byte blocks, you need room for almost five million index entries.
Since index entries are 50 bytes each, you need 250MB of index cache.
If the background index update process can make a single pass through
the index in an hour, which is possible, then you can sustain the
10MB/s indefinitely (at least until the arenas are all filled).
The bloom filter requires memory equal to its size on disk, as dis‐
cussed above.
A reasonable starting allocation is to divide memory equally (in
thirds) between the bloom filter, the index cache, and the lump and
block caches; the third of memory allocated to the lump and block
caches should be split unevenly, with more (say, two thirds) going to
the block cache.
Network
The venti server announces two network services, one (conventionally
TCP port venti, 17034) serving the venti protocol as described in
venti(6), and one serving HTTP (conventionally TCP port http, 80).
The venti web server provides the following URLs for accessing status
information:
/index A summary of the usage of the arenas and index sections.
/xindex
An XML version of /index.
/storage
Brief storage totals.
/set/variable
The current integer value of variable. Variables are: compress,
whether or not to compress blocks (for debugging); logging,
whether to write entries to the debugging logs; stats, whether
to collect run-time statistics; icachesleeptime, the time in
milliseconds between successive updates of megabytes of the in‐
dex cache; arenasumsleeptime, the time in milliseconds between
reads while checksumming an arena in the background. The two
sleep times should be (but are not) managed by venti; they exist
to provide more experience with their effects. The other vari‐
ables exist only for debugging and performance measurement.
/set/variable/value
Set variable to value.
/graph/name/param/param/...
A PNG image graphing the named run-time statistic over time.
The details of names and parameters are undocumented; see
httpd.c in the venti sources.
/log A list of all debugging logs present in the server's memory.
/log/name
The contents of the debugging log with the given name.
/flushicache
Force venti to begin flushing the index cache to disk. The re‐
quest response will not be sent until the flush has completed.
/flushdcache
Force venti to begin flushing the arena block cache to disk.
The request response will not be sent until the flush has com‐
pleted.
Requests for other files are served by consulting a directory named in
the configuration file (see webroot below).
Configuration File
A venti configuration file enumerates the various index sections and
arenas that constitute a venti system. The components are indicated by
the name of the file, typically a disk partition, in which they reside.
The configuration file is the only location that file names are used.
Internally, venti uses the names assigned when the components were for‐
matted with fmtarenas or fmtisect (see venti-fmt(8)). In particular,
only the configuration needs to be changed if a component is moved to a
different file.
The configuration file consists of lines in the form described below.
Lines starting with # are comments.
index name
Names the index for the system.
arenas file
File is an arena partition, formatted using fmtarenas.
isect file
File is an index section, formatted using fmtisect.
bloom file
File is a bloom filter, formatted using fmtbloom.
After formatting a venti system using fmtindex, the order of arenas and
index sections should not be changed. Additional arenas can be ap‐
pended to the configuration; run fmtindex with the -a flag to update
the index.
The configuration file also holds configuration parameters for the
venti server itself. These are:
mem size
lump cache size
bcmem size
block cache size
icmem size
index cache size
addr netaddr
network address to announce venti service (default tcp!*!venti)
httpaddr netaddr
network address to announce HTTP service (default tcp!*!http)
queuewrites
queue writes in memory (default is not to queue)
webroot dir
directory tree containing files for venti's internal HTTP server
to consult for unrecognized URLs
The units for the various cache sizes above can be specified by append‐
ing a or (case-insensitive) to indicate kilobytes, megabytes, or giga‐
bytes respectively.
The file name in the configuration lines above can be of the form
file:lo-hi to specify a range of the file. Lo and hi are specified in
bytes but can have the usual k, m, or g suffixes. Either lo or hi may
be omitted. This notation eliminates the need to partition raw disks
on non-Plan 9 systems.
Command Line
Many of the options to Venti duplicate parameters that can be specified
in the configuration file. The command line options override those
found in a configuration file. Additional options are:
-c config
The server configuration file (default venti.conf)
-d Produce various debugging information on standard error. Im‐
plies -s.
-L Enable logging. By default all logging is disabled. Logging
slows server operation considerably.
-m Allocate free-memory-percent percent of the available free RAM,
and partition it per the guidelines in the Memory subsection.
This percentage should be large enough to include the entire
bloom filter. This overrides all other memory sizing parame‐
ters, including those on the command line and in the configura‐
tion file. 25% is a reasonable choice.
-r Allow only read access to the venti data.
-s Do not run in the background. Normally, the foreground process
will exit once the Venti server is initialized and ready for
connections.
EXAMPLE
A simple configuration:
% cat venti.conf
index main
isect /tmp/disks/isect0
isect /tmp/disks/isect1
arenas /tmp/disks/arenas
bloom /tmp/disks/bloom
%
Format the index sections, the arena partition, the bloom filter, and
finally the main index:
% venti/fmtisect isect0. /tmp/disks/isect0
% venti/fmtisect isect1. /tmp/disks/isect1
% venti/fmtarenas arenas0. /tmp/disks/arenas &
% venti/fmtbloom /tmp/disks/bloom &
% wait
% venti/fmtindex venti.conf
%
Start the server and check the storage statistics:
% venti/venti
% hget http://$sysname/storage
SOURCE
/sys/src/cmd/venti/srv
SEE ALSO
venti(1), venti(2), venti(6), venti-backup(8), venti-fmt(8)
Sean Quinlan and Sean Dorward, ``Venti: a new approach to archival
storage'', Usenix Conference on File and Storage Technologies , 2002.
BUGS
Setting up a venti server is too complicated.
VENTI(8)