term% cat index.txt VENTI-FMT(8) System Manager's Manual VENTI-FMT(8)
NAME
buildindex, checkarenas, checkindex, conf, fmtarenas, fmtbloom, fmtin‐
dex, fmtisect - prepare and maintain a venti server
SYNOPSIS
venti/fmtarenas [ -Z ] [ -a arenasize ] [ -b blocksize ] name file
venti/fmtisect [ -1Z ] [ -b blocksize ] name file
venti/fmtbloom [ -n nblocks | -N nhash ] [ -s size ] file
venti/fmtindex [ -a ] venti.conf
venti/conf [ -w ] partition [ configfile ]
venti/buildindex [ -bd ] [ -i isect ] ... [ -M imemsize ] venti.conf
venti/checkindex [ -f ] [ -B blockcachesize ] venti.conf tmp
venti/checkarenas [ -afv ] file
DESCRIPTION
These commands aid in the setup, maintenance, and debugging of venti
servers. See venti(6) for an overview of the venti system and venti(8)
for an overview of the data structures used by the venti server.
Note that the units for the various sizes in the following commands can
be specified by appending or to indicate kilobytes, megabytes, or giga‐
bytes respectively.
Formatting
To prepare a server for its initial use, the arena partitions and the
index sections must be formatted individually, with fmtarenas and fmti‐
sect. Then the collection of index sections must be combined into a
venti index with fmtindex.
Fmtarenas formats the given file, typically a disk partition, into an
arena partition. The arenas in the partition are given names of the
form name%d, where %d is replaced with a sequential number starting at
0.
Options to fmtarenas are:
-a arenasize
The arenas are of arenasize bytes. The default is 512M, which
was selected to provide a balance between the number of arenas
and the ability to copy an arena to external media such as
recordable CDs and tapes.
-b blocksize
The size, in bytes, for read and write operations to the file.
The size is recorded in the file, and is used by applications
that access the arenas. The default is 8k.
-4 Create a `version 4' arena partition for backwards compatibility
with old servers. The default is version 5, used by the current
venti server.
-Z Do not zero the data sections of the arenas. Using this option
reduces the formatting time but should only be used when it is
known that the file was already zeroed. (Version 4 only; ver‐
sion 5 sections are not and do not need to be zeroed.)
Fmtisect formats the given file, typically a disk partition, as a venti
index section with the specified name. Each of the index sections in a
venti configuration must have a unique name.
Options to fmtisect are:
-b bucketsize
The size of an index bucket, in bytes. All the index sections
within a index must have the same bucket size. The default is
8k.
-1 Create a `version 1' index section for backwards compatibility
with old servers. The default is version 2, used by the current
venti server.
-Z Do not zero the index. Using this option reduces the formatting
time but should only be used when it is known that the file was
already zeroed. (Version 1 only; version 2 sections are not and
do not need to be zeroed.)
Fmtbloom formats the given file as a Bloom filter (see venti(6)). The
options are:
-n nblock | -N nhash
The number of blocks expected to be indexed by the filter or the
number of hash functions to use. If the -n option is given, it
is used, along with the total size of the filter, to compute an
appropriate nhash.
-s size
The size of the Bloom filter. The default is the total size of
the file. In either case, size is rounded down to a power of
two.
The file argument in the commands above can be of the form file:lo-hi
to specify a range of the file. Lo and hi are specified in bytes but
can have the usual k, m, or g suffixes. Either lo or hi may be omit‐
ted. This notation eliminates the need to partition raw disks on non-
Plan 9 systems.
Fmtindex reads the configuration file venti.conf and initializes the
index sections to form a usable index structure. The arena files and
index sections must have previously been formatted using fmtarenas and
fmtisect respectively.
The function of a venti index is to map a SHA1 fingerprint to a loca‐
tion in the data section of one of the arenas. The index is composed
of blocks, each of which contains the mapping for a fixed range of pos‐
sible fingerprint values. Fmtindex determines the mapping between SHA1
values and the blocks of the collection of index sections. Once this
mapping has been determined, it cannot be changed without rebuilding
the index. The basic assumption in the current implementation is that
the index structure is sufficiently empty that individual blocks of the
index will rarely overflow. The total size of the index should be
about 2% to 10% of the total size of the arenas, but the exact percent‐
age depends both on the index block size and the compressed size of
blocks stored. See the discussion in venti(8) for more.
Fmtindex also computes a mapping between a linear address space and the
data section of the collection of arenas. The -a option can be used to
add additional arenas to an index. To use this feature, add the new
arenas to venti.conf after the existing arenas and then run fmtindex
-a.
A copy of the above mappings is stored in the header for each of the
index sections. These copies enable buildindex to restore a single in‐
dex section without rebuilding the entire index.
To make it easier to bootstrap servers, the configuration file can be
stored in otherwise empty space at the beginning of any venti parti‐
tions using conf. A partition so branded with a configuration file can
be used in place of a configuration file when invoking any of the venti
commands. By default, conf prints the configuration stored in parti‐
tion. When invoked with the -w flag, conf reads a configuration file
from configfile (or else standard input) and stores it in partition.
Checking and Rebuilding
Buildindex populates the index for the Venti system described in
venti.conf. The index must have previously been formatted using fmtin‐
dex. This command is typically used to build a new index for a Venti
system when the old index becomes too small, or to rebuild an index af‐
ter media failure. Small errors in an index can usually be fixed with
checkindex, but checkindex requires a large temporary workspace and
buildindex does not.
Options to buildindex are:
-b Reinitialise the Bloom filter, if any.
-d `Dumb' mode; run all three passes.
-i isect
Only rebuild index section isect; may be repeated to rebuild
multiple sections. The name is special and just reads the are‐
nas.
-M imemsize
The amount of memory, in bytes, to use for caching raw disk ac‐
cesses while running buildindex. (This is not a property of the
created index.) The usual suffices apply. The default is 256M.
Checkindex examines the Venti index described in venti.conf. The pro‐
gram detects various error conditions including: blocks that are not
indexed, index entries for blocks that do not exist, and duplicate in‐
dex entries. If requested, an attempt can be made to fix errors that
are found.
The tmp file, usually a disk partition, must be large enough to store a
copy of the index. This temporary space is used to perform a merge
sort of index entries generated by reading the arenas.
Options to checkindex are:
-B blockcachesize
The amount of memory, in bytes, to use for caching raw disk ac‐
cesses while running checkindex. The default is 8k.
-f Attempt to fix any errors that are found.
Checkarenas examines the Venti arenas contained in the given file. The
program detects various error conditions, and optionally attempts to
fix any errors that are found.
Options to checkarenas are:
-a For each arena, scan the entire data section. If this option is
omitted, only the end section of the arena is examined.
-f Attempt to fix any errors that are found.
-v Increase the verbosity of output.
SOURCE
/sys/src/cmd/venti/srv
SEE ALSO
venti(6), venti(8)
BUGS
Buildindex should allow an individual index section to be rebuilt.
VENTI-FMT(8)