glenda.party
term% ls -F
term% cat index.txt
VENTI-FMT(8)                System Manager's Manual               VENTI-FMT(8)



NAME
       buildindex,  checkarenas, checkindex, conf, fmtarenas, fmtbloom, fmtin‐
       dex, fmtisect - prepare and maintain a venti server

SYNOPSIS
       venti/fmtarenas [ -Z ] [ -a arenasize ] [ -b blocksize ] name file

       venti/fmtisect [ -1Z ] [ -b blocksize ] name file

       venti/fmtbloom [ -n nblocks | -N nhash ] [ -s size ] file

       venti/fmtindex [ -a ] venti.conf

       venti/conf [ -w ] partition [ configfile ]

       venti/buildindex [ -bd ] [ -i isect ] ... [ -M imemsize ] venti.conf

       venti/checkindex [ -f ] [ -B blockcachesize ] venti.conf tmp

       venti/checkarenas [ -afv ] file

DESCRIPTION
       These commands aid in the setup, maintenance, and  debugging  of  venti
       servers.  See venti(6) for an overview of the venti system and venti(8)
       for an overview of the data structures used by the venti server.

       Note that the units for the various sizes in the following commands can
       be specified by appending or to indicate kilobytes, megabytes, or giga‐
       bytes respectively.

   Formatting
       To prepare a server for its initial use, the arena partitions  and  the
       index sections must be formatted individually, with fmtarenas and fmti‐
       sect.  Then the collection of index sections must be  combined  into  a
       venti index with fmtindex.

       Fmtarenas  formats  the given file, typically a disk partition, into an
       arena partition.  The arenas in the partition are given  names  of  the
       form  name%d, where %d is replaced with a sequential number starting at
       0.

       Options to fmtarenas are:

       -a arenasize
              The arenas are of arenasize bytes.  The default is  512M,  which
              was  selected  to provide a balance between the number of arenas
              and the ability to copy an  arena  to  external  media  such  as
              recordable CDs and tapes.

       -b blocksize
              The  size,  in bytes, for read and write operations to the file.
              The size is recorded in the file, and is  used  by  applications
              that access the arenas.  The default is 8k.

       -4     Create a `version 4' arena partition for backwards compatibility
              with old servers.  The default is version 5, used by the current
              venti server.

       -Z     Do  not zero the data sections of the arenas.  Using this option
              reduces the formatting time but should only be used when  it  is
              known  that  the file was already zeroed.  (Version 4 only; ver‐
              sion 5 sections are not and do not need to be zeroed.)

       Fmtisect formats the given file, typically a disk partition, as a venti
       index section with the specified name.  Each of the index sections in a
       venti configuration must have a unique name.

       Options to fmtisect are:

       -b bucketsize
              The size of an index bucket, in bytes.  All the  index  sections
              within  a  index must have the same bucket size.  The default is
              8k.

       -1     Create a `version 1' index section for  backwards  compatibility
              with old servers.  The default is version 2, used by the current
              venti server.

       -Z     Do not zero the index.  Using this option reduces the formatting
              time  but should only be used when it is known that the file was
              already zeroed.  (Version 1 only; version 2 sections are not and
              do not need to be zeroed.)

       Fmtbloom  formats the given file as a Bloom filter (see venti(6)).  The
       options are:

       -n nblock | -N nhash
              The number of blocks expected to be indexed by the filter or the
              number  of hash functions to use.  If the -n option is given, it
              is used, along with the total size of the filter, to compute  an
              appropriate nhash.

       -s size
              The  size of the Bloom filter.  The default is the total size of
              the file.  In either case, size is rounded down to  a  power  of
              two.

       The  file  argument in the commands above can be of the form file:lo-hi
       to specify a range of the file.  Lo and hi are specified in  bytes  but
       can  have  the usual k, m, or g suffixes.  Either lo or hi may be omit‐
       ted.  This notation eliminates the need to partition raw disks on  non-
       Plan 9 systems.

       Fmtindex  reads  the  configuration file venti.conf and initializes the
       index sections to form a usable index structure.  The arena  files  and
       index  sections must have previously been formatted using fmtarenas and
       fmtisect respectively.

       The function of a venti index is to map a SHA1 fingerprint to  a  loca‐
       tion  in  the data section of one of the arenas.  The index is composed
       of blocks, each of which contains the mapping for a fixed range of pos‐
       sible fingerprint values.  Fmtindex determines the mapping between SHA1
       values and the blocks of the collection of index sections.   Once  this
       mapping  has  been  determined, it cannot be changed without rebuilding
       the index.  The basic assumption in the current implementation is  that
       the index structure is sufficiently empty that individual blocks of the
       index will rarely overflow.  The total size  of  the  index  should  be
       about 2% to 10% of the total size of the arenas, but the exact percent‐
       age depends both on the index block size and  the  compressed  size  of
       blocks stored.  See the discussion in venti(8) for more.

       Fmtindex also computes a mapping between a linear address space and the
       data section of the collection of arenas.  The -a option can be used to
       add  additional  arenas  to an index.  To use this feature, add the new
       arenas to venti.conf after the existing arenas and  then  run  fmtindex
       -a.

       A  copy  of  the above mappings is stored in the header for each of the
       index sections.  These copies enable buildindex to restore a single in‐
       dex section without rebuilding the entire index.

       To  make  it easier to bootstrap servers, the configuration file can be
       stored in otherwise empty space at the beginning of  any  venti  parti‐
       tions using conf.  A partition so branded with a configuration file can
       be used in place of a configuration file when invoking any of the venti
       commands.   By  default, conf prints the configuration stored in parti‐
       tion.  When invoked with the -w flag, conf reads a  configuration  file
       from configfile (or else standard input) and stores it in partition.

   Checking and Rebuilding
       Buildindex  populates  the  index  for  the  Venti  system described in
       venti.conf.  The index must have previously been formatted using fmtin‐
       dex.   This  command is typically used to build a new index for a Venti
       system when the old index becomes too small, or to rebuild an index af‐
       ter  media failure.  Small errors in an index can usually be fixed with
       checkindex, but checkindex requires a  large  temporary  workspace  and
       buildindex does not.

       Options to buildindex are:

       -b     Reinitialise the Bloom filter, if any.

       -d     `Dumb' mode; run all three passes.

       -i isect
              Only  rebuild  index  section  isect; may be repeated to rebuild
              multiple sections.  The name is special and just reads the  are‐
              nas.

       -M imemsize
              The  amount of memory, in bytes, to use for caching raw disk ac‐
              cesses while running buildindex.  (This is not a property of the
              created index.)  The usual suffices apply.  The default is 256M.

       Checkindex  examines the Venti index described in venti.conf.  The pro‐
       gram detects various error conditions including: blocks  that  are  not
       indexed,  index entries for blocks that do not exist, and duplicate in‐
       dex entries.  If requested, an attempt can be made to fix  errors  that
       are found.

       The tmp file, usually a disk partition, must be large enough to store a
       copy of the index.  This temporary space is used  to  perform  a  merge
       sort of index entries generated by reading the arenas.

       Options to checkindex are:

       -B blockcachesize
              The  amount of memory, in bytes, to use for caching raw disk ac‐
              cesses while running checkindex.  The default is 8k.

       -f     Attempt to fix any errors that are found.

       Checkarenas examines the Venti arenas contained in the given file.  The
       program  detects  various  error conditions, and optionally attempts to
       fix any errors that are found.

       Options to checkarenas are:

       -a     For each arena, scan the entire data section.  If this option is
              omitted, only the end section of the arena is examined.

       -f     Attempt to fix any errors that are found.

       -v     Increase the verbosity of output.

SOURCE
       /sys/src/cmd/venti/srv

SEE ALSO
       venti(6), venti(8)

BUGS
       Buildindex should allow an individual index section to be rebuilt.



                                                                  VENTI-FMT(8)