org.apache.lucene.codecs

## Class BlockTreeTermsWriter

- java.lang.Object
- - org.apache.lucene.codecs.FieldsConsumer
  - - org.apache.lucene.codecs.BlockTreeTermsWriter

- All Implemented Interfaces:
    Closeable, AutoCloseable

      public class BlockTreeTermsWriter
      extends FieldsConsumer

  Block-based terms index and dictionary writer.

  Writes terms dict and index, block-encoding (column stride) each term's metadata for each set of terms between two index terms.

  Files:

  - `.tim`: Term Dictionary
  - `.tip`: Term Index

  ### Term Dictionary

  The .tim file contains the list of terms in each field along with per-term statistics (such as docfreq) and per-term metadata (typically pointers to the postings list for that term in the inverted index).

  The .tim is arranged in blocks: with blocks containing a variable number of entries (by default 25-48), where each entry is either a term or a reference to a sub-block.

  NOTE: The term dictionary can plug into different postings implementations: the postings writer/reader are actually responsible for encoding and decoding the Postings Metadata and Term Metadata sections.

  - TermsDict (.tim) --> Header, Postings Metadata, BlockNumBlocks, FieldSummary, DirOffset
  - Block --> SuffixBlock, StatsBlock, MetadataBlock
  - SuffixBlock --> EntryCount, SuffixLength, ByteSuffixLength
  - StatsBlock --> StatsLength, <DocFreq, TotalTermFreq>EntryCount
  - MetadataBlock --> MetaLength, <Term Metadata>EntryCount
  - FieldSummary --> NumFields, <FieldNumber, NumTerms, RootCodeLength, ByteRootCodeLength, SumDocFreq, DocCount>NumFields
  - Header --> `CodecHeader`
  - DirOffset --> `Uint64`
  - EntryCount,SuffixLength,StatsLength,DocFreq,MetaLength,NumFields, FieldNumber,RootCodeLength,DocCount --> `VInt`
  - TotalTermFreq,NumTerms,SumTotalTermFreq,SumDocFreq --> `VLong`

  Notes:

  - Header is a `CodecHeader` storing the version information for the BlockTree implementation.
  - DirOffset is a pointer to the FieldSummary section.
  - DocFreq is the count of documents which contain the term.
  - TotalTermFreq is the total number of occurrences of the term. This is encoded as the difference between the total number of occurrences and the DocFreq.
  - FieldNumber is the fields number from `FieldInfos`. (.fnm)
  - NumTerms is the number of unique terms for the field.
  - RootCode points to the root block for the field.
  - SumDocFreq is the total number of postings, the number of term-document pairs across the entire field.
  - DocCount is the number of documents that have at least one posting for this field.
  - PostingsMetadata and TermMetadata are plugged into by the specific postings implementation: these contain arbitrary per-file data (such as parameters or versioning information) and per-term data (such as pointers to inverted files).

  ### Term Index

  The .tip file contains an index into the term dictionary, so that it can be accessed randomly. The index is also used to determine when a given term cannot exist on disk (in the .tim file), saving a disk seek.

  - TermsIndex (.tip) --> Header, FSTIndexNumFields <IndexStartFP>NumFields, DirOffset
  - Header --> `CodecHeader`
  - DirOffset --> `Uint64`
  - IndexStartFP --> `VLong`
  - FSTIndex --> `FST<byte[]>`

  Notes:

  - The .tip file contains a separate FST for each field. The FST maps a term prefix to the on-disk block that holds all terms starting with that prefix. Each field's IndexStartFP points to its FST.
  - DirOffset is a pointer to the start of the IndexStartFPs for all fields
  - It's possible that an on-disk block would contain too many terms (more than the allowed maximum (default: 48)). When this happens, the block is sub-divided into new blocks (called "floor blocks"), and then the output in the FST for the block's prefix encodes the leading byte of each sub-block, and its file pointer.
  See Also:
    `BlockTreeTermsReader`
  WARNING: This API is experimental and might change in incompatible ways in the next release.

- - ### Field Summary

    Fields
    Modifier and Type | Field and Description
    --- | ---
    `static int` | `DEFAULT_MAX_BLOCK_SIZE` Suggested default value for the `maxItemsInBlock` parameter to `BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
    `static int` | `DEFAULT_MIN_BLOCK_SIZE` Suggested default value for the `minItemsInBlock` parameter to `BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
    `static int` | `TERMS_INDEX_VERSION_APPEND_ONLY` Append-only
    `static int` | `TERMS_INDEX_VERSION_CURRENT` Current index format.
    `static int` | `TERMS_INDEX_VERSION_START` Initial index format.
    `static int` | `TERMS_VERSION_APPEND_ONLY` Append-only
    `static int` | `TERMS_VERSION_CURRENT` Current terms format.
    `static int` | `TERMS_VERSION_START` Initial terms format.

  - ### Constructor Summary

    Constructors
    Constructor and Description
    `BlockTreeTermsWriter(SegmentWriteState state, PostingsWriterBase postingsWriter, int minItemsInBlock, int maxItemsInBlock)` Create a new writer.

  - ### Method Summary

    Methods
    Modifier and Type | Method and Description
    --- | ---
    `TermsConsumer` | `addField(FieldInfo field)` Add a new field
    `void` | `close()` Called when we are done adding everything.
    `protected void` | `writeHeader(IndexOutput out)` Writes the terms file header.
    `protected void` | `writeIndexHeader(IndexOutput out)` Writes the index file header.
    `protected void` | `writeIndexTrailer(IndexOutput indexOut, long dirStart)` Writes the index file trailer.
    `protected void` | `writeTrailer(IndexOutput out, long dirStart)` Writes the terms file trailer.

    - ### Methods inherited from class org.apache.lucene.codecs.FieldsConsumer

      `merge`

    - ### Methods inherited from class java.lang.Object

      `clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

- - ### Field Detail

    - #### DEFAULT_MIN_BLOCK_SIZE

          public static final int DEFAULT_MIN_BLOCK_SIZE

      Suggested default value for the `minItemsInBlock` parameter to `BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
      See Also:
        Constant Field Values

    - #### DEFAULT_MAX_BLOCK_SIZE

          public static final int DEFAULT_MAX_BLOCK_SIZE

      Suggested default value for the `maxItemsInBlock` parameter to `BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
      See Also:
        Constant Field Values

    - #### TERMS_VERSION_START

          public static final int TERMS_VERSION_START

      Initial terms format.
      See Also:
        Constant Field Values

    - #### TERMS_VERSION_APPEND_ONLY

          public static final int TERMS_VERSION_APPEND_ONLY

      Append-only
      See Also:
        Constant Field Values

    - #### TERMS_VERSION_CURRENT

          public static final int TERMS_VERSION_CURRENT

      Current terms format.
      See Also:
        Constant Field Values

    - #### TERMS_INDEX_VERSION_START

          public static final int TERMS_INDEX_VERSION_START

      Initial index format.
      See Also:
        Constant Field Values

    - #### TERMS_INDEX_VERSION_APPEND_ONLY

          public static final int TERMS_INDEX_VERSION_APPEND_ONLY

      Append-only
      See Also:
        Constant Field Values

    - #### TERMS_INDEX_VERSION_CURRENT

          public static final int TERMS_INDEX_VERSION_CURRENT

      Current index format.
      See Also:
        Constant Field Values

  - ### Constructor Detail

    - #### BlockTreeTermsWriter

          public BlockTreeTermsWriter(SegmentWriteState state,
                              PostingsWriterBase postingsWriter,
                              int minItemsInBlock,
                              int maxItemsInBlock)
                               throws IOException

      Create a new writer. The number of items (terms or sub-blocks) per block will aim to be between minItemsPerBlock and maxItemsPerBlock, though in some cases the blocks may be smaller than the min.
      Throws:
        `IOException`

  - ### Method Detail

    - #### writeHeader

          protected void writeHeader(IndexOutput out)
                              throws IOException

      Writes the terms file header.
      Throws:
        `IOException`

    - #### writeIndexHeader

          protected void writeIndexHeader(IndexOutput out)
                                   throws IOException

      Writes the index file header.
      Throws:
        `IOException`

    - #### writeTrailer

          protected void writeTrailer(IndexOutput out,
                          long dirStart)
                               throws IOException

      Writes the terms file trailer.
      Throws:
        `IOException`

    - #### writeIndexTrailer

          protected void writeIndexTrailer(IndexOutput indexOut,
                               long dirStart)
                                    throws IOException

      Writes the index file trailer.
      Throws:
        `IOException`

    - #### addField

          public TermsConsumer addField(FieldInfo field)
                                 throws IOException

      Description copied from class: `FieldsConsumer`
      Add a new field
      Specified by:
        `addField` in class `FieldsConsumer`
      Throws:
        `IOException`

    - #### close

          public void close()
                     throws IOException

      Description copied from class: `FieldsConsumer`
      Called when we are done adding everything.
      Specified by:
        `close` in interface `Closeable`
      Specified by:
        `close` in interface `AutoCloseable`
      Specified by:
        `close` in class `FieldsConsumer`
      Throws:
        `IOException`

Copyright © 2000-2013 Apache Software Foundation. All Rights Reserved.