Overview | Package | Class | Use | Tree | Deprecated | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

## org.apache.lucene.codecs
Class BlockTreeTermsWriter

    java.lang.Object
      org.apache.lucene.codecs.FieldsConsumer
          org.apache.lucene.codecs.BlockTreeTermsWriter

All Implemented Interfaces:
  Closeable

    public class BlockTreeTermsWriter

extends FieldsConsumer

Block-based terms index and dictionary writer.

Writes terms dict and index, block-encoding (column stride) each term's metadata for each set of terms between two index terms.

Files:

- `.tim`: Term Dictionary
- `.tip`: Term Index

### Term Dictionary

The .tim file contains the list of terms in each field along with per-term statistics (such as docfreq) and per-term metadata (typically pointers to the postings list for that term in the inverted index).

The .tim is arranged in blocks: with blocks containing a variable number of entries (by default 25-48), where each entry is either a term or a reference to a sub-block.

NOTE: The term dictionary can plug into different postings implementations: the postings writer/reader are actually responsible for encoding and decoding the Postings Metadata and Term Metadata sections.

- TermsDict (.tim) --> Header, Postings Metadata, BlockNumBlocks, FieldSummary, DirOffset
- Block --> SuffixBlock, StatsBlock, MetadataBlock
- SuffixBlock --> EntryCount, SuffixLength, ByteSuffixLength
- StatsBlock --> StatsLength, <DocFreq, TotalTermFreq>EntryCount
- MetadataBlock --> MetaLength, <Term Metadata>EntryCount
- FieldSummary --> NumFields, <FieldNumber, NumTerms, RootCodeLength, ByteRootCodeLength, SumDocFreq, DocCount>NumFields
- Header --> `CodecHeader`
- DirOffset --> `Uint64`
- EntryCount,SuffixLength,StatsLength,DocFreq,MetaLength,NumFields, FieldNumber,RootCodeLength,DocCount --> `VInt`
- TotalTermFreq,NumTerms,SumTotalTermFreq,SumDocFreq --> `VLong`

Notes:

- Header is a `CodecHeader` storing the version information for the BlockTree implementation.
- DirOffset is a pointer to the FieldSummary section.
- DocFreq is the count of documents which contain the term.
- TotalTermFreq is the total number of occurrences of the term. This is encoded as the difference between the total number of occurrences and the DocFreq.
- FieldNumber is the fields number from `FieldInfos`. (.fnm)
- NumTerms is the number of unique terms for the field.
- RootCode points to the root block for the field.
- SumDocFreq is the total number of postings, the number of term-document pairs across the entire field.
- DocCount is the number of documents that have at least one posting for this field.
- PostingsMetadata and TermMetadata are plugged into by the specific postings implementation: these contain arbitrary per-file data (such as parameters or versioning information) and per-term data (such as pointers to inverted files).

### Term Index

The .tip file contains an index into the term dictionary, so that it can be accessed randomly. The index is also used to determine when a given term cannot exist on disk (in the .tim file), saving a disk seek.

- TermsIndex (.tip) --> Header, FSTIndexNumFields <IndexStartFP>NumFields, DirOffset
- Header --> `CodecHeader`
- DirOffset --> `Uint64`
- IndexStartFP --> `VLong`
- FSTIndex --> `FST<byte[]>`

Notes:

- The .tip file contains a separate FST for each field. The FST maps a term prefix to the on-disk block that holds all terms starting with that prefix. Each field's IndexStartFP points to its FST.
- DirOffset is a pointer to the start of the IndexStartFPs for all fields
- It's possible that an on-disk block would contain too many terms (more than the allowed maximum (default: 48)). When this happens, the block is sub-divided into new blocks (called "floor blocks"), and then the output in the FST for the block's prefix encodes the leading byte of each sub-block, and its file pointer.

See Also:
  `BlockTreeTermsReader`
WARNING: This API is experimental and might change in incompatible ways in the next release.

Field Summary
`static int` | `DEFAULT_MAX_BLOCK_SIZE` Suggested default value for the `maxItemsInBlock` parameter to `BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
`static int` | `DEFAULT_MIN_BLOCK_SIZE` Suggested default value for the `minItemsInBlock` parameter to `BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.
`static int` | `TERMS_INDEX_VERSION_APPEND_ONLY` Append-only
`static int` | `TERMS_INDEX_VERSION_CURRENT` Current index format.
`static int` | `TERMS_INDEX_VERSION_START` Initial index format.
`static int` | `TERMS_VERSION_APPEND_ONLY` Append-only
`static int` | `TERMS_VERSION_CURRENT` Current terms format.
`static int` | `TERMS_VERSION_START` Initial terms format.

Constructor Summary
`BlockTreeTermsWriter(SegmentWriteState state, PostingsWriterBase postingsWriter, int minItemsInBlock, int maxItemsInBlock)` Create a new writer.

Method Summary
`TermsConsumer` | `addField(FieldInfo field)` Add a new field
`void` | `close()` Called when we are done adding everything.
`protected void` | `writeHeader(IndexOutput out)` Writes the terms file header.
`protected void` | `writeIndexHeader(IndexOutput out)` Writes the index file header.
`protected void` | `writeIndexTrailer(IndexOutput indexOut, long dirStart)` Writes the index file trailer.
`protected void` | `writeTrailer(IndexOutput out, long dirStart)` Writes the terms file trailer.

Methods inherited from class org.apache.lucene.codecs.FieldsConsumer
`merge`

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Field Detail

### DEFAULT_MIN_BLOCK_SIZE

    public static final int DEFAULT_MIN_BLOCK_SIZE

  Suggested default value for the `minItemsInBlock` parameter to `BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.

  See Also:
    Constant Field Values

### DEFAULT_MAX_BLOCK_SIZE

    public static final int DEFAULT_MAX_BLOCK_SIZE

  Suggested default value for the `maxItemsInBlock` parameter to `BlockTreeTermsWriter(SegmentWriteState,PostingsWriterBase,int,int)`.

  See Also:
    Constant Field Values

### TERMS_VERSION_START

    public static final int TERMS_VERSION_START

  Initial terms format.

  See Also:
    Constant Field Values

### TERMS_VERSION_APPEND_ONLY

    public static final int TERMS_VERSION_APPEND_ONLY

  Append-only

  See Also:
    Constant Field Values

### TERMS_VERSION_CURRENT

    public static final int TERMS_VERSION_CURRENT

  Current terms format.

  See Also:
    Constant Field Values

### TERMS_INDEX_VERSION_START

    public static final int TERMS_INDEX_VERSION_START

  Initial index format.

  See Also:
    Constant Field Values

### TERMS_INDEX_VERSION_APPEND_ONLY

    public static final int TERMS_INDEX_VERSION_APPEND_ONLY

  Append-only

  See Also:
    Constant Field Values

### TERMS_INDEX_VERSION_CURRENT

    public static final int TERMS_INDEX_VERSION_CURRENT

  Current index format.

  See Also:
    Constant Field Values

Constructor Detail

### BlockTreeTermsWriter

    public BlockTreeTermsWriter(SegmentWriteState state,
                                PostingsWriterBase postingsWriter,
                                int minItemsInBlock,
                                int maxItemsInBlock)
                         throws IOException

  Create a new writer. The number of items (terms or sub-blocks) per block will aim to be between minItemsPerBlock and maxItemsPerBlock, though in some cases the blocks may be smaller than the min.

  Throws:
    `IOException`

Method Detail

### writeHeader

    protected void writeHeader(IndexOutput out)
                        throws IOException

  Writes the terms file header.

  Throws:
    `IOException`

### writeIndexHeader

    protected void writeIndexHeader(IndexOutput out)
                             throws IOException

  Writes the index file header.

  Throws:
    `IOException`

### writeTrailer

    protected void writeTrailer(IndexOutput out,
                                long dirStart)
                         throws IOException

  Writes the terms file trailer.

  Throws:
    `IOException`

### writeIndexTrailer

    protected void writeIndexTrailer(IndexOutput indexOut,
                                     long dirStart)
                              throws IOException

  Writes the index file trailer.

  Throws:
    `IOException`

### addField

    public TermsConsumer addField(FieldInfo field)
                           throws IOException

  Description copied from class: `FieldsConsumer`
  Add a new field

  Specified by:
    `addField` in class `FieldsConsumer`
  Throws:
    `IOException`

### close

    public void close()
               throws IOException

  Description copied from class: `FieldsConsumer`
  Called when we are done adding everything.

  Specified by:
    `close` in interface `Closeable`
  Specified by:
    `close` in class `FieldsConsumer`
  Throws:
    `IOException`

Overview | Package | Class | Use | Tree | Deprecated | Help
PREV CLASS NEXT CLASS | FRAMES NO FRAMES
SUMMARY: NESTED | FIELD | CONSTR | METHOD | DETAIL: FIELD | CONSTR | METHOD

Copyright © 2000-2013 Apache Software Foundation. All Rights Reserved.