yaz-icu
Prev | Reference | Next

## Name

yaz-icu — YAZ ICU utility

## Synopsis

`yaz-icu` [-c `config`] [-p `opt`] [-s] [-x] [infile]

## DESCRIPTION

yaz-icu is utility which demonstrates the ICU chain module of yaz. (`yaz/icu.h`).

The utility can be used in two ways. It may read some text using an XML configuration for configuring ICU and show text analysis. This mode is triggered by option `-c` which specififies the configuration to be used. The input file is read from standard input or from a file if `infile` is specified.

The utility may also show ICU information. This is triggered by option `-p`.

## OPTIONS

-c `config`

  Specifies the file containing ICU chain configuration which is XML based.

-p `type`

  Specifies extra information to be printed about the ICU system. If `type` is `c` then ICU converters are printed. If `type` is `l` available locales are printed. If `type` is `t` available transliterators are printed.

-s

  Specifies that output should include sort key as well. Note that sort key differs between ICU versions.

-x

  Specifies that output should be XML based rather than "text" based.

## ICU chain configuration

The ICU chain configuration speicifies one or more rules to convert text data into tokens. The configuration format is XML based.

The toplevel element must be named `icu_chain`. The `icu_chain` element has one required attribute `locale` which specifies the ICU locale to be used in the conversion steps.

The `icu_chain` element must include elements where each element specifies a conversion step. The conversion is performed in the order in which the conversion steps are specified. Each conversion element takes one attribute: `rule` which serves as argument to the conversion step.

The following conversion elements are available:

casemap

  Converts case and rule specifies how:

  l

    Lowercase using ICU function u_strToLower.

  u

    Upper case using ICU function u_strToUpper.

  t

    To title using UCU function u_strToTitle.

  f

    Fold case using ICU function u_strFoldCase.

display

  This is a meta step which specifies that a term/token is to be displayed. This term is retrieved in an application using function icu_chain_token_display (`yaz/icu.h`).

transform

  Specifies an ICU transform rule using a transliterator Identifier. The rule attribute is the transliterator Identifier. See ICU Transforms for more information.

transliterate

  Specifies a rule-based transliterator. The rule attribute is the custom transformation rule to be used. See ICU Transforms for more information.

tokenize

  Breaks / tokenizes a string into components using ICU functions ubrk_open, ubrk_setText, .. . The rule is one of:

  l

    Line. ICU: UBRK_LINE.

  s

    Sentence. ICU: UBRK_SENTENCE.

  w

    Word. ICU: UBRK_WORD.

  c

    Character. ICU: UBRK_CHARACTER.

  t

    Title. ICU: UBRK_TITLE.

join

  Joins tokens into one string. The rule attribute is the joining string, which may be empty. The join conversion element was added in YAZ 4.2.49.

## EXAMPLES

The following command analyzes text in file `text` using ICU chain configuration `chain.xml`:

        cat text | yaz-icu -c chain.xml

The chain.xml might look as follows:

    <icu_chain locale="en">
      <transform rule="[:Control:] Any-Remove"/>
      <tokenize rule="w"/>
      <transform rule="[[:WhiteSpace:][:Punctuation:]] Remove"/>
      <transliterate rule="xy > z;"/>
      <display/>
      <casemap rule="l"/>
    </icu_chain>

## SEE ALSO

yaz(7)

ICU Home

ICU Transforms

Prev | Up | Next
yaz-illclient | Home | yaz-url