com.ibm.icu4jni.text
Class Normalizer

java.lang.Object
  extended bycom.ibm.icu4jni.text.Normalizer

public final class Normalizer
extends java.lang.Object

Unicode Normalization

Unicode normalization API

normalize transforms Unicode text into an equivalent composed or decomposed form, allowing for easier sorting and searching of text. normalize supports the standard normalization forms described in Unicode Standard Annex #15 — Unicode Normalization Forms. Characters with accents or other adornments can be encoded in several different ways in Unicode. For example, take the character A-acute. In Unicode, this can be encoded as a single character (the "composed" form):
      00C1    LATIN CAPITAL LETTER A WITH ACUTE
 
or as two separate characters (the "decomposed" form):
      0041    LATIN CAPITAL LETTER A
      0301    COMBINING ACUTE ACCENT
 
To a user of your program, however, both of these sequences should be treated as the same "user-level" character "A with acute accent". When you are searching or comparing text, you must ensure that these two sequences are treated equivalently. In addition, you must handle characters with more than one accent. Sometimes the order of a character's combining accents is significant, while in other cases accent sequences in different orders are really equivalent. Similarly, the string "ffi" can be encoded as three separate letters:
      0066    LATIN SMALL LETTER F
      0066    LATIN SMALL LETTER F
      0069    LATIN SMALL LETTER I
 
or as the single character
      FB03    LATIN SMALL LIGATURE FFI
 
The ffi ligature is not a distinct semantic character, and strictly speaking it shouldn't be in Unicode at all, but it was included for compatibility with existing character sets that already provided it. The Unicode standard identifies such characters by giving them "compatibility" decompositions into the corresponding semantic characters. When sorting and searching, you will often want to use these mappings. normalize helps solve these problems by transforming text into the canonical composed and decomposed forms as shown in the first example above. In addition, you can have it perform compatibility decompositions so that you can treat compatibility characters the same as their equivalents. Finally, normalize rearranges accents into the proper canonical order, so that you do not have to worry about accent rearrangement on your own. Form FCD, "Fast C or D", is also designed for collation. It allows to work on strings that are not necessarily normalized with an algorithm (like in collation) that works under "canonical closure", i.e., it treats precomposed characters and their decomposed equivalents the same. It is not a normalization form because it does not provide for uniqueness of representation. Multiple strings may be canonically equivalent (their NFDs are identical) and may all conform to FCD without being identical themselves. The form is defined such that the "raw decomposition", the recursive canonical decomposition of each character, results in a string that is canonically ordered. This means that precomposed characters are allowed for as long as their decompositions do not need canonical reordering. Its advantage for a process like collation is that all NFD and most NFC texts - and many unnormalized texts - already conform to FCD and do not need to be normalized (NFD) for such a process. The FCD quick check will return UNORM_YES for most strings in practice. normalize(UNORM_FCD) may be implemented with UNORM_NFD. For more details on FCD see the collation design document: http://dev.icu-project.org/cgi-bin/viewcvs.cgi/~checkout~/icuhtml/design/collation/ICU_collation_design.htm ICU collation performs either NFD or FCD normalization automatically if normalization is turned on for the collator object. Beyond collation and string search, normalized strings may be useful for string equivalence comparisons, transliteration/transcription, unique representations, etc. The W3C generally recommends to exchange texts in NFC. Note also that most legacy character encodings use only precomposed forms and often do not encode any combining marks by themselves. For conversion to such character encodings the Unicode text needs to be normalized to NFC. For more usage examples, see the Unicode Standard Annex.

Status:
Stable ICU 2.4.

Field Summary
static int UNORM_DEFAULT
          Default normalization
static int UNORM_FCD
          "Fast C or D" form
static int UNORM_MAYBE
          Indicates that string cannot be determined if it is in the normalized format without further thorough checks
static int UNORM_NFC
          Canonical decomposition followed by canonical composition
static int UNORM_NFD
          Canonical decomposition
static int UNORM_NFKC
          Compatibility decomposition followed by canonical composition
static int UNORM_NFKD
          Compatibility decomposition
static int UNORM_NO
          Indicates that string is not in the normalized format
static int UNORM_NONE
          No decomposition/composition
static int UNORM_YES
          Indicates that string is in the normalized format
 
Constructor Summary
Normalizer()
           
 
Method Summary
static boolean check(int normalization)
          Checks if argument is a valid normalization format for use
static int compose(char[] source, char[] target, boolean compat)
          Compose a string.
static java.lang.String compose(java.lang.String str, boolean compat)
          Compose a string.
static int decompose(char[] source, char[] target, boolean compat)
          Decompose a string.
static java.lang.String decompose(java.lang.String str, boolean compat)
          Decompose a string.
static int normalize(char[] source, char[] target, int normalizationMode)
          Normalize a string.
static java.lang.String normalize(java.lang.String str, int normalizationMode)
          Normalize a string.
static int quickCheck(char[] source, int mode)
          Performing quick check on a string, to quickly determine if the string is in a particular normalization format.
static int quickCheck(java.lang.String source, int mode)
          Conveinience method.
 
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Field Detail

UNORM_NONE

public static final int UNORM_NONE
No decomposition/composition

See Also:
Constant Field Values
Status:
Stable ICU 2.4.

UNORM_NFD

public static final int UNORM_NFD
Canonical decomposition

See Also:
Constant Field Values
Status:
Stable ICU 2.4.

UNORM_NFKD

public static final int UNORM_NFKD
Compatibility decomposition

See Also:
Constant Field Values
Status:
Stable ICU 2.4.

UNORM_NFC

public static final int UNORM_NFC
Canonical decomposition followed by canonical composition

See Also:
Constant Field Values
Status:
Stable ICU 2.4.

UNORM_DEFAULT

public static final int UNORM_DEFAULT
Default normalization

See Also:
Constant Field Values
Status:
Stable ICU 2.4.

UNORM_NFKC

public static final int UNORM_NFKC
Compatibility decomposition followed by canonical composition

See Also:
Constant Field Values
Status:
Stable ICU 2.4.

UNORM_FCD

public static final int UNORM_FCD
"Fast C or D" form

See Also:
Constant Field Values
Status:
Stable ICU 2.4.

UNORM_NO

public static final int UNORM_NO
Indicates that string is not in the normalized format

See Also:
Constant Field Values
Status:
Stable ICU 2.4.

UNORM_YES

public static final int UNORM_YES
Indicates that string is in the normalized format

See Also:
Constant Field Values
Status:
Stable ICU 2.4.

UNORM_MAYBE

public static final int UNORM_MAYBE
Indicates that string cannot be determined if it is in the normalized format without further thorough checks

See Also:
Constant Field Values
Status:
Stable ICU 2.4.
Constructor Detail

Normalizer

public Normalizer()
Method Detail

compose

public static java.lang.String compose(java.lang.String str,
                                       boolean compat)
                                throws java.lang.Exception
Compose a string. The string will be composed to according the the specified mode.

Parameters:
str - The string to compose.
compat - If true the char array will be composed accoding to NFKC rules and if false will be composed according to NFC rules.
Returns:
String The composed string
Throws:
java.lang.Exception
Status:
Stable ICU 2.4.

compose

public static int compose(char[] source,
                          char[] target,
                          boolean compat)
                   throws java.lang.Exception
Compose a string. The string will be composed to according the the specified mode.

Parameters:
source - The char array to compose.
target - A char buffer to receive the normalized text.
compat - If true the char array will be composed accoding to NFKC rules and if false will be composed according to NFC rules.
Returns:
int The total buffer size needed;if greater than length of result, the output was truncated.
Throws:
java.lang.Exception
Status:
Stable ICU 2.4.

decompose

public static java.lang.String decompose(java.lang.String str,
                                         boolean compat)
                                  throws java.lang.Exception
Decompose a string. The string will be decomposed to according the the specified mode.

Parameters:
str - The string to decompose.
compat - If true the char array will be decomposed accoding to NFKD rules and if false will be decomposed according to NFD rules.
Returns:
String The decomposed string
Throws:
java.lang.Exception
Status:
Stable ICU 2.4.

decompose

public static int decompose(char[] source,
                            char[] target,
                            boolean compat)
                     throws java.lang.Exception
Decompose a string. The string will be decomposed to according the the specified mode.

Parameters:
source - The char array to decompose.
target - A char buffer to receive the normalized text.
compat - If true the char array will be decomposed accoding to NFKD rules and if false will be decomposed according to NFD rules.
Returns:
int The total buffer size needed;if greater than length of result, the output was truncated.
Throws:
java.lang.Exception
Status:
Stable ICU 2.4.

normalize

public static java.lang.String normalize(java.lang.String str,
                                         int normalizationMode)
                                  throws java.lang.Exception
Normalize a string. The string will be normalized according the the specified normalization mode and options.

Parameters:
str - The string to normalize.
normalizationMode - The normalization mode; one of Normalizer.UNORM_NONE, Normalizer.UNORM_NFD, Normalizer.UNORM_NFC, Normalizer.UNORM_NFKC, Normalizer.UNORM_NFKD, Normalizer.UNORM_DEFAULT
Returns:
String The normalized string
Throws:
java.lang.Exception
Status:
Stable ICU 2.4.

normalize

public static int normalize(char[] source,
                            char[] target,
                            int normalizationMode)
                     throws java.lang.Exception
Normalize a string. The string will be normalized according the the specified normalization mode and options.

Parameters:
source - The char array to normalize.
target - A char buffer to receive the normalized text.
normalizationMode - The normalization mode; one of Normalizer.UNORM_NONE, Normalizer.UNORM_NFD, Normalizer.UNORM_NFC, Normalizer.UNORM_NFKC, Normalizer.UNORM_NFKD, Normalizer.UNORM_DEFAULT
Returns:
int The total buffer size needed;if greater than length of result, the output was truncated.
Throws:
java.lang.Exception
Status:
Stable ICU 2.4.

quickCheck

public static int quickCheck(java.lang.String source,
                             int mode)
                      throws java.lang.Exception
Conveinience method.

Parameters:
source - string for determining if it is in a normalized format
mode - normalization format (Normalizer.UNORM_NFC,Normalizer.UNORM_NFD, Normalizer.UNORM_NFKC,Normalizer.UNORM_NFKD)
Returns:
Return code to specify if the text is normalized or not (Normalizer.UNORM_YES, Normalizer.UNORM_NO or Normalizer.UNORM_MAYBE)
Throws:
java.lang.Exception
Status:
Stable ICU 2.4.

quickCheck

public static int quickCheck(char[] source,
                             int mode)
Performing quick check on a string, to quickly determine if the string is in a particular normalization format. Three types of result can be returned Normalizer.UNORM_YES, Normalizer.UNORM_NO or Normalizer.UNORM_MAYBE. Result Normalizer.UNORM_YES indicates that the argument string is in the desired normalized format, Normalizer.UNORM_NO determines that argument string is not in the desired normalized format. A Normalizer.UNORM_MAYBE result indicates that a more thorough check is required, the user may have to put the string in its normalized form and compare the results.

Parameters:
source - string for determining if it is in a normalized format
mode - normalization format (Normalizer.UNORM_NFC,Normalizer.UNORM_NFD, Normalizer.UNORM_NFKC,Normalizer.UNORM_NFKD)
Returns:
Return code to specify if the text is normalized or not (Normalizer.UNORM_YES, Normalizer.UNORM_NO or Normalizer.UNORM_MAYBE)
Status:
Stable ICU 2.4.

check

public static boolean check(int normalization)
Checks if argument is a valid normalization format for use

Parameters:
normalization - format
Returns:
true if strength is a valid collation strength, false otherwise
Status:
Stable ICU 2.4.