CharDet

Big5 frequency table by Taiwan’s Mandarin Promotion Council <www.edu.tw:81/mandr/>

128 –> 0.42261 256 –> 0.57851 512 –> 0.74851 1024 –> 0.89384 2048 –> 0.97583

Ideal Distribution Ratio = 0.74851/(1-0.74851) =2.98 Random Distribution Ration = 512/(5401-512)=0.105

Typical Distribution Ratio about 25% of Ideal one, still much higher than RDR

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Communicator client code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Communicator client code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Communicator client code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Universal charset detector code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 2001 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python
Shy Shalom - original C code

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is mozilla.org code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Universal charset detector code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 2001 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python
Shy Shalom - original C code

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is mozilla.org code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is mozilla.org code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is mozilla.org code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

128 –> 0.79 256 –> 0.92 512 –> 0.986 1024 –> 0.99944 2048 –> 0.99999

Idea Distribution Ratio = 0.98653 / (1-0.98653) = 73.24 Random Distribution Ration = 512 / (2350-512) = 0.279.

Typical Distribution Ratio

---

BEGIN LICENSE BLOCK ########################

The Original Code is mozilla.org code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

128 –> 0.42261 256 –> 0.57851 512 –> 0.74851 1024 –> 0.89384 2048 –> 0.97583

Idea Distribution Ratio = 0.74851/(1-0.74851) =2.98 Random Distribution Ration = 512/(5401-512)=0.105

Typical Distribution Ratio about 25% of Ideal one, still much higher than RDR

---

BEGIN LICENSE BLOCK ########################

The Original Code is mozilla.org code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

512 –> 0.79 – 0.79 1024 –> 0.92 – 0.13 2048 –> 0.98 – 0.06 6768 –> 1.00 – 0.02

Ideal Distribution Ratio = 0.79135/(1-0.79135) = 3.79 Random Distribution Ration = 512 / (3755 - 512) = 0.157

Typical Distribution Ratio about 25% of Ideal one, still much higher that RDR

---

BEGIN LICENSE BLOCK ########################

The Original Code is mozilla.org code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

windows-1255 / ISO-8859-8 code points of interest

---

128 –> 0.77094 256 –> 0.85710 512 –> 0.92635 1024 –> 0.97130 2048 –> 0.99431

Ideal Distribution Ratio = 0.92635 / (1-0.92635) = 12.58 Random Distribution Ration = 512 / (2965+62+83+86-512) = 0.191

Typical Distribution Ratio, 25% of IDR

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Communicator client code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

Character Mapping Table: this table is modified base on win1251BulgarianCharToOrderMap, so only number <64 is sure valid

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Communicator client code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Communicator client code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Universal charset detector code.

The Initial Developer of the Original Code is

Simon Montagu

Portions created by the Initial Developer are Copyright (C) 2005 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python
Shy Shalom - original C code
Shoshannah Forbes - original C code (?)

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Communicator client code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Communicator client code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Universal charset detector code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 2001 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python
Shy Shalom - original C code

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Universal charset detector code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 2001 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python
Shy Shalom - original C code
Proofpoint, Inc.

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Universal charset detector code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 2001 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python
Shy Shalom - original C code
Proofpoint, Inc.

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is mozilla.org code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Universal charset detector code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 2001 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python
Shy Shalom - original C code

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Universal charset detector code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 2001 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python
Shy Shalom - original C code

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is mozilla.org code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is Mozilla Universal charset detector code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 2001 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python
Shy Shalom - original C code

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

---

BEGIN LICENSE BLOCK ########################

The Original Code is mozilla.org code.

The Initial Developer of the Original Code is Netscape Communications Corporation. Portions created by the Initial Developer are Copyright (C) 1998 the Initial Developer. All Rights Reserved.

Contributor(s):

Jeff Hodges - port to Ruby
Mark Pilgrim - port to Python

This library is free software; you can redistribute it and/or modify it under the terms of the GNU Lesser General Public License as published by the Free Software Foundation; either version 2.1 of the License, or (at your option) any later version.

This library is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU Lesser General Public License for more details.

You should have received a copy of the GNU Lesser General Public License along with this library; if not, write to the Free Software Foundation, Inc., 51 Franklin St, Fifth Floor, Boston, MA 02110-1301 USA

END LICENSE BLOCK #########################

# File lib/rchardet.rb, line 58
def CharDet.detect(aBuf)
  aBuf = aBuf.dup.force_encoding(Encoding::BINARY) unless aBuf.valid_encoding?

  u = UniversalDetector.new
  u.reset
  u.feed(aBuf)
  u.close
  u.result
end