UnicodeTokenizer (1.1 API)

Overview

Package

Class

Use

Tree

Deprecated

Index

Help

PREV CLASS NEXT CLASS

FRAMES NO FRAMES

SUMMARY: NESTED | FIELD | CONSTR | METHOD

DETAIL: FIELD | CONSTR | METHOD

de.l3s.boilerpipe.util
Class UnicodeTokenizer

java.lang.Object
  de.l3s.boilerpipe.util.UnicodeTokenizer

Tokenizes text according to Unicode word boundaries and strips off non-word characters.

Constructor Summary
`UnicodeTokenizer()`

Method Summary
`static java.lang.String[]`	`tokenize(java.lang.CharSequence text)` Tokenizes the text and returns an array of tokens.

Methods inherited from class java.lang.Object
`clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait`

Constructor Detail

public UnicodeTokenizer()

Method Detail

public static java.lang.String[] tokenize(java.lang.CharSequence text)

Tokenizes the text and returns an array of tokens.