marytts.language.ru

Class Tokenizer

java.lang.Object

marytts.modules.InternalModule

marytts.modules.JTokeniser

marytts.language.ru.Tokenizer

All Implemented Interfaces:

MaryModule

public class Tokenizer
extends JTokeniser

Author:

Marc Schröder

Field Summary

Fields inherited from class marytts.modules.JTokeniser

TOKEN_MAXLENGTH

Fields inherited from class marytts.modules.InternalModule

logger, state

Fields inherited from interface marytts.modules.MaryModule

MODULE_OFFLINE, MODULE_RUNNING

Constructor Summary

Constructors

Constructor and Description
Tokenizer()

Method Summary

Methods

Modifier and Type	Method and Description
MaryData	process(MaryData d) Perform this module's processing on abstract "MaryData" input d.
protected void	splitOffDots(MaryData d) For Russian, treat all dots as standalone tokens that trigger end of sentence.

Methods inherited from class marytts.modules.JTokeniser

setTokenizerLanguage, startup

Methods inherited from class marytts.modules.InternalModule

getInputType, getLocale, getOutputType, getState, inputType, name, outputType, powerOnSelfTest, shutdown

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

Tokenizer

public Tokenizer()

Method Detail

process

public MaryData process(MaryData d)
                 throws Exception

Description copied from class: InternalModule

Perform this module's processing on abstract "MaryData" input d. Subclasses need to make sure that the process() method is thread-safe, because in server-mode, it will be called from different threads at the same time. A sensible way to do this seems to be not to use any global or static variables, or to use them read-only.

Specified by:

process in interface MaryModule

Overrides:

process in class JTokeniser

Parameters:

d - d

Returns:

A MaryData object of type outputType() encapsulating the processing result.

This method just returns its input. Subclasses should override this.

Throws:

Exception - Exception

splitOffDots

protected void splitOffDots(MaryData d)

For Russian, treat all dots as standalone tokens that trigger end of sentence.

Parameters:

d - d