marytts.language.en

Class Preprocess

java.lang.Object

marytts.modules.InternalModule

marytts.language.en.Preprocess

All Implemented Interfaces:

MaryModule

public class Preprocess
extends InternalModule

Author:

Tristan Hamilton

Can process following formats:

• cardinal (handled by real number)
• ordinal
• year (as a 4 digit number or any number followed by AD/BC variation)
• currency
• numberandword together
• dashes (read each number singly) or (split into two words)
• underscores
• decimal point, minus symbol (real numbers) also handles %, however Jtokeniser splits % into separate tokens
• time
• dates (in format mm/dd/yyyy)
• acronyms (only split into single characters, never expanded)
• abbreviations (list of known expansions in resource preprocess/abbrev.dat, a properties file separated by whitespace. If an abbrev has two different expansions then the capitalized version comes first, followed by a comma)
• contractions → first check lexicon, if not then → split and check if map contains contraction, if not then just remove apostrophe else → split before apostrophe into two tokens, use map to manually add ph → for 's if word ends in c,f,k,p,t then add ph = s otherwise ph = z
• ampersand &, "at" @ symbol, → symbols
• urls → note that jtokeniser splits off http[s]?://
• number ranges "18-35"
• words without vowels → first check lexicon, if not then separate into single character tokens
• #hashtags
• single "A/a" character → if there is no next token or the next token is punctuation or next token string.length == 1
• should also as a last processing attempt, split by punctuation,symbols,etc. and attempt to process these tokens separately
• durations hours:minutes:seconds(:milliseconds)
• numbers followed by an s
• punctuation → add ph attribute to tag to prevent phonemisation

May include:

• roman numerals

Field Summary

Fields

Modifier and Type	Field and Description
protected String	cardinalRule
protected String	ordinalRule
protected String	yearRule

Fields inherited from class marytts.modules.InternalModule

logger, state

Fields inherited from interface marytts.modules.MaryModule

MODULE_OFFLINE, MODULE_RUNNING

Constructor Summary

Constructors

Constructor and Description
Preprocess()

Method Summary

Methods

Modifier and Type	Method and Description
protected void	expand(Document doc) processes a document in mary xml format, from Tokens to Words which can be phonemised.
protected String	expandAbbreviation(String abbrev, boolean isCapital)
protected String	expandAcronym(String acronym)
protected String	expandConsonants(String consonants) add a space between each char of a string
protected String	expandDate(String date)
protected String	expandDuration(String duration)
protected String	expandHashtag(String hashtag)
protected String	expandMoney(String money, String currency)
protected String	expandNumber(double number)
protected String	expandNumberS(String numberS) expands a digit followed by an s.
protected String	expandOrdinal(double number)
protected String	expandRange(String range)
protected String	expandRealNumber(String number)
protected String	expandTime(String time, boolean isNextTokenTime)
protected String	expandURL(String email) expand a URL string partially by splitting by @, / and .
protected String	expandWordNumber(String wordnumseq)
protected String	expandYear(double number)
protected String	expandYearBCAD(String year)
protected static String	getOrdinalRuleName(com.ibm.icu.text.RuleBasedNumberFormat rbnf) Try to extract the rule name for "expand ordinal" from the given RuleBasedNumberFormat.
protected static String	getYearRuleName(com.ibm.icu.text.RuleBasedNumberFormat rbnf) Try to extract the rule name for "expand year" from the given RuleBasedNumberFormat.
static Map<Object,Object>	loadAbbrevMap()
MaryData	process(MaryData d) Perform this module's processing on abstract "MaryData" input d.
protected String	splitContraction(String contraction)

Methods inherited from class marytts.modules.InternalModule

getInputType, getLocale, getOutputType, getState, inputType, name, outputType, powerOnSelfTest, shutdown, startup

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

cardinalRule

protected final String cardinalRule

ordinalRule

protected final String ordinalRule

yearRule

protected final String yearRule

Constructor Detail

Preprocess

public Preprocess()

Method Detail

process

public MaryData process(MaryData d)
                 throws Exception

Description copied from class: InternalModule

Perform this module's processing on abstract "MaryData" input d. Subclasses need to make sure that the process() method is thread-safe, because in server-mode, it will be called from different threads at the same time. A sensible way to do this seems to be not to use any global or static variables, or to use them read-only.

Specified by:

process in interface MaryModule

Overrides:

process in class InternalModule

Parameters:

d - d

Returns:

A MaryData object of type outputType() encapsulating the processing result.

This method just returns its input. Subclasses should override this.

Throws:

Exception - Exception

expand

protected void expand(Document doc)
               throws ParseException,
                      IOException,
                      MaryConfigurationException

processes a document in mary xml format, from Tokens to Words which can be phonemised.

Parameters:

doc - doc

Throws:

ParseException - parse exception

IOException - IO Exception

MaryConfigurationException - mary configuration exception

expandNumber

protected String expandNumber(double number)

expandOrdinal

protected String expandOrdinal(double number)

expandYear

protected String expandYear(double number)

expandDuration

protected String expandDuration(String duration)

expandAcronym

protected String expandAcronym(String acronym)

expandURL

protected String expandURL(String email)

expand a URL string partially by splitting by @, / and . symbols (but retaining them)

Parameters:

email - email

Returns:

Arrays.toString(tokens).replaceAll("[,\\]\\[]", "")

expandYearBCAD

protected String expandYearBCAD(String year)

expandConsonants

protected String expandConsonants(String consonants)

add a space between each char of a string

Parameters:

consonants - consonants

Returns:

Joiner.on(" ").join(Lists.charactersOf(consonants))

expandHashtag

protected String expandHashtag(String hashtag)

expandRange

protected String expandRange(String range)

expandNumberS

protected String expandNumberS(String numberS)

expands a digit followed by an s. e.g. 7s and 8s and the 60s

Parameters:

numberS - numberS

Returns:

number

splitContraction

protected String splitContraction(String contraction)

expandAbbreviation

protected String expandAbbreviation(String abbrev,
                        boolean isCapital)

Parameters:

abbrev - the token to be expanded

isCapital - whether the following token begins with a capital letter

Returns:

abbrev

expandDate

protected String expandDate(String date)
                     throws ParseException

Throws:

ParseException

expandTime

protected String expandTime(String time,
                boolean isNextTokenTime)

Parameters:

time - the token to be expanded

isNextTokenTime - whether the following token contains am or pm

Returns:

theTime

expandRealNumber

protected String expandRealNumber(String number)

expandWordNumber

protected String expandWordNumber(String wordnumseq)

expandMoney

protected String expandMoney(String money,
                 String currency)

getOrdinalRuleName

protected static String getOrdinalRuleName(com.ibm.icu.text.RuleBasedNumberFormat rbnf)

Try to extract the rule name for "expand ordinal" from the given RuleBasedNumberFormat.

The rule name is locale sensitive, but usually starts with "%spellout-ordinal".

Parameters:

rbnf - The RuleBasedNumberFormat from where we will try to extract the rule name.

Returns:

The rule name for "ordinal spell out".

getYearRuleName

protected static String getYearRuleName(com.ibm.icu.text.RuleBasedNumberFormat rbnf)

Try to extract the rule name for "expand year" from the given RuleBasedNumberFormat.

The rule name is locale sensitive, but usually starts with "%spellout-numbering-year".

Parameters:

rbnf - The RuleBasedNumberFormat from where we will try to extract the rule name.

Returns:

The rule name for "year spell out".

loadAbbrevMap

public static Map<Object,Object> loadAbbrevMap()
                                        throws IOException

Throws:

IOException