marytts.features

Class FeatureDefinition

java.lang.Object

marytts.features.FeatureDefinition

public class FeatureDefinition
extends Object

A feature definition object represents the "meaning" of feature vectors. It consists of a list of byte-valued, short-valued and continuous features by name and index position in the feature vector; the respective possible feature values (and corresponding byte and short codes); and, optionally, the weights and, for continuous features, weighting functions for each feature.

Author:

Marc Schröder, steiner

Field Summary

Fields

Modifier and Type	Field and Description
static String	BYTEFEATURES
static String	CONTINUOUSFEATURES
static String	EDGEFEATURE
static String	EDGEFEATURE_END
static String	EDGEFEATURE_START
static String	FEATURESIMILARITY
static String	NULLVALUE
static String	SHORTFEATURES
static char	WEIGHT_SEPARATOR

Constructor Summary

Constructors

Constructor and Description
FeatureDefinition(BufferedReader input, boolean readWeights) Create a feature definition object, reading textual data from the given BufferedReader.
FeatureDefinition(ByteBuffer bb) Create a feature definition object, reading binary data from the given byte buffer.
FeatureDefinition(DataInput input) Create a feature definition object, reading binary data from the given DataInput.

Method Summary

Methods

Modifier and Type	Method and Description
boolean	contains(FeatureDefinition other) Determine whether this FeatureDefinition is a superset of, or equal to, another FeatureDefinition.
FeatureVector	createEdgeFeatureVector(int unitIndex, boolean start) Create a feature vector that marks a start or end of a unit.
static int	diff(FeatureVector v1, FeatureVector v2) Compares two feature vectors in terms of how many discrete features they have in common.
boolean	equals(Object obj) Determine whether two feature definitions are equal, regarding both the actual feature definitions and the weights.
boolean	featureEquals(FeatureDefinition other) Determine whether two feature definitions are equal, with respect to number, names, and possible values of the three kinds of features (byte-valued, short-valued, continuous).
String	featureEqualsAnalyse(FeatureDefinition other) An extension of the previous method.
void	generateAllDotDescForWagon(PrintWriter out) Export this feature definition in the "all.desc" format which can be read by wagon.
void	generateAllDotDescForWagon(PrintWriter out, Set<String> featuresToIgnore) Export this feature definition in the "all.desc" format which can be read by wagon.
void	generateFeatureWeightsFile(PrintWriter out) Print this feature definition plus weights to a .txt file
String[]	getByteFeatureNameArray() Get names of byte features
String[]	getContinuousFeatureNameArray() Get names of continuous features
int	getFeatureIndex(String featureName) Translate between a feature name and a feature index.
int[]	getFeatureIndexArray(String[] featureName) Translate between an array of feature names and an array of feature indexes.
String	getFeatureName(int index) Translate between a feature index and a feature name.
String[]	getFeatureNameArray() Get names of all features
String[]	getFeatureNameArray(int[] index) Translate between an array of feature indexes and an array of feature names.
String	getFeatureNames() List all feature names, separated by white space, in their order of definition.
byte	getFeatureValueAsByte(int featureIndex, String value) For the feature with the given index number, translate its String value to its byte value.
byte	getFeatureValueAsByte(String featureName, String value) For the feature with the given name, translate its String value to its byte value.
short	getFeatureValueAsShort(int featureIndex, String value) For the feature with the given name, translate its String value to its short value.
short	getFeatureValueAsShort(String featureName, String value) For the feature with the given name, translate its String value to its short value.
String	getFeatureValueAsString(int featureIndex, int value) For the feature with the given index number, translate its byte or short value to its String value.
String	getFeatureValueAsString(String featureName, FeatureVector fv) Simple access to string-based features.
float[]	getFeatureWeights()
int	getNumberOfByteFeatures() Get the number of byte features.
int	getNumberOfContinuousFeatures() Get the number of continuous features.
int	getNumberOfFeatures() Get the total number of features.
int	getNumberOfShortFeatures() Get the number of short features.
int	getNumberOfValues(int featureIndex) Get the number of possible values for the feature with the given index number.
String[]	getPossibleValues(int featureIndex) Get the list of possible String values for the feature with the given index number.
String[]	getShortFeatureNameArray() Get names of short features
float	getSimilarity(int featureIndex, byte i, byte j) To get a similarity between two feature values
float	getWeight(int featureIndex) For the feature with the given index, return the weight.
String	getWeightFunctionName(int featureIndex) Get the name of any weighting function associated with the given feature index.
boolean	hasFeature(String name) Indicate whether the feature definition contains the feature with the given name
boolean	hasFeatureValue(int featureIndex, String featureValue) Query a feature as identified by the given featureIndex as to whether the given featureValue is a known value of that feature.
boolean	hasFeatureValue(String featureName, String featureValue) Query a feature as identified by the given featureName as to whether the given featureValue is a known value of that feature.
boolean	hasSimilarityMatrix(int featureIndex) true, if given feature index contains similarity matrix
boolean	hasSimilarityMatrix(String featureName) true, if given feature name contains similarity matrix
boolean	isByteFeature(int index) Determine whether the feature with the given index number is a byte feature.
boolean	isByteFeature(String featureName) Determine whether the feature with the given name is a byte feature.
boolean	isContinuousFeature(int index) Determine whether the feature with the given index number is a continuous feature.
boolean	isContinuousFeature(String featureName) Determine whether the feature with the given name is a continuous feature.
boolean	isShortFeature(int index) Determine whether the feature with the given index number is a short feature.
boolean	isShortFeature(String featureName) Determine whether the feature with the given name is a short feature.
FeatureVector	readFeatureVector(int currentUnitIndex, ByteBuffer bb) Create a feature vector consistent with this feature definition by reading the data from the byte buffer.
FeatureVector	readFeatureVector(int currentUnitIndex, DataInput input) Create a feature vector consistent with this feature definition by reading the data from the given input.
FeatureDefinition	subset(String[] featureNamesToDrop) Create a new FeatureDefinition that contains a subset of the features in this.
String	toFeatureString(FeatureVector fv) Convert a feature vector into a String representation.
FeatureVector	toFeatureVector(int unitIndex, byte[] bytes, short[] shorts, float[] floats)
FeatureVector	toFeatureVector(int unitIndex, String featureString) Create a feature vector consistent with this feature definition by reading the data from a String representation.
void	writeBinaryTo(DataOutput out) Write this feature definition in binary format to the given output.
void	writeTo(PrintWriter out, boolean writeWeights) Export this feature definition in the text format which can also be read by this class.

Methods inherited from class java.lang.Object

clone, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Field Detail

BYTEFEATURES

public static final String BYTEFEATURES

See Also:

Constant Field Values

SHORTFEATURES

public static final String SHORTFEATURES

See Also:

Constant Field Values

CONTINUOUSFEATURES

public static final String CONTINUOUSFEATURES

See Also:

Constant Field Values

FEATURESIMILARITY

public static final String FEATURESIMILARITY

See Also:

Constant Field Values

WEIGHT_SEPARATOR

public static final char WEIGHT_SEPARATOR

See Also:

Constant Field Values

EDGEFEATURE

public static final String EDGEFEATURE

See Also:

Constant Field Values

EDGEFEATURE_START

public static final String EDGEFEATURE_START

See Also:

Constant Field Values

EDGEFEATURE_END

public static final String EDGEFEATURE_END

See Also:

Constant Field Values

NULLVALUE

public static final String NULLVALUE

See Also:

Constant Field Values

Constructor Detail

FeatureDefinition

public FeatureDefinition(BufferedReader input,
                 boolean readWeights)
                  throws IOException

Create a feature definition object, reading textual data from the given BufferedReader.

Parameters:

input - a BufferedReader from which a textual feature definition can be read.

readWeights - a boolean indicating whether or not to read weights from input. If weights are read, they will be normalized so that they sum to one.

Throws:

IOException - if a reading problem occurs

FeatureDefinition

public FeatureDefinition(DataInput input)
                  throws IOException

Create a feature definition object, reading binary data from the given DataInput.

Parameters:

input - a DataInputStream or a RandomAccessFile from which a binary feature definition can be read.

Throws:

IOException - if a reading problem occurs

FeatureDefinition

public FeatureDefinition(ByteBuffer bb)
                  throws IOException

Create a feature definition object, reading binary data from the given byte buffer.

Parameters:

bb - a byte buffer from which a binary feature definition can be read.

Throws:

IOException - if a reading problem occurs

Method Detail

writeBinaryTo

public void writeBinaryTo(DataOutput out)
                   throws IOException

Write this feature definition in binary format to the given output.

Parameters:

out - a DataOutputStream or RandomAccessFile to which the FeatureDefinition should be written.

Throws:

IOException - if a problem occurs while writing.

getNumberOfFeatures

public int getNumberOfFeatures()

Get the total number of features.

Returns:

the number of features

getNumberOfByteFeatures

public int getNumberOfByteFeatures()

Get the number of byte features.

Returns:

the number of features

getNumberOfShortFeatures

public int getNumberOfShortFeatures()

Get the number of short features.

Returns:

the number of features

getNumberOfContinuousFeatures

public int getNumberOfContinuousFeatures()

Get the number of continuous features.

Returns:

the number of features

getWeight

public float getWeight(int featureIndex)

For the feature with the given index, return the weight.

Parameters:

featureIndex - featureIndex

Returns:

a non-negative weight.

getFeatureWeights

public float[] getFeatureWeights()

getWeightFunctionName

public String getWeightFunctionName(int featureIndex)

Get the name of any weighting function associated with the given feature index. For byte-valued and short-valued features, this method will always return null; for continuous features, the method will return the name of a weighting function, or null.

Parameters:

featureIndex - featureIndex

Returns:

the name of a weighting function, or null

getFeatureName

public String getFeatureName(int index)

Translate between a feature index and a feature name.

Parameters:

index - a feature index, as could be used to access a feature value in a FeatureVector.

Returns:

the name of the feature corresponding to the index

Throws:

IndexOutOfBoundsException - if index<0 or index>getNumberOfFeatures()

getFeatureNameArray

public String[] getFeatureNameArray(int[] index)

Translate between an array of feature indexes and an array of feature names.

Parameters:

index - an array of feature indexes, as could be used to access a feature value in a FeatureVector.

Returns:

an array with the name of the features corresponding to the index

Throws:

IndexOutOfBoundsException - if any of the indexes is <0 or >getNumberOfFeatures()

getFeatureNameArray

public String[] getFeatureNameArray()

Get names of all features

Returns:

an array of all feature name strings

getByteFeatureNameArray

public String[] getByteFeatureNameArray()

Get names of byte features

Returns:

an array of byte feature name strings

getShortFeatureNameArray

public String[] getShortFeatureNameArray()

Get names of short features

Returns:

an array of short feature name strings

getContinuousFeatureNameArray

public String[] getContinuousFeatureNameArray()

Get names of continuous features

Returns:

an array of continuous feature name strings

getFeatureNames

public String getFeatureNames()

List all feature names, separated by white space, in their order of definition.

Returns:

buf converted into a string

hasFeature

public boolean hasFeature(String name)

Indicate whether the feature definition contains the feature with the given name

Parameters:

name - the feature name in question, e.g. "next_next_phone"

Returns:

featureNames.contains(name)

hasFeatureValue

public boolean hasFeatureValue(String featureName,
                      String featureValue)

Query a feature as identified by the given featureName as to whether the given featureValue is a known value of that feature. In other words, this will return true exactly if the given feature is a byte feature and getFeatureValueAsByte(featureName, featureValue) will not throw an exception or if the given feature is a short feature and getFeatureValueAsShort(featureName, featureValue) will not throw an exception.

Parameters:

featureName - featureName

featureValue - featureValue

Returns:

hasFeatureValue(getFeatureIndex(featureName), featureValue)

hasFeatureValue

public boolean hasFeatureValue(int featureIndex,
                      String featureValue)

Query a feature as identified by the given featureIndex as to whether the given featureValue is a known value of that feature. In other words, this will return true exactly if the given feature is a byte feature and getFeatureValueAsByte(featureIndex, featureValue) will not throw an exception or if the given feature is a short feature and getFeatureValueAsShort(featureIndex, featureValue) will not throw an exception.

Parameters:

featureIndex - featureIndex

featureValue - featureValue

Returns:

false if featureIndex < 0, byteFeatureValues[featureIndex].contains(featureValue) if featureIndex < numByteFeatures, shortFeatureValues[featureIndex - numByteFeatures].contains(featureValue) if featureIndex < numByteFeatures + numShortFeatures, false otherwise

isByteFeature

public boolean isByteFeature(String featureName)

Determine whether the feature with the given name is a byte feature.

Parameters:

featureName - featureName

Returns:

true if the feature is a byte feature, false if the feature is not known or is not a byte feature

isByteFeature

public boolean isByteFeature(int index)

Determine whether the feature with the given index number is a byte feature.

Parameters:

index - index

Returns:

true if the feature is a byte feature, false if the feature is not a byte feature or is invalid

isShortFeature

public boolean isShortFeature(String featureName)

Determine whether the feature with the given name is a short feature.

Parameters:

featureName - featureName

Returns:

true if the feature is a short feature, false if the feature is not known or is not a short feature

isShortFeature

public boolean isShortFeature(int index)

Determine whether the feature with the given index number is a short feature.

Parameters:

index - index

Returns:

true if the feature is a short feature, false if the feature is not a short feature or is invalid

isContinuousFeature

public boolean isContinuousFeature(String featureName)

Determine whether the feature with the given name is a continuous feature.

Parameters:

featureName - featureName

Returns:

true if the feature is a continuous feature, false if the feature is not known or is not a continuous feature

isContinuousFeature

public boolean isContinuousFeature(int index)

Determine whether the feature with the given index number is a continuous feature.

Parameters:

index - index

Returns:

true if the feature is a continuous feature, false if the feature is not a continuous feature or is invalid

hasSimilarityMatrix

public boolean hasSimilarityMatrix(int featureIndex)

true, if given feature index contains similarity matrix

Parameters:

featureIndex - featureIndex

Returns:

true if this.similarityMatrices different from null and this.similarityMatrices[featureIndex] different from null, false otherwise

hasSimilarityMatrix

public boolean hasSimilarityMatrix(String featureName)

true, if given feature name contains similarity matrix

Parameters:

featureName - featureName

Returns:

hasSimilarityMatrix(this.getFeatureIndex(featureName))

getSimilarity

public float getSimilarity(int featureIndex,
                  byte i,
                  byte j)

To get a similarity between two feature values

Parameters:

featureIndex - featureIndex

i - i

j - j

Returns:

this.similarityMatrices[featureIndex][i][j]

getFeatureIndex

public int getFeatureIndex(String featureName)

Translate between a feature name and a feature index.

Parameters:

featureName - a valid feature name

Returns:

a feature index, as could be used to access a feature value in a FeatureVector.

Throws:

IllegalArgumentException - if the feature name is unknown.

getFeatureIndexArray

public int[] getFeatureIndexArray(String[] featureName)

Translate between an array of feature names and an array of feature indexes.

Parameters:

featureName - an array of valid feature names

Returns:

an array of feature indexes, as could be used to access a feature value in a FeatureVector.

Throws:

IllegalArgumentException - if one of the feature names is unknown.

getNumberOfValues

public int getNumberOfValues(int featureIndex)

Get the number of possible values for the feature with the given index number. This method must only be called for byte-valued or short-valued features.

Parameters:

featureIndex - the index number of the feature.

Returns:

for byte-valued and short-valued features, return the number of values.

Throws:

IndexOutOfBoundsException - if featureIndex < 0 or featureIndex ≥ getNumberOfByteFeatures() + getNumberOfShortFeatures().

getPossibleValues

public String[] getPossibleValues(int featureIndex)

Get the list of possible String values for the feature with the given index number. This method must only be called for byte-valued or short-valued features. The position in the String array corresponds to the byte or short value of the feature obtained from a FeatureVector.

Parameters:

featureIndex - the index number of the feature.

Returns:

for byte-valued and short-valued features, return the array of String values.

Throws:

IndexOutOfBoundsException - if featureIndex < 0 or featureIndex ≥ getNumberOfByteFeatures() + getNumberOfShortFeatures().

getFeatureValueAsString

public String getFeatureValueAsString(int featureIndex,
                             int value)

For the feature with the given index number, translate its byte or short value to its String value. This method must only be called for byte-valued or short-valued features.

Parameters:

featureIndex - the index number of the feature.

value - the feature value. This must be in the range of acceptable values for the given feature.

Returns:

for byte-valued and short-valued features, return the String representation of the feature value.

Throws:

IndexOutOfBoundsException - if featureIndex < 0 or featureIndex ≥ getNumberOfByteFeatures() + getNumberOfShortFeatures()

IndexOutOfBoundsException - if value is not a legal value for this feature

getFeatureValueAsString

public String getFeatureValueAsString(String featureName,
                             FeatureVector fv)

Simple access to string-based features.

Parameters:

featureName - featureName

fv - fv

Returns:

getFeatureValueAsString(i, fv.getFeatureAsInt(i))F

getFeatureValueAsByte

public byte getFeatureValueAsByte(String featureName,
                         String value)

For the feature with the given name, translate its String value to its byte value. This method must only be called for byte-valued features.

Parameters:

featureName - the name of the feature.

value - the feature value. This must be among the acceptable values for the given feature.

Returns:

for byte-valued features, return the byte representation of the feature value.

Throws:

IllegalArgumentException - if featureName is not a valid feature name, or if featureName is not a byte-valued feature.

IllegalArgumentException - if value is not a legal value for this feature

getFeatureValueAsByte

public byte getFeatureValueAsByte(int featureIndex,
                         String value)

For the feature with the given index number, translate its String value to its byte value. This method must only be called for byte-valued features.

Parameters:

featureIndex - the name of the feature.

value - the feature value. This must be among the acceptable values for the given feature.

Returns:

for byte-valued features, return the byte representation of the feature value.

Throws:

IllegalArgumentException - if featureName is not a valid feature name, or if featureName is not a byte-valued feature.

IllegalArgumentException - if value is not a legal value for this feature

getFeatureValueAsShort

public short getFeatureValueAsShort(String featureName,
                           String value)

For the feature with the given name, translate its String value to its short value. This method must only be called for short-valued features.

Parameters:

featureName - the name of the feature.

value - the feature value. This must be among the acceptable values for the given feature.

Returns:

for short-valued features, return the short representation of the feature value.

Throws:

IllegalArgumentException - if featureName is not a valid feature name, or if featureName is not a short-valued feature.

IllegalArgumentException - if value is not a legal value for this feature

getFeatureValueAsShort

public short getFeatureValueAsShort(int featureIndex,
                           String value)

For the feature with the given name, translate its String value to its short value. This method must only be called for short-valued features.

Parameters:

featureIndex - the name of the feature.

value - the feature value. This must be among the acceptable values for the given feature.

Returns:

for short-valued features, return the short representation of the feature value.

Throws:

IllegalArgumentException - if featureName is not a valid feature name, or if featureName is not a short-valued feature.

IllegalArgumentException - if value is not a legal value for this feature

featureEquals

public boolean featureEquals(FeatureDefinition other)

Determine whether two feature definitions are equal, with respect to number, names, and possible values of the three kinds of features (byte-valued, short-valued, continuous). This method does not compare any weights.

Parameters:

other - the feature definition to compare to

Returns:

true if all features and values are identical, false otherwise

featureEqualsAnalyse

public String featureEqualsAnalyse(FeatureDefinition other)

An extension of the previous method.

Parameters:

other - other

Returns:

number of byte features, or number of short features, or number of continuous features, or feature name

equals

public boolean equals(Object obj)

Determine whether two feature definitions are equal, regarding both the actual feature definitions and the weights. The comparison of weights will succeed if both have no weights or if both have exactly the same weights

Overrides:

equals in class Object

Parameters:

obj - the feature definition to compare to

Returns:

true if all features, values and weights are identical, false otherwise

See Also:

featureEquals(FeatureDefinition)

contains

public boolean contains(FeatureDefinition other)

Determine whether this FeatureDefinition is a superset of, or equal to, another FeatureDefinition.

Specifically,

1. every byte-valued feature in other must be in this, likewise for short-valued and continuous-valued features;
2. for byte-valued and short-valued features, the possible feature values must be the same in this and other.

Parameters:

other - FeatureDefinition

Returns:

true if

1. all features in other are also in this, and every feature in other is of the same type in this; and
2. every feature in other has the same possible values as the feature in this

false otherwise

subset

public FeatureDefinition subset(String[] featureNamesToDrop)

Create a new FeatureDefinition that contains a subset of the features in this.

Parameters:

featureNamesToDrop - array of Strings containing the names of the features to drop from the new FeatureDefinition

Returns:

new FeatureDefinition

toFeatureVector

public FeatureVector toFeatureVector(int unitIndex,
                            String featureString)

Create a feature vector consistent with this feature definition by reading the data from a String representation. In that String, the String values for each feature must be separated by white space. For example, this format is created by toFeatureString(FeatureVector).

Parameters:

unitIndex - an index number to assign to the feature vector

featureString - the string representation of a feature vector.

Returns:

the feature vector created from the String.

Throws:

IllegalArgumentException - if the feature values listed are not consistent with the feature definition.

See Also:

toFeatureString(FeatureVector)

toFeatureVector

public FeatureVector toFeatureVector(int unitIndex,
                            byte[] bytes,
                            short[] shorts,
                            float[] floats)

readFeatureVector

public FeatureVector readFeatureVector(int currentUnitIndex,
                              DataInput input)
                                throws IOException

Create a feature vector consistent with this feature definition by reading the data from the given input.

Parameters:

input - a DataInputStream or RandomAccessFile to read the feature values from.

currentUnitIndex - currentUnitIndex

Returns:

a FeatureVector.

Throws:

IOException - IOException

readFeatureVector

public FeatureVector readFeatureVector(int currentUnitIndex,
                              ByteBuffer bb)
                                throws IOException

Create a feature vector consistent with this feature definition by reading the data from the byte buffer.

Parameters:

currentUnitIndex - currentUnitIndex

bb - a byte buffer to read the feature values from.

Returns:

a FeatureVector.

Throws:

IOException - IOException

createEdgeFeatureVector

public FeatureVector createEdgeFeatureVector(int unitIndex,
                                    boolean start)

Create a feature vector that marks a start or end of a unit. All feature values are set to the neutral value "0", except for the EDGEFEATURE, which is set to start if start == true, to end otherwise.

Parameters:

unitIndex - index of the unit

start - true creates a start vector, false creates an end vector.

Returns:

a feature vector representing an edge.

toFeatureString

public String toFeatureString(FeatureVector fv)

Convert a feature vector into a String representation.

Parameters:

fv - a feature vector which must be consistent with this feature definition.

Returns:

a String containing the String values of all features, separated by white space.

Throws:

IllegalArgumentException - if the feature vector is not consistent with this feature definition

IndexOutOfBoundsException - if any value of the feature vector is not consistent with this feature definition

writeTo

public void writeTo(PrintWriter out,
           boolean writeWeights)

Export this feature definition in the text format which can also be read by this class.

Parameters:

out - the destination of the data

writeWeights - whether to write weights before every line

generateAllDotDescForWagon

public void generateAllDotDescForWagon(PrintWriter out)

Export this feature definition in the "all.desc" format which can be read by wagon.

Parameters:

out - the destination of the data

generateAllDotDescForWagon

public void generateAllDotDescForWagon(PrintWriter out,
                              Set<String> featuresToIgnore)

Export this feature definition in the "all.desc" format which can be read by wagon.

Parameters:

out - the destination of the data

featuresToIgnore - a set of Strings containing the names of features that wagon should ignore. Can be null.

generateFeatureWeightsFile

public void generateFeatureWeightsFile(PrintWriter out)

Print this feature definition plus weights to a .txt file

Parameters:

out - the destination of the data

diff

public static int diff(FeatureVector v1,
       FeatureVector v2)

Compares two feature vectors in terms of how many discrete features they have in common. WARNING: this assumes that the feature vectors are issued from the same FeatureDefinition; only the number of features is checked for compatibility.

Parameters:

v1 - A feature vector.

v2 - Another feature vector to compare v1 with.

Returns:

The number of common features.