#1. Convert numeric type from double to float

#2. Refactor configuration file and command line parameter
#3. use SIMD for backward pass in output layer

Author: zhongkaifu

README.md

@@ -22,11 +22,11 @@ Here is the neural network for sequence-to-sequence task. "TokenN" are from sour
 ![](https://github.com/zhongkaifu/RNNSharp/blob/master/RNNSharpSeq2Seq.jpg)
 
 ## Supported Feature Types
-RNNSharp supports four types of feature set. They are template features, context template features, run time feature and word embedding features. These features are controlled by configuration file, the following paragraph will introduce how these feaures work.
+RNNSharp supports many different feature types, so the following paragraph will introduce how these feaures work.  
 
 ## Template Features
 
-Template features are generated by templates. By given templates, according corpus, the features are generated automatically. The template feature is binary feature. If the feature exists in current token, its value will be 1, otherwise, the value will be 0. It's similar as CRFSharp features. In RNNSharp, TFeatureBin.exe is the console tool to generate this type of features.
+Template features are generated by templates. By given templates and corpus, these features can be automatically generated. In RNNSharp, template features are sparse features, so if the feature exists in current token, the feature value will be 1 (or feature frequency), otherwise, it will be 0. It's similar as CRFSharp features. In RNNSharp, TFeatureBin.exe is the console tool to generate this type of features.
 
 In template file, each line describes one template which consists of prefix, id and rule-string. The prefix indicates template type. So far, RNNSharp supports U-type feature, so the prefix is always as "U". Id is used to distinguish different templates. And rule-string is the feature body.
 
@@ -93,62 +93,88 @@ U15:Care/VBP
 
 Although U07 and U08, U11 and U12’s rule-string are the same, we can still distinguish them by id string.
 
-In feature configuration file, keyword TFEATURE_FILENAME is the file name of template feature set in binary format
-
 ## Context Template Features
 
 Context template features are based on template features and combined with context. In this example, if the context setting is "-1,0,1", the feature will combine the features of current token with its previous token and next token. For instance, if the sentence is "how are you". the generated feature set will be {Feature("how"), Feature("are"), Feature("you")}.
 
-In feature configuration file, keyword TFEATURE_CONTEXT is used to specify the tokens' context range for the feature.
-
-## Word Embedding Features
+## Pretrained Features
 
-Word embedding features are used to describe the features of given token. It's very useful when we only have small labeled corpus, but have lots of unlabeled corpus. This feature is generated by Txt2Vec project. With lots of unlabeled corpus, Txt2Vec is able to generate vectors for each token. Note that, the token's granularity between word embedding feature and RNN training corpus should be consistent, otherwise, tokens in training corpus are not able to be matched with the feature. For more detailed information about how to generate word embedding features, please visit Txt2Vec homepage.
+RNNSharp supports two types of pretrained features. The one is embedding features, and the other is auto-encoder features. Both of them are able to present a given token by a fixd-length vector. This feature is dense feature in RNNSharp.  
 
-In RNNSharp, this feature also supports context feature. It will combine all features of given contexts into a single word embedding feature.
+For embedding features, they are trained from unlabled corpus by Text2Vec project. And RNNSharp uses them as static features for each given token. However, for auto-encoder features, they are trained by RNNSharp as well, and then they can be used as dense features for other trainings. Note that, the token's granularity in pretrained features should be consistent with training corpus in main training, otherwise, some tokens will mis-match with pretrained feature.  
 
-In feature configuration, it has three keywords: WORDEMBEDDING_FILENAME is used to specify the encoded word embedding data file name generated by Txt2Vec. WORDEMBEDDING_CONTEXT is used to specify the token's context range. And WORDEMBEDDING_COLUMN is used to specify the column index applied the feature in corpus
+Likes template features, embedding feature also supports context feature. It can combine all features of given contexts into a single embedding feature. For auto-encoder features, it does not support it yet.
 
 ## Run Time Features
 
 Compared with other features generated offline, this feature is generated in run time. It uses the result of previous tokens as run time feature for current token. This feature is only available for forward-RNN, bi-directional RNN does not support it.
 
-In feature configuration, keyword RTFEATURE_CONTEXT is used to specify the context range of this feature.
+## Source Sequence Encoding Feature
+
+This feature is only for sequence-to-sequence task. In sequence-to-sequence task, RNNSharp encodes given source sequence into a fixed-length vector, and then pass it as dense feature to generate target sequence.  
+
+## Configuration File
+
+The configuration file describes model structure and features. In console tool, use -cfgfile as parameter to specify this file. Here is an example for sequence labeling task:  
+
+\#Working directory. It is the parent directory of below relatived paths.  
+CURRENT_DIRECTORY = .  
+
+\#Model type. Sequence labeling (SEQLABEL) and sequence-to-sequence (SEQ2SEQ) are supported.  
+MODEL_TYPE = SEQLABEL  
+
+\#Model direction. Forward and BiDirectional are supported  
+MODEL_DIRECTION = BiDirectional  
 
-## Feature Configuration File
+\#Model file path  
+MODEL_FILEPATH = Data\Models\ParseORG_CHS\model.bin  
 
-The configuration file has settings for different feature types introduced in above. Here is an example.. In console tool, use -ftrfile as parameter to specify this file.  
+\#Hidden layers settings. BPTT, LSTM, Dropout are supported. Here are examples of these layer types  
+\#BPTT: 200:BPTT:5 -- Layer size is 200, BPTT value is 5  
+\#Dropout: 200:Dropout:0.5 -- Layer size is 200, Drop out ratio is 0.5  
+\#If the model has more than one hidden layer, each layer settings are separated by comma. For example:  
+\#"300:LSTM, 200:LSTM" means the model has two LSTM layers. The first layer size is 300, and the second layer size is 200  
+HIDDEN_LAYER = 200:LSTM  
 
-\#The file name of template feature set  
-TFEATURE_FILENAME:tfeatures  
+\#Output layer settings. Softmax ands NCESoftmax are supported. Here is an example of NCESoftmax:  
+\#"NCESoftmax:20" means the output layer is NCESoftmax layer and its negative sample size is 20  
+OUTPUT_LAYER = Softmax  
 
-\#The context range of template feature set. In below example, the context is current token, next token and next after next token  
-TFEATURE_CONTEXT: 0,1,2  
+\#CRF layer settings  
+CRF_LAYER = True  
 
-\#Pretrain model type: Currently, it supports two types: 'Embedding' and 'Autoencoder'. The default type is 'Embedding'.  
+\#The file name for template feature set  
+TFEATURE_FILENAME = Data\Models\ParseORG_CHS\tfeatures  
+\#The context range for template feature set. In below, the context is current token, next token and next after next token  
+TFEATURE_CONTEXT = 0,1,2  
+\#The feature weight type. Binary and Freq are supported  
+TFEATURE_WEIGHT_TYPE = Binary  
+
+\#Pretrained features type: 'Embedding' and 'Autoencoder' are supported.  
 \#For 'Embedding', the pretrained model is trained by Text2Vec, which looks like word embedding model.  
 \#For 'Autoencoder', the pretrained model is trained by RNNSharp itself.  You need to train an auto encoder-decoder model by RNNSharp at first, and then use this pretrained model for your task.  
-PRETRAIN_TYPE:AUTOENCODER  
+PRETRAIN_TYPE = Embedding  
 
 \#The following settings are for pretrained model in 'Embedding' type.  
-\#The word embedding model generated by Txt2Vec. If embedding model is raw text format, we should use WORDEMBEDDING_RAW_FILENAME instead of WORDEMBEDDING_FILENAME as keyword  
-WORDEMBEDDING_FILENAME:word_vector.bin  
-
+\#The embedding model generated by Txt2Vec (https://github.com/zhongkaifu/Txt2Vec). If it is raw text format, we should use WORDEMBEDDING_RAW_FILENAME instead of WORDEMBEDDING_FILENAME as keyword  
+WORDEMBEDDING_FILENAME = Data\WordEmbedding\wordvec_chs.bin  
 \#The context range of word embedding. In below example, the context is current token, previous token and next token  
+\#If more than one token are combined, this feature would use a plenty of memory.  
 WORDEMBEDDING_CONTEXT: -1,0,1  
+\#The column index applied word embedding feature  
+WORDEMBEDDING_COLUMN = 0  
 
-\#The column index for word embedding feature  
-WORDEMBEDDING_COLUMN: 0  
-
-\#The following settings are for pretrained model in 'Autoencoder' type.  
-\#The auto encoder model generated by RNNSharp itself.  
-AUTOENCODER_MODEL: D:\RNNSharpDemoPackage\AutoEncoder\model.bin  
-
+\#The following setting is for pretrained model in 'Autoencoder' type.  
 \#The feature configuration file for pretrained model.  
-AUTOENCODER_FEATURECONFIG: D:\RNNSharpDemoPackage\features_autoencoder.txt  
+AUTOENCODER_CONFIG: D:\RNNSharpDemoPackage\config_autoencoder.txt  
 
-\#The context range of run time feature. In below exampl, RNNSharp will use the output of previous token as run time feature for current token  
-RTFEATURE_CONTEXT: -1
+\#The following setting is the configuration file for source sequence encoder which is only for sequence-to-sequence task that MODEL_TYPE equals to SEQ2SEQ.  
+\#In this example, since MODEL_TYPE is SEQLABEL, so we comment it out.  
+\#SEQ2SEQ_AUTOENCODER_CONFIG: D:\RNNSharpDemoPackage\config_seq2seq_autoencoder.txt  
+
+\#The context range of run time feature. In below example, RNNSharp will use the output of previous token as run time feature for current token  
+\#Note that, bi-directional model does not support run time feature, so we comment it out.  
+\#RTFEATURE_CONTEXT: -1
 
 ## Training file format
 
@@ -234,48 +260,31 @@ RNNSharpConsole.exe -mode train <parameters>
  Parameters for training RNN based model  
 -trainfile <string>: training corpus file  
 -validfile <string>: validated corpus for training  
--modelfile <string>: encoded model file  
--hiddenlayertype <string>: hidden layer type. BPTT and LSTM are supported, default is BPTT  
--outputlayertype <string>: output layer type. Softmax and NCESoftmax are supported, default is Softmax  
--ncesamplesize <int>: noise contrastive estimation(NCE) sample size, default is 15  
--ftrfile <string>: feature configuration file  
--tagfile <string>: supported output tagid-name list file  
+-cfgfile <string>: configuration file  
+-tagfile <string>: output tag or vocabulary file  
 -alpha <float>: learning rate, default is 0.1  
--dropout <float>: hidden layer node drop out ratio, default is 0  
--bptt <int>: the step for back-propagation through time, default is 4  
--layersize <int>: the size of each hidden layer, default is 200 for a single layer. If you want to have more than one layer, each layer size is split by character ',' For example: "-layersize = 200,100" means the neural network has two hidden layers, the first hidden layer size is 200, and the second hidden layer size is 100  
--crf <0/1>: training model by standard RNN(0) or RNN-CRF(1), default is 0  
 -maxiter <int>: maximum iteration for training. 0 is no limition, default is 20  
 -savestep <int>: save temporary model after every <int> sentence, default is 0  
--dir <int> : RNN directional: 0 - Forward RNN, 1 - Bi-directional RNN, default is 0  
 -vq <int> : Model vector quantization, 0 is disable, 1 is enable. default is 0  
--seq2seq <boolean> : Train a sequence-to-sequence model if it's true, otherwise, train a sequence labeling model. Default is false  
-
-Example for sequence labeling task: RNNSharpConsole.exe -mode train -trainfile train.txt -validfile valid.txt -modelfile model.bin -ftrfile features.txt -tagfile tags.txt -hiddenlayertype BPTT -outputlayertype softmax -layersize 200,100 -alpha 0.1 -crf 1 -maxiter 20 -savestep 200K -dir 1 -vq 0 -grad 15.0  
-  
-This command trains a bi-directional recurrent neural network with CRF output. The network has two BPTT hidden layers and one softmax output layer. The first hidden layer size is 200 and the second hidden layer size is 100  
-
-Example for sequence-to-sequence task: RNNSharpConsole.exe -mode train -trainfile train.txt -modelfile model.bin -ftrfile features_seq2seq.txt -tagfile tags.txt -hiddenlayertype lstm -outputlayertype ncesoftmax -ncesamplesize 20 -layersize 300 -alpha 0.1 -crf 0 -maxiter 0 -savestep 200K -dir 0 -dropout 0 -seq2seq true  
 
-This command trains a forward-directional sequence-to-sequence LSTM model, and the output layer is negative sampling softmax. The encoder is defined in [AUTOENCODER_XXX] section in features_seq2seq.txt file.  
+Example: RNNSharpConsole.exe -mode train -trainfile train.txt -validfile valid.txt -cfgfile config.txt -tagfile tags.txt -alpha 0.1 -maxiter 20 -savestep 200K -vq 0 -grad 15.0  
 
 ### Decode Model
 
-In this mode, the console tool is used to predict output tags of given corpus. The usage as follows:  
+In this mode, given test corpus file, RNNSharp predicts output tags in sequence labeling task or generates a target sequence in sequence-to-sequence task.  
 
 RNNSharpConsole.exe -mode test <parameters>  
  Parameters for predicting iTagId tag from given corpus  
--testfile <string>: training corpus file  
--modelfile <string>: encoded model file  
--tagfile <string>: supported output tagid-name list file  
--ftrfile <string>: feature configuration file  
+-testfile <string>: test corpus file  
+-tagfile <string>: output tag or vocabulary file  
+-cfgfile <string>: configuration file  
 -outfile <string>: result output file  
 
-Example: RNNSharpConsole.exe -mode test -testfile test.txt -modelfile model.bin -tagfile tags.txt -ftrfile features.txt -outfile result.txt    
+Example: RNNSharpConsole.exe -mode test -testfile test.txt -tagfile tags.txt -cfgfile config.txt -outfile result.txt    
 
 ## TFeatureBin
 
-It's used to generate template feature set by given template and corpus files. For high performance accessing and save memory cost, the indexed feature set is built as double array in trie-tree by AdvUtils. The tool supports three modes as follows:
+It's used to generate template feature set by given template and corpus files. For high performance accessing and save memory cost, the indexed feature set is built as float array in trie-tree by AdvUtils. The tool supports three modes as follows:
 
 TFeatureBin.exe <parameters>  
  The tool is to generate template feature from corpus and index them into file