recurrentlanguagemodel.lua with bidirectional option

[tsuyoshi-okita]

Hi nicholas,

recurrentlanguagemodel.lua works fine without bidirectional options. However, if I use the
bidirectional option, it yields error as follows. Any help will be welcome.

$ th recurrentlanguagemodel.lua --lstm --cuda --dataset PennTreeBank --bidirectional
{
accUpdate : false
batchSize : 64
bidirectional : true
cuda : true
cutoffNorm : -1
dataPath : "/home/tokita/data"
dataset : "PennTreeBank"
decayFactor : 0.001
dropout : false
dropoutProb : 0.5
evalSize : 100
forestGaterSize : "{}"
hiddenSize : {200}
learningRate : 0.1
lrDecay : "linear"
lstm : true
maxEpoch : 400
maxOutNorm : 2
maxTries : 30
maxWait : 4
minLR : 1e-05
momentum : 0
progress : false
rho : 5
saturateEpoch : 300
schedule : {}
silent : false
small : false
softmaxforest : false
softmaxtree : false
testFile : "test.txt"
tiny : false
trainEpochSize : 400000
trainFile : "train.txt"
trainOnly : false
uniform : -1
useDevice : 1
validEpochSize : 24000
validFile : "valid.txt"
xpPath : ""
zeroFirst : false
}
Warning : the Perplexity of a bidirectional RNN/LSTM isn't necessarily mathematically valid as it uses P(x_t|x_{/neq t}) instead of P(x_t|x_{<t}), which is used for unidirectional RNN/LSTMs. You can however still use predictions to measure pseudo-likelihood.
Language Model :
nn.Sequential {
[input -> (1) -> (2) -> (3) -> (4) -> output]
(1): nn.LookupTable
(2): nn.SplitTable
(3): nn.BiSequencerLM {
( fwd ): nn.Sequential {
| [input -> (1) -> output]
| (1): nn.Sequencer @ nn.FastLSTM
| }
( bwd ): nn.Sequential {
| [input -> (1) -> (2) -> (3) -> output]
| (1): nn.ReverseTable
| (2): nn.Sequential {
| [input -> (1) -> output]
| (1): nn.Sequencer @ nn.FastLSTM
| }
| (3): nn.ReverseTable
| }
( merge ): nn.Sequential { trainOnly : false
uniform : -1
useDevice : 1
validEpochSize : 24000
validFile : "valid.txt"
xpPath : ""
zeroFirst : false
}
Warning : the Perplexity of a bidirectional RNN/LSTM isn't necessarily mathematically val
id as it uses P(x_t|x_{/neq t}) instead of P(x_t|x_{<t}), which is used for unidirectiona
l RNN/LSTMs. You can however still use predictions to measure pseudo-likelihood.
Language Model :
nn.Sequential {
[input -> (1) -> (2) -> (3) -> (4) -> output]
(1): nn.LookupTable
(2): nn.SplitTable
(3): nn.BiSequencerLM {
( fwd ): nn.Sequential {
| [input -> (1) -> output]
| (1): nn.Sequencer @ nn.FastLSTM
| }
( bwd ): nn.Sequential {
| [input -> (1) -> (2) -> (3) -> output]
| (1): nn.ReverseTable
| (2): nn.Sequential {
| [input -> (1) -> output]
| (1): nn.Sequencer @ nn.FastLSTM
| }
| (3): nn.ReverseTable
| }

  |      [input -> (1) -> (2) -> output]
  |      (1): nn.ZipTable
  |      (2): nn.Sequencer @ nn.JoinTable
  |    }

}
(4): nn.Sequencer @ nn.Sequential {
input -> (1) -> (2) -> output: nn.Linear(400 -> 10000)
(2): nn.LogSoftMax
}
}
FileLogger: log will be written to /home/tokita/save/zett-laptop:1443950935:1/log
==> epoch # 1 for optimizer :
==> example speed = 8530.3733795048 examples/s
/home/tokita/torch/install/bin/luajit: ...a/torch/install/share/lua/5.1/dp/sampler/textsampler.lua:38: bad argument #3 to 'narrow' (out of range at /tmp/luarocks_torch-scm-1-9343/torch7/lib/TH/generic/THTensor.c:351)
stack traceback:
[C]: in function 'narrow'
...a/torch/install/share/lua/5.1/dp/sampler/textsampler.lua:38: in function 'sampler'
...torch/install/share/lua/5.1/dp/propagator/propagator.lua:117: in function 'propagateEpoch'
...torch/install/share/lua/5.1/dp/propagator/experiment.lua:116: in function 'run'
recurrentlanguagemodel.lua:360: in main chunk
[C]: in function 'dofile'
...kita/torch/install/lib/luarocks/rocks/trepl/scm-1/bin/th:131: in main chunk
[C]: at 0x00405f60

[nicholas-leonard]

Yeah sorry about that. The --bidirectional option is kind of experimental. It was working at one point, but I must of changed something in the TextSet to break it...