Control symbols (here it is [CLS] and [SEP] ) are tokenized which should not be

[weiczhu]

Loaded a trained SentencePiece model.
['▁', '[', 'C', 'L', 'S', ']', 'あ', 'と', '、', '梱包', 'に', '関', 'して', 'は', '、', '塩', 'が', '袋', 'から', '少し', '漏', 'れ', '出', 'て', 'いた', 'ので', '少し', '雑', 'な', '印', '象', 'を', '受', 'け', 'ました', '。', '[', 'S', 'E', 'P', ']']

We should expect the tokenizer should NEVER to break the control symbols

[taku910]

You might want to use user defined symbols instead of user defined symbols.

% spm_train --input=... --model_prefix=... --user_defined_symbols=[CLS],[SEP]

You can see #215 for more details.

[DomHudson]

Note: If you want to mimic BERT use

            --unk_piece=[UNK]
            --pad_piece=[PAD]
            --user_defined_symbols=[CLS],[SEP],[MASK]

[wppply]

Note: If you want to mimic BERT use

            --unk_piece=[UNK]
            --pad_piece=[PAD]
            --user_defined_symbols=[CLS],[SEP],[MASK]

In this case, the inital [CLS] will be tokenize to two word, will that affect the training of bert?

['▁', '[CLS]', '▁good', '▁good', '▁stud', 'y', ',', '▁day', 'day', '▁up', '[SEP]']