Tokenizes translation units on a tmx file.
tmx-tokenize file.tmx # creates t_file.tmx tmx-tokenize -o=out.tmx file.tmx
Although this script is bundled in \*(C`XML::TMX\*(C', it has a soft dependency on \*(C`Lingua::FreeLing3\*(C'. Soft means that the dependency is not ensured at install time, and other features of the module can still be used without \*(C`Lingua::FreeLing3\*(C'. Nevertheless, if you want to use this tool you should install that module.
At the moment the supported languages are the same as supported by FreeLing3: English, Spanish, Russian, Portuguese and Italian.
It your \s-1TMX\s0 file includes any other language, they will be maintained without a change. This behavior can change in the future, as a basic regexp based tokenizer might be implemented.
\s-1XML::TMX\s0, Lingua::FreeLing3
Alberto Manuel Branda\*~o Simo\*~es, <[email protected]>
Copyright (C) 2012 by Alberto Manuel Branda\*~o Simo\*~es