remove-non-printing-char.perl « tokenizer « scripts - github.com/moses-smt/mosesdecoder.git - Unnamed repository; edit this file 'description' to name the repository.

Welcome to mirror list, hosted at ThFree Co, Russian Federation.

remove-non-printing-char.perl « tokenizer « scripts - github.com/moses-smt/mosesdecoder.git - Unnamed repository; edit this file 'description' to name the repository.

index : github.com/moses-smt/mosesdecoder.git

Unnamed repository; edit this file 'description' to name the repository.

www-data

summary refs log tree commit diff

path: root/scripts/tokenizer/remove-non-printing-char.perl

blob: 4dadd1d77d5fd85af4ae45086668458d0fa2d43e (plain)

#!/usr/bin/env perl 

use utf8; 

binmode(STDIN, ":utf8");
binmode(STDOUT, ":utf8");
binmode(STDERR, ":utf8");

while (my $line = <STDIN>) {
  chomp($line);
  #$line =~ tr/\040-\176/ /c;
  #$line =~ s/[^[:print:]]/ /g;
  #$line =~ s/\s+/ /g; 
  $line =~ s/\p{C}/ /g; 

  print "$line\n";
}