Tweak the handling of invalid text not to split UTF-8 characters apart
commit6f5579f121b29e37d9b999ea60442971d4ea3ddd
authorOwen W. Taylor <otaylor@fishsoup.net>
Sun, 11 Nov 2007 19:55:53 +0000 (11 14:55 -0500)
committerOwen W. Taylor <otaylor@fishsoup.net>
Sun, 11 Nov 2007 19:55:53 +0000 (11 14:55 -0500)
tree07cc99f99462bd2090a50a64e9e19492bb30b5cf
parent9f0fc3dd1373efe0afe511fd45826188c9de88d6
Tweak the handling of invalid text not to split UTF-8 characters apart
lib/reinteract/tokenize.py