SentencePiece
Google's 2018 tokenizer, which treats text as a raw character stream including the spaces, and therefore works the same on Japanese as on English.
Earlier tokenizers assumed the text had first been split into words on whitespace, a Western habit that does not hold for Japanese, Chinese or Thai. SentencePiece skips that step entirely: spaces are encoded as an ordinary character, ▁, which also makes decoding lossless — glue the pieces back together and you get exactly the original. The tool implements both BPE and a unigram language model, where the vocabulary is trimmed down from a deliberately oversized seed set by measuring which pieces contribute least. T5, ALBERT, XLNet and Llama all use it, making it one of the most widely deployed components in modern language modelling.