Japanese LLMs: Start with the Corpus and the Evaluation

A Japanese-language assistant cannot be judged only by how well a general model performs in English. At Tokyo AI's 2024 NLP session, talks on Japanese model training returned repeatedly to the material beneath the model: the corpus, the tokenizer and the evaluation cases that represent real Japanese usage.

Corpus choices become product behavior

Kakeru Hattori described the work of building a Japanese web corpus for the Swallow project. A large collection of text is not automatically a useful training set. Duplicates, low-quality pages, uneven domains and licensing decisions shape what a model learns. The corpus recipe matters because a team must later explain why a model handles one type of Japanese document well and another poorly. Keeping records of sources and filtering decisions also makes a later rebuild possible.

Leonard Lin's account of training Shisa included tokenizer extension, continued pretraining, synthetic data and fine-tuning. These are not interchangeable fixes. A tokenizer affects how Japanese text is represented; corpus composition affects what the model sees; fine-tuning affects the behavior requested of it. Changing one layer may improve an example while harming another. Published training artifacts help other teams inspect and reproduce the work, but a buyer still needs to test against its own tasks.

Evaluate the language people actually use

Imagine an assistant that reads customer-support messages containing polite Japanese, abbreviated product names and occasional English terms. A pilot should test those combinations, including ambiguous requests and the expected refusal or escalation path. Reviewers should judge whether the answer preserved the customer's intent, cited the right record and used a tone suitable for the service. A benchmark score alone cannot tell the team whether that business process is safe to run.

For a production decision, separate the training-data question from the deployment question. Identify which Japanese text the system may process, define task-specific test cases, and re-evaluate after model or retrieval changes. The lesson from the session is practical: language quality is an engineering property of the whole pipeline, not a label attached to a model.

Tokyo AI is a community of builders, researchers and business leaders in Japan, run by Foundry Labs. See the event page.

Materials from the speakers’ talks are available upon request. Please use our contact form and include the event title and speaker name.

Speakers

• Leonard Lin, founder and CTO, Shisa.AI.
• Kakeru Hattori, researcher in the Swallow Japanese LLM project.

All articles