The keyword search component of Cloudflare AI Search has been updated to use the trigram tokenizer for Japanese, Chinese, and Korean content. This change enables natural‑language questions in these languages to return relevant keyword matches, whereas previously such queries were treated as exact phrases and often returned no results.
What Changed in the Tokenizer
Earlier configurations relied on the porter tokenizer, which does not handle CJK scripts effectively. The new default for CJK workloads is the trigram tokenizer, which breaks text into overlapping three‑character sequences, providing a broader matching surface for languages without explicit word boundaries.
How to Enable the New Behavior
For indexes that store Japanese, Chinese, or Korean documents, set the index configuration key keyword_tokenizer to trigram. After changing this setting, trigger a full reindex of the affected data set. The reindex operation is required because the tokenization strategy determines how terms are stored in the search index.
Operational Considerations
- Reindexing can be resource‑intensive; schedule it during low‑traffic windows to avoid performance degradation.
- Switching tokenizers may increase index size due to the higher number of generated trigrams.
- Search relevance for CJK queries will improve, but existing queries that relied on exact‑phrase matching may need to be reviewed.
Related CloudNinjas coverage: hands-on guides.
What This Means For Practitioners
AI, cloud, and DevOps teams should audit their AI Search indexes for CJK content and adjust the keyword_tokenizer accordingly. Plan a controlled reindex to minimize impact on service availability, and monitor index growth and query latency after the change. Security engineers should note that the reindex process touches stored data, so appropriate access controls and audit logging remain important during the operation.
