AlloyDB powers enterprise-grade search for some of the largest organizations, providing robust hybrid search capabilities that combine text, vector, and keyword searches into a simple ranked SQL query. And with our recent launch of RUM index support, AlloyDB customers now have even more powerful full-text search capabilities at their fingertips.
However, database developers often face limitations when indexing continuous text in logographical languages like Chinese, Japanese, and Korean, where traditional whitespace-based tokenization fails. Gemini’s multilingual capabilities enable you to intelligently parse text in these languages to implement intelligent word segmentation and stop-word removal, but orchestrating row-wise API calls on massive datasets is slow and fragile. Now you can integrate the world knowledge of Gemini models natively into your database using AlloyDB AI Functions, enabling highly accurate full-text search for logographical languages without the administrative overhead of complex ETL pipelines.
Continuous text and logographical languages
To understand why this native integration is such a significant advancement, we must first examine the underlying mechanics of text search and why traditional indexing methods fail when processing continuous text.
To build an effective full-text search index in PostgreSQL, the engine must parse text into search tokens (lexemes) using the to_tsvector function. By default, standard text search configurations like simple or english assume that words are separated by whitespace. The database engine extracts search terms by splitting the input string at these spaces.
However, Chinese and other logographical languages do not use spaces between words. Words are written continuously, with punctuation serving as the only boundaries. Because of this, standard PostgreSQL parsers fail to extract individual keywords. Instead, they treat entire sentences or long clauses as a single, continuous lexeme.
For example, consider this input string: “你们研究所有十个图书馆” (Your research institute has ten libraries)
Without spaces, passing this to to_tsvector(‘simple’, …) produces a single, massive lexeme: ‘你们研究所有十个图书馆’
If you search for “研究所” (research institute) or “图书馆” (library), the query fails to return a match. The keywords are trapped inside the larger string, forcing you to search for the exact, long-form sentence to get a result.
Traditional workarounds and their limits
You might try to resolve this using traditional tools and pipelines, each of which introduces significant operational friction or accuracy limits:
-
Third-party database extensions: Extensions like zhparser or pg_jieba add Chinese tokenization to PostgreSQL. However, these are often not supported in fully-managed database environments. They also rely on static dictionaries, which frequently fail to parse modern jargon, brand names, or context-dependent terms correctly.
-
External preprocessing pipelines: Exporting text to an external application (such as a Python microservice running jieba or spaCy) to insert spaces before saving it to the database. This pattern introduces substantial ETL complexity, network latency, and data exposure risks by moving your data out of the database tier.
-
Inadequacy of rule-based and dictionary-driven segmentation: Traditional tokenizers rely on static dictionaries and hand-coded syntactic rules to split text. They often struggle to resolve semantic ambiguity, where the exact same sequence of characters must be segmented differently depending on the context. To perform accurate segmentation, you need the world knowledge and contextual intelligence of a large language model like Gemini.
In-database pre-processing with Gemini
AlloyDB AI bypasses these workarounds — and their limits — by introducing native, in-database AI Functions like ai.generate(). This allows you to call Gemini directly from SQL, keeping your data and intelligence in one place.
This approach provides three core advantages:
-
No data movement: All text preprocessing and segmentation happen directly within the database engine. This minimizes network latency and keeps your data protected within your database boundaries.
-
In-database intelligence: You do not need to build, deploy, or maintain external microservices or orchestration frameworks. The database engine coordinates the model calls natively.
-
Stored procedure-based batching: By using a PL/pgSQL stored procedure with array aggregation, you can process rows in parallel batches, unpack the results safely using GENERATE_SERIES, and commit each batch immediately. This prevents database memory exhaustion, bypasses row lock contention, and supports stable, performant execution even when handling massive tables.
Implementing semantic word segmentation
To implement this solution, we will create a database table where the raw content, the segmented text, the search vector, and the vector embeddings are stored together.
