Semantic chunking
Splitting a document at points where the topic actually shifts, instead of at a fixed character limit. It measures the similarity between consecutive sentences and sets a chunk boundary where similarity drops — so each chunk holds a coherent topic.
Mechanic: embed each sentence, compare neighboring sentences; when the similarity between two sentences falls below a threshold the topic has shifted → set a boundary there. The result is chunks of varying length that each cover one thing, giving cleaner embeddings and better matching than arbitrary splitting mid-argument. Trade-off: more expensive to prepare (embed everything first) and requires threshold tuning. Distinct from fixed-size chunking (simple but clumsy) and structure-based (by headings). A step toward smarter RAG indexing, often in concert with late chunking. Related to chunking and late chunking.