详解如何利用稀疏向量优化信息检索系统_AI热点日报

详解如何利用稀疏向量优化信息检索系统

类型：热点整理2026-06-18

信息检索领域这些年经历了翻天覆地的变化。从最初靠统计关键词匹配的朴素思路，一路发展到像BERT这样的深度学习模型，可以说是一步一个脚印。传统方法虽然扎实、经得起推敲，但在捕捉文本的语义关系上，总是差那么一口气。BERT这类稠密检索方法倒是聪明，它能利用高维向量吃透文本的上下文语义，给搜索技术带来质的

信息检索领域这些年经历了翻天覆地的变化。从最初靠统计关键词匹配的朴素思路，一路发展到像BERT这样的深度学习模型，可以说是一步一个脚印。传统方法虽然扎实、经得起推敲，但在捕捉文本的语义关系上，总是差那么一口气。BERT这类稠密检索方法倒是聪明，它能利用高维向量吃透文本的上下文语义，给搜索技术带来质的飞跃。但问题也随之而来——这些模型太依赖训练时见过的领域套路，一旦遇到没见过的新领域（out-of-domain），往往就会露怯。

好在，学习得到的稀疏Embedding（Learned Sparse Embedding）交出了一套独特的解决方案。它将稀疏表示的可解释性，和深度学习模型的语境理解能力拧在了一起。这篇文章就来聊聊学习得到的稀疏向量的底层机制、优势，以及在实际场景里能怎么用。特别是和Milvus向量数据库搭配起来之后，能如何帮助信息检索系统提升效率、给出更丰富的上下文答案，最终让系统性能再上一个台阶。

01.
信息检索方式演变：从关键词匹配到上下文理解

早期的信息检索系统，靠的基本都是基于统计的关键词匹配方法，比如TF-IDF和BM25这类词袋（Bag of Words）算法。它们的逻辑很简单：分析一个词在文档里出现了多少次、在整个文档库里分布如何，以此判断文档跟查询的相关性。这些方法虽然简单但足够实用，在互联网快速增长的那段时间里，成了相当流行的工具。

2013年，Google推出了Word2Vec——这是第一次尝试用高维向量来表达单词，并捕捉它们之间细微的语义差异。这一步标志着信息检索开始转向机器学习驱动。

再后来，BERT出现了——这个基于Transformer的革命性预训练语言模型，彻底改写了信息检索的玩法。BERT利用Transformer强大的注意力机制，把文档里复杂的上下文语义压缩成稠密向量，做到对查询既强大又语义精准的检索，从底层优化了整个信息检索流程。

BERT发布三个月后，Nogueira和Cho就把它用在了MS MARCO段落排名任务上。跟之前最强的IRNet方法相比，BERT的效果提升了30%，是BM25 baseline的2倍。到了2024年3月，文本排名的分数更是达到了0.450（eval）和0.463（dev）。

这些突破清楚地说明了一个趋势：信息检索方法一直在进化。从那以后，我们正式迈入了自然语言处理（NLP）的纪元。

02.
领域外信息检索挑战

稠密向量技术（比如BERT）相比传统词袋模型，有一个很明显的优势——能精确把握文本中的复杂语境。这个特性让信息检索系统在处理熟悉领域里的查询时，表现相当出色。

但换个场景就不一样了：当BERT进入没怎么见过的领域做Out-of-Domain（OOD）检索时，就会遇到麻烦。因为它的训练过程本质上偏向训练数据的特点，所以当面对OOD数据集中从未见过的文本片段时，很难生成有意义的embedding。尤其在那些含有大量领域特有术语的数据集里，这个限制会更加突出。

解决这个问题的一个思路是微调。但很多时候，这条路要么走不通，要么成本太高。微调需要一个中到大型的标注数据集——包括跟目标领域相关的查询、正面文档和负面文档。而且，建立这样的数据集还得靠领域专家来把控质量，耗时长、花费高。

有意思的是，另一边，传统词袋算法在处理OOD检索时，虽然也会遇到词不匹配的问题（查询和相关的文档之间往往缺少术语重叠），但在某些情况下性能反而比BERT好。原因很简单：词袋算法里，那些没见过的词不是被“学习”的，而是被“精确匹配”的。

03.
学习得到的稀疏向量：将传统稀疏向量与上下文信息相结合

既要保留词袋模型那种精确匹配的硬功夫，又要吸收BERT这类稠密向量检索方法在语义理解上的长处——把这两者结合起来，一直是信息检索领域的核心任务。好在，现在有了新的解决思路：学习得到的稀疏embedding。

那么，到底什么是学习得到的稀疏embedding向量呢？

简单说，它就是通过复杂的机器学习模型（比如SPLADE和BGE-M3）生成的稀疏向量表示。跟传统靠统计方法（如BM25）生成的稀疏向量不一样，学习得到的稀疏embedding在保留关键词搜索能力的同时，还为稀疏表示注入了丰富的上下文信息。它能识别出相邻或相关词语的重要性，哪怕这些词语在原文里没有明确出现。最终产出的，是一个擅长捕捉相关关键词和类别的“学习型”稀疏表示。

拿SPLADE来举例。在编码给定文本时，SPLADE生成的稀疏embedding是一条token-to-weight映射，像这样：

{"hello": 0.33, "world": 0.72}

乍一看，这些embedding跟传统统计方法生成的稀疏embedding差不多。但关键区别在于维度（词汇）和权重的来源：学习型稀疏embedding的维度和权重，是由带上下文信息的机器学习模型决定的。这种稀疏表示和学习得到的上下文的结合，为信息检索提供了强大的工具，无缝弥合了精确词匹配和语义理解之间的鸿沟。

和稠密向量不同，学习得到的稀疏embedding走的是简洁路线——只保留文本信息中的关键内容。这种简化有助于防止模型过度拟合训练数据，提高模型在新数据模式上的泛化能力，尤其在Out-Of-Domain检索场景中效果明显。

通过优先处理关键文本元素、同时舍弃不必要的细节，学习得到的稀疏embedding在捕获相关信息和防止过拟合之间找到了漂亮的平衡，从而增强了在各种检索任务中的适用价值。

学习得到的稀疏Embedding和稠密检索方法结合，在领域内检索任务中已经展现出强大的性能。来自BGE-M3的研究表明，学习得到的稀疏向量在多语言或跨语言检索任务中，效果比稠密向量更好——准确性更高。而当这两种embedding一起使用时，检索准确性又得到了进一步提升，说明它们确实是相辅相成的关系。

除此之外，这些embedding天生的稀疏性也让向量相似性搜索大大简化，减少了计算资源的消耗。还有一个亮点：学习得到的稀疏向量通过匹配增强上下文理解，可以快速检查匹配的文档，找出对应的匹配词。这个功能让检索过程更加透明和可控，给系统带来了更高的可用性。

04.
代码示例

下面来点实际的——在密集检索效果不理想的场景下，学习得到的稀疏向量是怎么表现的。

数据集：MIRACL。
查询：What years did Zhu Xi live?
备注：

朱熹是宋朝时期的中国书法家、历史学家、哲学家、诗人和整治家。
MIRACL数据集是多语言的，这个演示只用了英文部分的“训练”切分。它包含26746篇文章，其中七篇与朱熹相关。我们分别用密集和稀疏检索方法检索这七个相关故事。首先把所有故事编码成密集或稀疏embedding，存到Milvus向量数据库。然后对查询编码，用KNN搜索找出与查询embedding最接近的前10个故事。

Sparse Search Results(IP distance, larger is closer):
    Score: 26.221 Id: 244468#1 Text: Zhu Xi, whose family originated in Wuyuan County, Huizhou ...
    Score: 26.041 Id: 244468#3 Text: In 1208, eight years after his death, Emperor Ningzong of Song rehabilitated Zhu Xi and honored him ...
    Score: 25.772 Id: 244468#2 Text: In 1179, after not serving in an official capacity since 1156, Zhu Xi was appointed Prefect of Nankang Military District (南康軍) ...
    Score: 23.905 Id: 5823#39 Text: During the Song dynasty, the scholar Zhu Xi (AD 1130–1200) added ideas from Daoism and Buddhism into Confucianism ...
    Score: 23.639 Id: 337115#2 Text: ... During the Song Dynasty, the scholar Zhu Xi (AD 1130–1200) added ideas from Taoism and Buddhism into Confucianism ...
    Score: 23.061 Id: 10443808#22 Text: Zhu Xi was one of many critics who argued that ...
    Score: 22.577 Id: 55800148#11 Text: In February 1930, Zhu decided to live the life of a devoted revolutionary after lea ving his family at home ...
    Score: 20.779 Id: 12662060#3 Text: Holding to Confucius and Mencius' conception of humanity as innately good, Zhu Xi articulated an understanding of "li" as the basic pattern of the universe ...
    Score: 20.061 Id: 244468#28 Text: Tao Chung Yi (around 1329～1412) of the Ming dynasty: Whilst Master Zhu inherited the orthodox teaching and propagated it to the realm of sages ...
    Score: 19.991 Id: 13658250#10 Text: Zhu Shugui was 66 years old (by Chinese reckoning; 65 by Western standards) at the time of his suicide ...
Dense Search Results(L2 distance, smaller is closer):
    Score: 0.600 Id: 244468#1 Text: Zhu Xi, whose family originated in Wuyuan County, Huizhou ...
    Score: 0.706 Id: 244468#3 Text: In 1208, eight years after his death, Emperor Ningzong of Song rehabilitated Zhu Xi and honored him ...
    Score: 0.759 Id: 13658250#10 Text: Zhu Shugui was 66 years old (by Chinese reckoning; 65 by Western standards) at the time of his suicide ...
    Score: 0.804 Id: 6667852#0 Text: King Zhaoxiang of Qin (; 325–251 BC), or King Zhao of Qin (秦昭王), born Ying Ji ...
    Score: 0.818 Id: 259901#4 Text: According to the 3rd-century historical text "Records of the Three Kingdoms", Liu Bei was born in Zhuo County ...
    Score: 0.868 Id: 343207#0 Text: Ruzi Ying (; 5 – 25), also known as Emperor Ruzi of Han and the personal name of Liu Ying (劉嬰), was the last emperor ...
    Score: 0.876 Id: 31071887#1 Text: The founder of the Ming dynasty was the Hongwu Emperor (21 October 1328 – 24 June 1398), who is also known variably by his personal name "Zhu Yuanzhang" ...
    Score: 0.882 Id: 2945225#0 Text: Emperor Ai of Tang (27 October 89226 March 908), also known as Emperor Zhaoxuan (), born Li Zuo, later known as Li Chu ...
    Score: 0.890 Id: 33230345#0 Text: Li Xi or Li Qi (李谿 per the "Zizhi Tongjian" and the "History of the Five Dynasties" or 李磎 per the "Old Book of Tang" ...
    Score: 0.893 Id: 172890#1 Text: The Wusun originally lived between the Qilian Mountains and Dunhuang (Gansu) near the Yuezhi ...

仔细看看结果：在稀疏检索中，7个与朱熹相关的故事都排进了前10。而在稠密检索中，只有2个出现在前10。虽然两种方法都正确找到了编号244468#1和244468#3的段落，但稠密检索没能抓到其他相关故事——返回的另外8个结果都跟中国的其他历史故事有关，虽然模型觉得它们跟朱熹沾边，但实际上并不相关。

如果对背后的原理感兴趣，欢迎接着看下面这部分——怎么用Milvus做向量搜索。

05.
如何使用Milvus进行向量搜索

Milvus是一款高度可扩展、性能出色的开源向量数据库。在最新的2.4版本中，已经支持了稀疏和稠密向量（公测中）。下面就用Milvus 2.4来存储MIRACL数据集并执行向量搜索。

演示的目标：在MIRACL数据集上查询“朱熹生活在哪个年代？”。

我们使用SPLADE和MiniLM-L6-v2模型，将查询内容及MIRACL源数据集转化为稀疏和稠密向量。

首先，创建一个目录，配置环境与Milvus服务。确保系统中已安装python(>=3.8)、virtualenv、docker以及docker-compose。

mkdir milvus_sparse_demo && cd milvus_sparse_demo
# spin up a milvus local cluster
wget https://github.com/milvus-io/milvus/releases/download/v2.4.0-rc.1/milvus-standalone-docker-compose.yml -O docker-compose.yml
docker-compose up -d
# create a virtual environment
virtualenv -p python3.8 .venv && source .venv/bin/activate
touch milvus_sparse_demo.py

从2.4版本开始，pymilvus（Milvus的Python SDK）引入了一个可选的model模块。这个模块简化了用模型将文本编码成稀疏或稠密向量的流程。另外，用pip安装pymilvus model模块，以便访问HuggingFace上的数据集。

pip install "pymilvus[model]" datasets tqdm

首先，用HuggingFace的Datasets库下载数据集，收集所有段落：

from datasets import load_dataset
miracl = load_dataset('miracl/miracl', 'en')['train']
# collect all passages in the dataset
docs = list({doc['docid']: doc for entry in miracl for doc in entry['positive_passages'] + entry['negative_passages']}.values())
print(docs[0])
# {'docid': '2078963#10', 'text': 'The third thread in the development of quantum field theory...'}

将查询编码：

from pymilvus import model
sparse_ef = model.sparse.SpladeEmbeddingFunction(
    model_name="na ver/splade-cocondenser-selfdistil", 
    device="cpu",
)
dense_ef = model.dense.SentenceTransformerEmbeddingFunction(
    model_name='all-MiniLM-L6-v2',
    device='cpu',
)

query = "What years did Zhu Xi live?"

query_sparse_emb = sparse_ef([query])
query_dense_emb = dense_ef([query])

查看生成的稀疏向量并找到权重最高的Token：

sparse_token_weights = sorted([(sparse_ef.model.tokenizer.decode(col), query_sparse_emb[0, col]) 
                              for col in query_sparse_emb.indices[query_sparse_emb.indptr[0]:query_sparse_emb.indptr[1]]], key=lambda item: item[1], reverse=True)
print(sparse_token_weights[:7])
[('zhu', 3.0623178), ('xi', 2.4944391), ('zhang', 1.4790473), ('date', 1.4589322), ('years', 1.4154341), ('live', 1.3365831), ('china', 1.2351396)]

将所有文档编码：

from tqdm import tqdm
doc_sparse_embs = [sparse_ef([doc['text']]) for doc in tqdm(docs, desc='Encoding Sparse')]
doc_dense_embs = [dense_ef([doc['text']])[0] for doc in tqdm(docs, desc='Encoding Dense')]

接着，在Milvus中创建Collection以存储文档id、文本、稠密和稀疏向量等，然后插入数据：

from pymilvus import (
    MilvusClient, FieldSchema, CollectionSchema, DataType
)
milvus_client = MilvusClient("http://localhost:19530")
collection_name = 'miracl_demo'
fields = [
    FieldSchema(name="doc_id", dtype=DataType.VARCHAR, is_primary=True, auto_id=False, max_length=100),
    FieldSchema(name="text", dtype=DataType.VARCHAR, max_length=65530),
    FieldSchema(name="sparse_vector", dtype=DataType.SPARSE_FLOAT_VECTOR),
    FieldSchema(name="dense_vector", dtype=DataType.FLOAT_VECTOR, dim=384),
]
milvus_client.create_collection(collection_name, schema=CollectionSchema(fields, "miracl demo"), timeout=5, consistency_level="Strong")

batch_size = 30
n_docs = len(docs)
for i in tqdm(range(0, n_docs, batch_size), desc='Inserting documents'):
    milvus_client.insert(collection_name, [
        {
            "doc_id": docs[idx]['docid'],
            "text": docs[idx]['text'],
            "sparse_vector": doc_sparse_embs[idx],
            "dense_vector": doc_dense_embs[idx]
        }
        for idx in range(i, min(i+batch_size, n_docs))
    ])

为加速搜索，对向量字段创建索引：

index_params = milvus_client.prepare_index_params()
index_params.add_index(
    field_name="sparse_vector",
    index_type="SPARSE_INVERTED_INDEX",
    metric_type="IP",
)
index_params.add_index(
    field_name="dense_vector",
    index_type="FLAT",
    metric_type="L2",
)
milvus_client.create_index(collection_name, index_params=index_params)
milvus_client.load_collection(collection_name)

进行搜索并查看结果：

k = 10
sparse_results = milvus_client.search(collection_name, query_sparse_emb, anns_field="sparse_vector", limit=k, search_params={"metric_type": "IP"}, output_fields=['doc_id', 'text'])[0]
dense_results = milvus_client.search(collection_name, query_dense_emb, anns_field="dense_vector", limit=k, search_params={"metric_type": "L2"}, output_fields=['doc_id', 'text'])[0]

print(f'Sparse Search Results:')
for result in sparse_results:
    print(f"tScore: {result['distance']} Id: {result['entity']['doc_id']} Text: {result['entity']['text']}")

print(f'Dense Search Results:')
for result in dense_results:
    print(f"tScore: {result['distance']} Id: {result['entity']['doc_id']} Text: {result['entity']['text']}")

以下为输出。稠密向量搜索结果中，仅前两个结果为和朱熹有关的故事。示例中精简了文本以提升可读性。

Sparse Search Results:
    Score: 26.221 Id: 244468#1 Text: Zhu Xi, whose family originated in Wuyuan County, Huizhou ...
    Score: 26.041 Id: 244468#3 Text: In 1208, eight years after his death, Emperor Ningzong of Song rehabilitated Zhu Xi and honored him ...
    Score: 25.772 Id: 244468#2 Text: In 1179, after not serving in an official capacity since 1156, Zhu Xi was appointed Prefect of Nankang Military District (南康軍) ...
    Score: 23.905 Id: 5823#39 Text: During the Song dynasty, the scholar Zhu Xi (AD 1130–1200) added ideas from Daoism and Buddhism into Confucianism ...
    Score: 23.639 Id: 337115#2 Text: ... During the Song Dynasty, the scholar Zhu Xi (AD 1130–1200) added ideas from Taoism and Buddhism into Confucianism ...
    Score: 23.061 Id: 10443808#22 Text: Zhu Xi was one of many critics who argued that ...
    Score: 22.577 Id: 55800148#11 Text: In February 1930, Zhu decided to live the life of a devoted revolutionary after lea ving his family at home ...
    Score: 20.779 Id: 12662060#3 Text: Holding to Confucius and Mencius' conception of humanity as innately good, Zhu Xi articulated an understanding of "li" as the basic pattern of the universe ...
    Score: 20.061 Id: 244468#28 Text: Tao Chung Yi (around 1329～1412) of the Ming dynasty: Whilst Master Zhu inherited the orthodox teaching and propagated it to the realm of sages ...
    Score: 19.991 Id: 13658250#10 Text: Zhu Shugui was 66 years old (by Chinese reckoning; 65 by Western standards) at the time of his suicide ...
Dense Search Results:
    Score: 0.600 Id: 244468#1 Text: Zhu Xi, whose family originated in Wuyuan County, Huizhou ...
    Score: 0.706 Id: 244468#3 Text: In 1208, eight years after his death, Emperor Ningzong of Song rehabilitated Zhu Xi and honored him ...
    Score: 0.759 Id: 13658250#10 Text: Zhu Shugui was 66 years old (by Chinese reckoning; 65 by Western standards) at the time of his suicide ...
    Score: 0.804 Id: 6667852#0 Text: King Zhaoxiang of Qin (; 325–251 BC), or King Zhao of Qin (秦昭王), born Ying Ji ...
    Score: 0.818 Id: 259901#4 Text: According to the 3rd-century historical text "Records of the Three Kingdoms", Liu Bei was born in Zhuo County ...
    Score: 0.868 Id: 343207#0 Text: Ruzi Ying (; 5 – 25), also known as Emperor Ruzi of Han and the personal name of Liu Ying (劉嬰), was the last emperor ...
    Score: 0.876 Id: 31071887#1 Text: The founder of the Ming dynasty was the Hongwu Emperor (21 October 1328 – 24 June 1398), who is also known variably by his personal name "Zhu Yuanzhang" ...
    Score: 0.882 Id: 2945225#0 Text: Emperor Ai of Tang (27 October 89226 March 908), also known as Emperor Zhaoxuan (), born Li Zuo, later known as Li Chu ...
    Score: 0.890 Id: 33230345#0 Text: Li Xi or Li Qi (李谿 per the "Zizhi Tongjian" and the "History of the Five Dynasties" or 李磎 per the "Old Book of Tang" ...
    Score: 0.893 Id: 172890#1 Text: The Wusun originally lived between the Qilian Mountains and Dunhuang (Gansu) near the Yuezhi ...

至此，示例已完成。如果不再使用，可以通过以下代码删除相关内容：

docker-compose down
cd .. && rm -rf milvus_sparse_demo

06.
总结

这篇文章带大家走了一遍复杂的Embedding向量空间——从传统的稀疏向量检索和稠密向量检索，到创新型的学习得到的稀疏向量检索，信息检索方法正在持续演进。我们还详细介绍了两种机器学习模型——BGE-M3和SPLADE，以及它们是怎么生成学习得到的稀疏向量的。

利用这些先进的Embedding技术，搜索和检索系统的优化空间大大拓宽了，也为打造更直观、更快速响应的平台注入了更多可能性。

来源：https://www.53ai.com/news/RAG/2024072379623.html

ai 人工智能

延伸阅读

补充最近整理过的热点入口。