在看了阿里最有诚意的开源项目FunAudioLLM之后，一个念头就冒了出来：为什么不把它整合成一套语音输入、语音输出的工具呢？这样一来，J.A.R.V.I.S的基础部分——语音对话能力——就有了雏形。FunAudioLLM恰好提供了语音输入与语音输出两个模型，输出端甚至支持声音定制。 至于大脑，阿里另一个诚意满满的开源项目QWen2，同样是开源模型里的扛把子。于是，下面这个架构是不是看起来挺完整的？ 实际效果如何？这是一个完全基于Web的版本，当然，如果想改成其他类型的App，也不算难事。毕竟有了GPT一类的助手，做个Wrapper非常轻松。 给你听听最终生成的声音： ## 安装 整个安装过程没有想象中那么复杂，需要搭建三个基础模型的运行环境。 ### 第一个：CosyVoice 其实完全可以按照官方说明来安装配置，但国内网络环境有些地方需要特殊处理。 ```bash # 下面这几步居然需要魔法了，你敢信？？？？？程序员得罪谁了？ git clone --recursive https://github.com/FunAudioLLM/CosyVoice.git # 如果克隆子模块因网络失败，可反复执行以下命令直到成功 cd CosyVoice git submodule update --init --recursive ``` Conda的安装就不展开了，如果连Conda都不会用，大概率也不会看到这里。假设你用的是Ubuntu系统（我的是22.04）： ```bash conda create -n cosyvoice python=3.8 conda activate cosyvoice # pynini 被 WeTextProcessing 依赖，用conda安装可以跨平台 conda install -y -c conda-forge pynini==2.1.5 pip install -r requirements.txt -i https://mirrors.aliyun.com/pypi/simple/ --trusted-host=mirrors.aliyun.com # 如果遇到 sox 兼容问题 # Ubuntu sudo apt-get install sox libsox-dev ``` 环境装完后，下载模型。国内的ModelScope下载镜像真的很快。 可以在Python交互环境中执行以下代码： ```python from modelscope import snapshot_download snapshot_download('iic/CosyVoice-300M', local_dir='pretrained_models/CosyVoice-300M') snapshot_download('iic/CosyVoice-300M-SFT', local_dir='pretrained_models/CosyVoice-300M-SFT') snapshot_download('iic/CosyVoice-300M-Instruct', local_dir='pretrained_models/CosyVoice-300M-Instruct') snapshot_download('iic/CosyVoice-ttsfrd', local_dir='pretrained_models/CosyVoice-ttsfrd') ``` 测试一下： ```bash export PYTHONPATH=third_party/Matcha-TTS # 这一步很重要 ``` 然后将下面的代码存成Python文件直接测试： ```python from cosyvoice.cli.cosyvoice import CosyVoice from cosyvoice.utils.file_utils import load_wa v import torchaudio cosyvoice = CosyVoice('pretrained_models/CosyVoice-300M-SFT') # sft usage print(cosyvoice.list_a valiable_spks()) output = cosyvoice.inference_sft('你好，我是通义生成式语音大模型，请问有什么可以帮您的吗？', '中文女') torchaudio.sa ve('sft.wa v', output['tts_speech'], 22050) cosyvoice = CosyVoice('pretrained_models/CosyVoice-300M') # zero_shot usage, <|zh|><|en|><|jp|><|yue|><|ko|> 分别对应中/英/日/粤/韩 prompt_speech_16k = load_wa v('zero_shot_prompt.wa v', 16000) output = cosyvoice.inference_zero_shot('收到好友从远方寄来的生日礼物，那份意外的惊喜与深深的祝福让我心中充满了甜蜜的快乐，笑容如花儿般绽放。', '希望你以后能够做的比我还好呦。', prompt_speech_16k) torchaudio.sa ve('zero_shot.wa v', output['tts_speech'], 22050) # cross_lingual usage prompt_speech_16k = load_wa v('cross_lingual_prompt.wa v', 16000) output = cosyvoice.inference_cross_lingual('<|en|>And then later on, fully acquiring that company. So keeping management in line, interest in line with the asset that's coming into the family is a reason why sometimes we don't buy the whole thing.', prompt_speech_16k) torchaudio.sa ve('cross_lingual.wa v', output['tts_speech'], 22050) cosyvoice = CosyVoice('pretrained_models/CosyVoice-300M-Instruct') # instruct usage, support [laughter][breath] output = cosyvoice.inference_instruct('在面对挑战时，他展现了非凡的勇气与智慧。', '中文男', 'Theo \'Crimson\', is a fiery, passionate rebel leader. Fights with fervor for justice, but struggles with impulsiveness.') torchaudio.sa ve('instruct.wa v', output['tts_speech'], 22050) ``` 如果能顺利生成几个音频文件，就说明成功了一大半。 ### 第二个：SenseVoice 如果CosyVoice顺利安装，下面操作就简单了。 ```bash git clone https://github.com/FunAudioLLM/SenseVoice.git cd SenseVoice pip install -r requirements.txt # 想快一点可以加 -i https://pypi.tuna.tsinghua.edu.cn/simple ``` 然后执行测试代码： ```python from model import SenseVoiceSmall model_dir = "iic/SenseVoiceSmall" m, kwargs = SenseVoiceSmall.from_pretrained(model=model_dir) res = m.inference( data_in="https://isv-data.oss-cn-hangzhou.aliyuncs.com/ics/MaaS/ASR/test_audio/asr_example_zh.wa v", language="zh", # "zn", "en", "yue", "ja", "ko", "nospeech" use_itn=False, **kwargs, ) print(res) ``` 如果正常执行，输出类似： ``` ([{'key': 'wa v_file_tmp_name', 'text': '<|zh|><|NEUTRAL|><|Speech|><|woitn|>欢迎大家来体验打摩院推出的语音识别模型'}], {'load_data': '0.338', 'extract_feat': '0.020', 'batch_data_time': 5.58}) ``` ### 第三个：Ollama + Qwen2 STT和TTS都验证通过后，就该给系统装上大脑——Qwen2。为了服务的便捷和可替换性，选用了流行的LLM服务框架Ollama。 ```bash # Linux安装方法，前提是已安装好NVIDIA驱动 sudo curl -L https://ollama.com/download/ollama-linux-amd64 -o /usr/bin/ollama sudo chmod +x /usr/bin/ollama sudo useradd -r -s /bin/false -m -d /usr/share/ollama ollama ``` 编辑 /etc/systemd/system/ollama.service： ``` [Unit] Description=Ollama Service After=network-online.target [Service] ExecStart=/usr/bin/ollama serve User=ollama Group=ollama Restart=always RestartSec=3 [Install] WantedBy=default.target ``` 执行服务命令： ```bash sudo systemctl daemon-reload sudo systemctl enable ollama sudo systemctl start ollama ``` 最后运行特定模型，进入简单对话界面： ```bash ollama run qwen2:7b ``` 至此，所有基础部件都安装完成。 ## 接口调用 写一个简单的Python文件，将三个组件串起来： ```python import sys sys.path.append("/data/home/todo/SenseVoice") sys.path.append("/data/home/todo/CosyVoice") import requests from model import SenseVoiceSmall from cosyvoice.cli.cosyvoice import CosyVoice import torchaudio # Paths and URLs audio_file_path = "/data/home/todo/aliAllInOne/asr_example_zh.wa v" ollama_url = "http://localhost:11434/api/generate" cosyvoice_model_path = '/data/home/todo/CosyVoice/pretrained_models/CosyVoice-300M-SFT' generated_audio_path = "/data/home/todo/aliAllInOne/generated_audio.wa v" # Load the SenseVoiceSmall model model_dir = "iic/SenseVoiceSmall" model_dir = "/home/todo/.cache/modelscope/hub/iic/SenseVoiceSmall" m, kwargs = SenseVoiceSmall.from_pretrained(model=model_dir) def call_sense_voice_small(audio_path, language="zh"): res = m.inference( data_in=audio_path, language=language, use_itn=False, **kwargs, ) return res[0][0]['text'] def call_ollama_service(content): response = requests.post( ollama_url, json={ "model": "qwen2:7b", "prompt": content, "format": "json", "stream": False } ) if response.status_code == 200: return response.json()["response"] else: return "Error: Unable to get response from Ollama service." def generate_speech_with_cosyvoice(text, speaker="中文女"): cosyvoice = CosyVoice(cosyvoice_model_path) output = cosyvoice.inference_sft(text, speaker) torchaudio.sa ve(generated_audio_path, output['tts_speech'], 22050) def main(): sense_voice_result = call_sense_voice_small(audio_file_path) print("SenseVoiceSmall Result:", sense_voice_result) ollama_result = call_ollama_service(sense_voice_result) print("Ollama Service Result:", ollama_result) generate_speech_with_cosyvoice(ollama_result) print(f"Generated audio sa ved to {generated_audio_path}") if __name__ == "__main__": main() ``` 生成的语音是什么？下面是程序实际生成的： （此处应有音频链接或描述，原文中已提供） ## 界面 这部分交给你自己完成——毕竟基础框架已经搭好了。