大模型语音交互的技术是： STT（Speech To Text）语音转文字，TTS（Text To Speech）文字转语音
你微信里那个"按住说话，松开发送"、手机输入法的语音输入，全是 STT。
导航里"前方 500 米左转"、抖音的"系统配音"、你说的小爱同学，全是 TTS。
你开口说话
│ 🎤 声波
↓
【<span>STT</span>】速记员：把声音敲成字
│ <span>"一分钟后提醒我喝水"</span>
↓
【大模型】书呆子：读字、思考、调工具
│ <span>"好的，已设置提醒"</span>
↓
【<span>TTS</span>】配音员：把字念成声音
│ 🔊 音频
↓
你听到回答
一.获取apikey
进入腾讯云语音技术：console.cloud.tencent.com/asr/resourc…
领取语音合成资源包和语音识别资源包
然后进入访问管理，获取SecretId和SecretKey
拿到 SecretId 和 SecretKey 以后,我们使用网站的ai助手就能够直接拿到使用案例。如图所示：
二.创建项目
nest <span>new</span> test-stt
npm install tencentcloud-sdk-nodejs --save
nest g resource voice --no-spec
2.1 固定文本合成语音-TTS
就是TTS（Text To Speech）文字转语音，这个例子的特殊之处在于，文本是固定的，有了固定的字符串，然后才开始读。
实现voice.service.ts文件
直接照抄官网案例代码即可,你要是想修改音色，直接进入cloud.tencent.com/document/pr…然后将音色 ID改到下面即可
<span>import</span> { <span>Injectable</span> } <span>from</span> <span>'@nestjs/common'</span>;
<span>import</span> * <span>as</span> tencentcloud <span>from</span> <span>'tencentcloud-sdk-nodejs'</span>;
<span>// 语音合成 TTS 客户端（API 版本 v20190823）</span>
<span>const</span> <span>TtsClient</span> = tencentcloud.<span>tts</span>.<span>v20190823</span>.<span>Client</span>;
@<span>Injectable</span>()
<span>export</span> <span>class</span> <span>VoiceService</span> {
private ttsClient;
<span>constructor</span>(<span></span>) {
<span>const</span> credential = {
<span>secretId</span>: <span>'AKIDJSN3V4d9jWLDulWLB6g7d3TuPvQzPbXJ'</span>,
<span>secretKey</span>: <span>'f5HPL5cuwkQ2TjDgv2xeMvGdmhPtKcgz'</span>,
};
<span>const</span> region = <span>'ap-guangzhou'</span>;
<span>this</span>.<span>ttsClient</span> = <span>new</span> <span>TtsClient</span>({
credential,
region,
<span>profile</span>: { <span>httpProfile</span>: { <span>endpoint</span>: <span>'tts.tencentcloudapi.com'</span> } },
});
}
<span>// ===== 语音合成 TTS：文字转语音 =====</span>
<span>async</span> <span>textToVoice</span>(<span>text: string</span>) {
<span>const</span> params = {
<span>Text</span>: text, <span>// 待合成文本</span>
<span>SessionId</span>: <span>`tts_<span>${<span>Date</span>.now()}</span>`</span>, <span>// 会话 ID，每次请求唯一</span>
<span>ModelType</span>: <span>1</span>, <span>// 模型类型</span>
<span>VoiceType</span>: <span>101001</span>, <span>// 音色 ID，具体取值以官方文档为准</span>
<span>Codec</span>: <span>'mp3'</span>, <span>// 输出格式：mp3 / wav / pcm</span>
};
<span>// 返回 Audio 字段为 Base64 编码的音频数据</span>
<span>const</span> res = <span>await</span> <span>this</span>.<span>ttsClient</span>.<span>TextToVoice</span>(params);
<span>return</span> res.<span>Audio</span>;
}
}
实现voice.module.ts文件
你要记得exports: [VoiceService]
<span>import</span> { <span>Module</span> } <span>from</span> <span>'@nestjs/common'</span>;
<span>import</span> { <span>VoiceService</span> } <span>from</span> <span>'./voice.service.js'</span>;
<span>import</span> { <span>VoiceController</span> } <span>from</span> <span>'./voice.controller.js'</span>;
@<span>Module</span>({
<span>providers</span>: [<span>VoiceService</span>],
<span>controllers</span>: [<span>VoiceController</span>],
})
<span>export</span> <span>class</span> <span>VoiceModule</span> {}
实现voice.controller.ts文件
import * as fs from 'node:fs';
import * as fs from 'node:fs/promises';
他们俩生成的fs是有区别的，首先他们的方法都查不到，但是node:fs生成的fs是同步的，
node:fs/promises是异步的。在我们的案例里面，两个都能用。
await fs.mkdir(outDir, { recursive: true }); 里面{ recursive: true }表示如果有outDir就不重新创建，如果没有就创建。
<span>import</span> { <span>Body</span>, <span>Controller</span>, <span>Get</span>, <span>Post</span>, <span>Query</span> } <span>from</span> <span>'@nestjs/common'</span>;
<span>import</span> { <span>VoiceService</span> } <span>from</span> <span>'./voice.service.js'</span>;
<span>import</span> { <span>Response</span> } <span>from</span> <span>'express'</span>;
<span>import</span> * <span>as</span> fs <span>from</span> <span>'node:fs/promises'</span>;
<span>import</span> * <span>as</span> path <span>from</span> <span>'node:path'</span>;
@<span>Controller</span>(<span>'voice'</span>)
<span>export</span> <span>class</span> <span>VoiceController</span> {
<span>constructor</span>(<span>private readonly voiceService: VoiceService</span>) {}
@<span>Get</span>(<span>'tts'</span>)
<span>async</span> <span>tts</span>(<span>@Query(<span>'text'</span>) text: string</span>) {
<span>const</span> audioBase64 = <span>await</span> <span>this</span>.<span>voiceService</span>.<span>textToVoice</span>(text);
<span>//audioBase64是string</span>
<span>if</span> (!audioBase64) {
<span>return</span> <span>''</span>;
}
<span>const</span> audioBuffer = <span>Buffer</span>.<span>from</span>(audioBase64, <span>'base64'</span>);
<span>const</span> outDir = path.<span>resolve</span>(<span>'./audio'</span>); <span>// 文件夹路径</span>
<span>const</span> outPath = path.<span>join</span>(outDir, <span>'tts.mp3'</span>); <span>// 文件完整路径</span>
<span>await</span> fs.<span>mkdir</span>(outDir, { <span>recursive</span>: <span>true</span> }); <span>// ← 就加这一行！ recursive: true ,没有audio就建，又就不管了</span>
<span>await</span> fs.<span>writeFile</span>(outPath, audioBuffer);
<span>await</span> fs.<span>writeFile</span>(outPath, audioBuffer);
<span>return</span> <span>'已保存成功！'</span>;
}
}
测试
测试代码: npm run start:dev
点开MP3播放即可。
2.2 流式文本合成语音-TTS
他也是是TTS（Text To Speech）文字转语音，这个例子的特殊之处在于，文本是流式的，那语音也是流式的。你生成多少文本，就读多少文本。
上面那个语音合成是生成好文本，再读出来。
流式文本语音是边生产边读。
为了测试流式语音方便，我把他的接口做成了get请求的。
实现tts-stream.controller.ts
下面信息是腾讯云获取的。
- appId: Number(this.configService.get('TENCENT_APP_ID')),
- secretId: this.configService.get('TENCENTCLOUD_SECRET_ID'),
- secretKey: this.configService.get('TENCENTCLOUD_SECRET_KEY'),
上一个案例，我们将获取到的音频写成文件存起来的代码写到了controller里面，这个案例我将这个代码放到了service里面。
<span>import</span> { <span>Controller</span>, <span>Get</span>, <span>Res</span> } <span>from</span> <span>'@nestjs/common'</span>;
<span>import</span> * <span>as</span> path <span>from</span> <span>'path'</span>;
<span>import</span> { <span>TtsService</span> } <span>from</span> <span>'./tts-stream.service.js'</span>;
<span>import</span> { <span>ConfigService</span> } <span>from</span> <span>'@nestjs/config'</span>;
@<span>Controller</span>(<span>'tts'</span>)
<span>export</span> <span>class</span> <span>TtsController</span> {
<span>constructor</span>(<span>
private readonly ttsService: TtsService,
private readonly configService: ConfigService,
</span>) {}
@<span>Get</span>(<span>'synthesize'</span>)
<span>async</span> <span>synthesize</span>(<span>@Res() res: any</span>) {
<span>// ===== 参数定死 =====</span>
<span>const</span> texts = [
<span>'你好，欢迎使用流式语音合成。'</span>,
<span>'我是你的语音小助手，你有任何需要都可以给我说'</span>,
]; <span>// 必须以 。！？； 结尾</span>
<span>const</span> <span>codec</span>: <span>'pcm'</span> | <span>'mp3'</span> = <span>'mp3'</span>;
<span>const</span> filename = <span>'demo'</span>;
<span>const</span> options = {
<span>appId</span>: <span>Number</span>(<span>this</span>.<span>configService</span>.<span>get</span>(<span>'TENCENT_APP_ID'</span>)),
<span>secretId</span>: <span>this</span>.<span>configService</span>.<span>get</span>(<span>'TENCENTCLOUD_SECRET_ID'</span>),
<span>secretKey</span>: <span>this</span>.<span>configService</span>.<span>get</span>(<span>'TENCENTCLOUD_SECRET_KEY'</span>),
codec,
};
<span>const</span> ext = codec === <span>'mp3'</span> ? <span>'mp3'</span> : <span>'wav'</span>;
<span>const</span> outputPath = path.<span>join</span>(process.<span>cwd</span>(), <span>'output'</span>, <span>`<span>${filename}</span>.<span>${ext}</span>`</span>);
res.<span>setHeader</span>(<span>'Content-Type'</span>, codec === <span>'mp3'</span> ? <span>'audio/mpeg'</span> : <span>'audio/pcm'</span>);
res.<span>setHeader</span>(
<span>'Content-Disposition'</span>,
<span>`inline; filename="<span>${filename}</span>.<span>${ext}</span>"`</span>,
);
res.<span>setHeader</span>(<span>'Cache-Control'</span>, <span>'no-store'</span>); <span>// 防止浏览器缓存，改代码后能立刻生效</span>
<span>try</span> {
<span>await</span> <span>this</span>.<span>ttsService</span>.<span>synthesize</span>(
options,
texts,
{
<span>onAudio</span>: <span>(<span>chunk: any</span>) =></span> res.<span>write</span>(chunk),
},
outputPath,
);
res.<span>end</span>();
} <span>catch</span> (<span>err</span>: any) {
<span>if</span> (!res.<span>headersSent</span>) res.<span>status</span>(<span>500</span>).<span>end</span>(err.<span>message</span>);
<span>else</span> res.<span>destroy</span>();
}
}
}
实现tts-stream.module.ts
<span>import</span> { <span>Module</span> } <span>from</span> <span>'@nestjs/common'</span>;
<span>import</span> { <span>TtsController</span> } <span>from</span> <span>'./tts-stream.controller.js'</span>;
<span>import</span> { <span>TtsService</span> } <span>from</span> <span>'./tts-stream.service.js'</span>;
@<span>Module</span>({
<span>controllers</span>: [<span>TtsController</span>],
<span>providers</span>: [<span>TtsService</span>],
})
<span>export</span> <span>class</span> <span>TtsModule</span> {}
实现tts-stream.service.ts
<span>import</span> { <span>Injectable</span>, <span>Logger</span> } <span>from</span> <span>'@nestjs/common'</span>;
<span>import</span> * <span>as</span> fs <span>from</span> <span>'node:fs'</span>;
<span>import</span> * <span>as</span> path <span>from</span> <span>'node:path'</span>;
<span>import</span> <span>WebSocket</span> <span>from</span> <span>'ws'</span>;
<span>import</span> { buildWsUrl, <span>TtsOptions</span> } <span>from</span> <span>'./tts-signature.js'</span>;
<span>import</span> * <span>as</span> crypto <span>from</span> <span>'crypto'</span>;
<span>export</span> interface <span>TtsCallbacks</span> {
onAudio?: <span>(<span>chunk: Buffer</span>) =></span> <span>void</span>; <span>// 收到音频 binary 帧，可边收边播</span>
onEvent?: <span>(<span>event: any</span>) =></span> <span>void</span>; <span>// READY / FINAL / 心跳 / 时间戳</span>
onReady?: <span>() =></span> <span>void</span>;
onFinal?: <span>() =></span> <span>void</span>;
onError?: <span>(<span>err: <span>Error</span></span>) =></span> <span>void</span>;
onClose?: <span>() =></span> <span>void</span>;
}
<span>/** 生成 44 字节 WAV 文件头（PCM 16bit 单声道） */</span>
<span>function</span> <span>buildWavHeader</span>(<span>
dataSize: number,
sampleRate = <span>16000</span>,
channels = <span>1</span>,
bitsPerSample = <span>16</span>,
</span>): <span>Buffer</span> {
<span>const</span> buf = <span>Buffer</span>.<span>alloc</span>(<span>44</span>);
buf.<span>write</span>(<span>'RIFF'</span>, <span>0</span>);
buf.<span>writeUInt32LE</span>(<span>36</span> + dataSize, <span>4</span>);
buf.<span>write</span>(<span>'WAVE'</span>, <span>8</span>);
buf.<span>write</span>(<span>'fmt '</span>, <span>12</span>);
buf.<span>writeUInt32LE</span>(<span>16</span>, <span>16</span>); <span>// fmt 块长度</span>
buf.<span>writeUInt16LE</span>(<span>1</span>, <span>20</span>); <span>// 1 = PCM</span>
buf.<span>writeUInt16LE</span>(channels, <span>22</span>);
buf.<span>writeUInt32LE</span>(sampleRate, <span>24</span>);
buf.<span>writeUInt32LE</span>((sampleRate * channels * bitsPerSample) / <span>8</span>, <span>28</span>); <span>// 字节率</span>
buf.<span>writeUInt16LE</span>((channels * bitsPerSample) / <span>8</span>, <span>32</span>); <span>// 块对齐</span>
buf.<span>writeUInt16LE</span>(bitsPerSample, <span>34</span>);
buf.<span>write</span>(<span>'data'</span>, <span>36</span>);
buf.<span>writeUInt32LE</span>(dataSize, <span>40</span>);
<span>return</span> buf;
}
@<span>Injectable</span>()
<span>export</span> <span>class</span> <span>TtsService</span> {
private readonly logger = <span>new</span> <span>Logger</span>(<span>TtsService</span>.<span>name</span>);
<span>/**
* 一次性合成：传入文本数组，自动逐句发送并收齐音频
* <span>@param</span> outputPath 可选。传了就同时把音频写到该文件（pcm→.wav，mp3→.mp3）
*/</span>
<span>async</span> <span>synthesize</span>(
<span>options</span>: <span>TtsOptions</span>,
<span>texts</span>: string[],
<span>callbacks</span>: <span>TtsCallbacks</span> = {},
outputPath?: string,
): <span>Promise</span><<span>void</span>> {
<span>const</span> url = <span>buildWsUrl</span>(options);
<span>const</span> ws = <span>new</span> <span>WebSocket</span>(url, { <span>perMessageDeflate</span>: <span>false</span> });
<span>// ===== 写文件：准备 =====</span>
<span>const</span> sampleRate = options.<span>sampleRate</span> ?? <span>16000</span>;
<span>const</span> isPcm = (options.<span>codec</span> ?? <span>'pcm'</span>) === <span>'pcm'</span>;
<span>let</span> <span>fileStream</span>: fs.<span>WriteStream</span> | <span>null</span> = <span>null</span>;
<span>let</span> bytes = <span>0</span>;
<span>if</span> (outputPath) {
fs.<span>mkdirSync</span>(path.<span>dirname</span>(path.<span>resolve</span>(outputPath)), { <span>recursive</span>: <span>true</span> });
fileStream = fs.<span>createWriteStream</span>(outputPath);
<span>if</span> (isPcm) fileStream.<span>write</span>(<span>buildWavHeader</span>(<span>0</span>, sampleRate)); <span>// 先占位，收完回填</span>
}
<span>/** 音频帧统一出口：回调 + 落盘 */</span>
<span>const</span> <span>handleAudio</span> = (<span>chunk: Buffer</span>) => {
callbacks.<span>onAudio</span>?.(chunk);
<span>if</span> (fileStream) {
bytes += chunk.<span>length</span>;
fileStream.<span>write</span>(chunk);
}
};
<span>/** 收尾：补 WAV 头、关文件流 */</span>
<span>const</span> <span>finalizeFile</span> = <span>async</span> (<span></span>) => {
<span>if</span> (!fileStream) <span>return</span>;
<span>const</span> stream = fileStream;
fileStream = <span>null</span>;
<span>await</span> <span>new</span> <span>Promise</span><<span>void</span>>(<span>(<span>resolve, reject</span>) =></span> {
stream.<span>end</span>(<span>(<span>err?: <span>Error</span> | <span>null</span></span>) =></span> (err ? <span>reject</span>(err) : <span>resolve</span>()));
});
<span>if</span> (isPcm) {
<span>// 回填真实长度，否则播放器会认为文件损坏</span>
<span>const</span> fh = <span>await</span> fs.<span>promises</span>.<span>open</span>(outputPath!, <span>'r+'</span>);
<span>await</span> fh.<span>write</span>(<span>buildWavHeader</span>(bytes, sampleRate), <span>0</span>, <span>44</span>, <span>0</span>);
<span>await</span> fh.<span>close</span>();
}
<span>this</span>.<span>logger</span>.<span>log</span>(
<span>`音频已写入 <span>${path.resolve(outputPath!)}</span>（<span>${bytes}</span> 字节）`</span>,
);
};
<span>return</span> <span>new</span> <span>Promise</span><<span>void</span>>(<span>(<span>resolve, reject</span>) =></span> {
<span>let</span> <span>error</span>: <span>Error</span> | <span>null</span> = <span>null</span>;
<span>let</span> settled = <span>false</span>;
<span>const</span> timer = <span>setTimeout</span>(<span>() =></span> {
<span>this</span>.<span>logger</span>.<span>warn</span>(<span>'30秒没收到结束信号，强制断开'</span>);
ws.<span>close</span>();
}, <span>30000</span>);
<span>const</span> <span>done</span> = <span>async</span> (<span></span>) => {
<span>clearTimeout</span>(timer);
<span>if</span> (settled) <span>return</span>;
settled = <span>true</span>;
<span>try</span> {
<span>await</span> <span>finalizeFile</span>();
} <span>catch</span> (e) {
error = error ?? (e <span>as</span> <span>Error</span>);
}
callbacks.<span>onClose</span>?.();
<span>if</span> (error) <span>reject</span>(error);
<span>else</span> <span>resolve</span>();
};
ws.<span>on</span>(<span>'open'</span>, <span>() =></span> <span>this</span>.<span>logger</span>.<span>log</span>(<span>'WebSocket 已连接，等待 READY'</span>));
ws.<span>on</span>(<span>'message'</span>, <span>(<span>data: any, isBinary?: boolean</span>) =></span> {
<span>// 1. 统一归一化成 Buffer（兼容 Buffer / Buffer[] / ArrayBuffer / string）</span>
<span>let</span> <span>buf</span>: <span>Buffer</span>;
<span>if</span> (<span>Buffer</span>.<span>isBuffer</span>(data)) buf = data;
<span>else</span> <span>if</span> (<span>Array</span>.<span>isArray</span>(data)) buf = <span>Buffer</span>.<span>concat</span>(data);
<span>else</span> <span>if</span> (data <span>instanceof</span> <span>ArrayBuffer</span>) buf = <span>Buffer</span>.<span>from</span>(data);
<span>else</span> buf = <span>Buffer</span>.<span>from</span>(<span>String</span>(data), <span>'utf8'</span>);
<span>// 2. 用内容判断：JSON 事件帧一定以 `{"` 开头</span>
<span>const</span> looksJson = buf[<span>0</span>] === <span>0x7b</span> && buf[<span>1</span>] === <span>0x22</span>;
<span>const</span> isTextFrame = isBinary === <span>false</span> || looksJson;
<span>if</span> (isTextFrame) {
<span>let</span> <span>event</span>: any;
<span>try</span> {
event = <span>JSON</span>.<span>parse</span>(buf.<span>toString</span>(<span>'utf8'</span>));
} <span>catch</span> {
<span>return</span>;
}
<span>this</span>.<span>logger</span>.<span>debug</span>(<span>`事件帧: <span>${buf.toString(<span>'utf8'</span>)}</span>`</span>); <span>// 关键：看清服务端到底回什么</span>
callbacks.<span>onEvent</span>?.(event);
<span>if</span> (event.<span>code</span> && event.<span>code</span> !== <span>0</span>) {
error = <span>new</span> <span>Error</span>(<span>`TTS 错误 <span>${event.code}</span>: <span>${event.message}</span>`</span>);
callbacks.<span>onError</span>?.(error);
ws.<span>close</span>();
<span>return</span>;
}
<span>if</span> (event.<span>ready</span> === <span>1</span>) {
<span>this</span>.<span>logger</span>.<span>log</span>(<span>'收到 READY，开始发送文本'</span>);
callbacks.<span>onReady</span>?.();
<span>const</span> sessionId = event.<span>session_id</span>;
texts.<span>forEach</span>(<span>(<span>text</span>) =></span> {
ws.<span>send</span>(
<span>JSON</span>.<span>stringify</span>({
<span>session_id</span>: sessionId,
<span>message_id</span>: crypto.<span>randomUUID</span>(),
<span>action</span>: <span>'ACTION_SYNTHESIS'</span>,
<span>data</span>: text,
}),
);
});
ws.<span>send</span>(
<span>JSON</span>.<span>stringify</span>({
<span>session_id</span>: sessionId,
<span>message_id</span>: crypto.<span>randomUUID</span>(),
<span>action</span>: <span>'ACTION_COMPLETE'</span>,
<span>data</span>: <span>''</span>,
}),
);
}
<span>if</span> (event.<span>final</span> === <span>1</span>) {
callbacks.<span>onFinal</span>?.();
ws.<span>close</span>();
}
<span>return</span>;
}
<span>// 3. 到这里一定是音频帧</span>
<span>handleAudio</span>(buf);
});
ws.<span>on</span>(<span>'error'</span>, <span>(<span>err</span>) =></span> {
<span>console</span>.<span>error</span>(<span>'[WS错误]'</span>, err); <span>// ← 加这行</span>
error = err;
callbacks.<span>onError</span>?.(err);
});
ws.<span>on</span>(<span>'close'</span>, <span>() =></span> {
<span>void</span> <span>done</span>();
});
});
}
}
实现tts-signature.ts
<span>import</span> * <span>as</span> crypto <span>from</span> <span>'crypto'</span>;
<span>const</span> <span>HOST</span> = <span>'tts.cloud.tencent.com/stream_wsv2'</span>;
<span>export</span> interface <span>TtsOptions</span> {
<span>appId</span>: number;
<span>secretId</span>: string;
<span>secretKey</span>: string;
voiceType?: number; <span>// 音色 ID，默认 101001</span>
volume?: number; <span>// 音量 [-10, 10]</span>
speed?: number; <span>// 语速 [-2, 6]</span>
sampleRate?: number; <span>// 8000 / 16000 / 24000</span>
codec?: <span>'pcm'</span> | <span>'mp3'</span>;
enableSubtitle?: boolean;
emotionCategory?: string;
emotionIntensity?: number;
segmentRate?: number;
}
<span>/** 签名：GET + 域名地址 + 请求参数，HMAC-SHA1 + base64 + urlencode */</span>
<span>export</span> <span>function</span> <span>generateSignature</span>(<span>
params: Record<string, string | number>,
secretKey: string,
</span>): string {
<span>const</span> query = <span>Object</span>.<span>keys</span>(params)
.<span>sort</span>()
.<span>map</span>(<span>(<span>k</span>) =></span> <span>`<span>${k}</span>=<span>${params[k]}</span>`</span>)
.<span>join</span>(<span>'&'</span>);
<span>const</span> source = <span>`GET<span>${HOST}</span>?<span>${query}</span>`</span>;
<span>const</span> sig = crypto
.<span>createHmac</span>(<span>'sha1'</span>, secretKey)
.<span>update</span>(source)
.<span>digest</span>(<span>'base64'</span>);
<span>return</span> <span>encodeURIComponent</span>(sig);
}
<span>/** 构建握手阶段 wss URL */</span>
<span>export</span> <span>function</span> <span>buildWsUrl</span>(<span>options: TtsOptions</span>): string {
<span>const</span> timestamp = <span>Math</span>.<span>floor</span>(<span>Date</span>.<span>now</span>() / <span>1000</span>);
<span>const</span> <span>params</span>: <span>Record</span><string, string | number> = {
<span>Action</span>: <span>'TextToStreamAudioWSv2'</span>,
<span>AppId</span>: options.<span>appId</span>,
<span>SecretId</span>: options.<span>secretId</span>,
<span>Timestamp</span>: timestamp,
<span>Expired</span>: timestamp + <span>3600</span>, <span>// 有效期 1 小时，差须小于 90 天</span>
<span>SessionId</span>: crypto.<span>randomUUID</span>().<span>replace</span>(<span>/-/g</span>, <span>''</span>),
<span>VoiceType</span>: options.<span>voiceType</span> ?? <span>101001</span>,
<span>Volume</span>: options.<span>volume</span> ?? <span>0</span>,
<span>Speed</span>: options.<span>speed</span> ?? <span>0</span>,
<span>SampleRate</span>: options.<span>sampleRate</span> ?? <span>16000</span>,
<span>Codec</span>: options.<span>codec</span> ?? <span>'pcm'</span>,
<span>EnableSubtitle</span>: options.<span>enableSubtitle</span> ? <span>'True'</span> : <span>'False'</span>,
};
<span>if</span> (options.<span>emotionCategory</span>) params.<span>EmotionCategory</span> = options.<span>emotionCategory</span>;
<span>if</span> (options.<span>emotionIntensity</span> !== <span>undefined</span>)
params.<span>EmotionIntensity</span> = options.<span>emotionIntensity</span>;
<span>if</span> (options.<span>segmentRate</span> !== <span>undefined</span>)
params.<span>SegmentRate</span> = options.<span>segmentRate</span>;
<span>const</span> signature = <span>generateSignature</span>(params, options.<span>secretKey</span>);
<span>const</span> query = <span>Object</span>.<span>keys</span>(params)
.<span>sort</span>()
.<span>map</span>(
<span>(<span>k</span>) =></span>
<span>`<span>${<span>encodeURIComponent</span>(k)}</span>=<span>${<span>encodeURIComponent</span>(<span>String</span>(params[k]))}</span>`</span>,
)
.<span>join</span>(<span>'&'</span>);
<span>console</span>.<span>log</span>(<span>`wss://<span>${HOST}</span>?<span>${query}</span>&Signature=<span>${signature}</span>`</span>);
<span>return</span> <span>`wss://<span>${HOST}</span>?<span>${query}</span>&Signature=<span>${signature}</span>`</span>;
}
测试
流式语音的好处就是，在controller里面，你可以加入很多条信息，加多少，输出多少。
2.3 语音转化成文本-ASR/STT
语音识别 ASR（Automatic Speech Recognition），叫 STT （Speech To Text） 也可以，但 ASR 用的多一些。
他的api还是在tencentcloud-sdk-nodejs里面。
pnpm install tencentcloud-sdk-nodejs
实现 asr.controller.ts
<span>import</span> { <span>AsrService</span> } <span>from</span> <span>'./asr.service.js'</span>;
<span>import</span> { <span>Controller</span>, <span>Get</span> } <span>from</span> <span>'@nestjs/common'</span>;
@<span>Controller</span>(<span>'asr'</span>)
<span>export</span> <span>class</span> <span>AsrController</span> {
<span>constructor</span>(<span>private readonly asrService: AsrService</span>) {}
@<span>Get</span>(<span>'asr'</span>)
<span>async</span> <span>asr</span>(<span></span>) {
<span>const</span> data = <span>await</span> <span>this</span>.<span>asrService</span>.<span>voiceToText</span>();
<span>return</span> data;
}
}
实现asr.module.ts
<span>import</span> { <span>AsrService</span> } <span>from</span> <span>'./asr.service.js'</span>;
<span>import</span> { <span>Controller</span>, <span>Get</span> } <span>from</span> <span>'@nestjs/common'</span>;
@<span>Controller</span>(<span>'asr'</span>)
<span>export</span> <span>class</span> <span>AsrController</span> {
<span>constructor</span>(<span>private readonly asrService: AsrService</span>) {}
@<span>Get</span>(<span>'asr'</span>)
<span>async</span> <span>asr</span>(<span></span>) {
<span>const</span> data = <span>await</span> <span>this</span>.<span>asrService</span>.<span>voiceToText</span>();
<span>return</span> data;
}
}
实现asr.service.ts
<span>import</span> { <span>Injectable</span> } <span>from</span> <span>'@nestjs/common'</span>;
<span>import</span> * <span>as</span> tencentcloud <span>from</span> <span>'tencentcloud-sdk-nodejs'</span>;
<span>import</span> { <span>ConfigService</span> } <span>from</span> <span>'@nestjs/config'</span>;
<span>import</span> * <span>as</span> fs <span>from</span> <span>'node:fs'</span>;
<span>import</span> * <span>as</span> path <span>from</span> <span>'node:path'</span>;
<span>const</span> <span>AsrClient</span> = tencentcloud.<span>asr</span>.<span>v20190614</span>.<span>Client</span>;
<span>const</span> <span>AUDIO_FILE</span> = path.<span>join</span>(process.<span>cwd</span>(), <span>'src'</span>, <span>'asr'</span>, <span>'demo.mp3'</span>);
@<span>Injectable</span>()
<span>export</span> <span>class</span> <span>AsrService</span> {
private client;
<span>constructor</span>(<span>private readonly configService: ConfigService</span>) {
<span>const</span> credential = {
<span>secretId</span>: <span>this</span>.<span>configService</span>.<span>get</span>(<span>'TENCENTCLOUD_SECRET_ID'</span>),
<span>secretKey</span>: <span>this</span>.<span>configService</span>.<span>get</span>(<span>'TENCENTCLOUD_SECRET_KEY'</span>),
};
<span>const</span> region = <span>'ap-guangzhou'</span>;
<span>this</span>.<span>client</span> = <span>new</span> <span>AsrClient</span>({
credential,
region,
<span>profile</span>: {
<span>httpProfile</span>: { <span>reqMethod</span>: <span>'POST'</span>, <span>reqTimeout</span>: <span>30</span> },
},
});
}
<span>async</span> <span>voiceToText</span>(<span></span>) {
<span>const</span> base64 = fs.<span>readFileSync</span>(<span>AUDIO_FILE</span>, <span>'base64'</span>);
<span>const</span> params = {
<span>EngSerViceType</span>: <span>'16k_zh'</span>,
<span>SourceType</span>: <span>1</span>,
<span>Data</span>: base64,
<span>DataLen</span>: <span>Buffer</span>.<span>byteLength</span>(base64),
<span>VoiceFormat</span>: <span>'mp3'</span>,
};
<span>try</span> {
<span>const</span> data = <span>await</span> <span>this</span>.<span>client</span>.<span>SentenceRecognition</span>(params);
<span>console</span>.<span>log</span>(<span>'识别结果：'</span>, data.<span>Result</span>);
<span>return</span> data.<span>Result</span>;
} <span>catch</span> (e) {
<span>console</span>.<span>log</span>(e);
}
}
}
测试
他会将上个案例里面生成的MP3文件里面的内容转化成text文件显示在页面上。
2.4 实现豆包语音模式回答问题
先生成一个简单的ai模块
nest g resource ai --no-spec
pnpm install @langchain/openai @langchain/core
实现 ai.service.ts
<span>import</span> { <span>Injectable</span>, <span>Inject</span> } <span>from</span> <span>'@nestjs/common'</span>;
<span>import</span> { <span>ChatOpenAI</span> } <span>from</span> <span>'@langchain/openai'</span>;
<span>import</span> { <span>PromptTemplate</span> } <span>from</span> <span>'@langchain/core/prompts'</span>;
<span>import</span> type { <span>Runnable</span> } <span>from</span> <span>'@langchain/core/runnables'</span>;
<span>import</span> { <span>StringOutputParser</span> } <span>from</span> <span>'@langchain/core/output_parsers'</span>;
@<span>Injectable</span>()
<span>export</span> <span>class</span> <span>AiService</span> {
private readonly <span>chain</span>: <span>Runnable</span>;
<span>constructor</span>(<span>@Inject(<span>'CHAT_MODEL'</span>) model: ChatOpenAI</span>) {
<span>const</span> prompt = <span>PromptTemplate</span>.<span>fromTemplate</span>(<span>`请回答以下问题：{query}`</span>);
<span>this</span>.<span>chain</span> = prompt.<span>pipe</span>(model).<span>pipe</span>(<span>new</span> <span>StringOutputParser</span>());
}
<span>async</span> *<span>streamChain</span>(<span>query</span>: string): <span>AsyncGenerator</span><string> {
<span>const</span> <span>stream</span>: any = <span>await</span> <span>this</span>.<span>chain</span>.<span>stream</span>({ <span>query</span>: query });
<span>for</span> <span>await</span> (<span>const</span> chunk <span>of</span> stream) {
<span>yield</span> chunk;
}
}
}
实现 ai.controller.ts
<span>import</span> { <span>Controller</span>, <span>Get</span>, <span>Query</span>, <span>Sse</span> } <span>from</span> <span>'@nestjs/common'</span>;
<span>import</span> { <span>from</span>, map, <span>Observable</span> } <span>from</span> <span>'rxjs'</span>;
<span>import</span> { <span>AiService</span> } <span>from</span> <span>'./ai.service.js'</span>;
@<span>Controller</span>(<span>'ai'</span>)
<span>export</span> <span>class</span> <span>AiController</span> {
<span>constructor</span>(<span>private readonly aiService: AiService</span>) {}
@<span>Sse</span>(<span>'chat/stream'</span>)
<span>chatStream</span>(@<span>Query</span>(<span>'query'</span>) <span>query</span>: string): <span>Observable</span><{ <span>data</span>: string }> {
<span>return</span> <span>from</span>(<span>this</span>.<span>aiService</span>.<span>streamChain</span>(query)).<span>pipe</span>(
<span>map</span>(<span>(<span>chunk</span>) =></span> ({ <span>data</span>: chunk })),
);
}
}
实现 ai.module.ts
<span>import</span> { <span>Module</span> } <span>from</span> <span>'@nestjs/common'</span>;
<span>import</span> { <span>AiService</span> } <span>from</span> <span>'./ai.service.js'</span>;
<span>import</span> { <span>AiController</span> } <span>from</span> <span>'./ai.controller.js'</span>;
<span>import</span> { <span>ChatOpenAI</span> } <span>from</span> <span>'@langchain/openai'</span>;
<span>import</span> { <span>ConfigService</span> } <span>from</span> <span>'@nestjs/config'</span>;
@<span>Module</span>({
<span>controllers</span>: [<span>AiController</span>],
<span>providers</span>: [
<span>AiService</span>,
{
<span>provide</span>: <span>'CHAT_MODEL'</span>,
<span>useFactory</span>: <span>async</span> (<span>configService</span>: <span>ConfigService</span>) => {
<span>const</span> model = <span>new</span> <span>ChatOpenAI</span>({
<span>model</span>: configService.<span>get</span>(<span>'MODEL_NAME'</span>),
<span>apiKey</span>: configService.<span>get</span>(<span>'OPENAI_API_KEY'</span>),
<span>configuration</span>: {
<span>baseURL</span>: configService.<span>get</span>(<span>'OPENAI_BASE_URL'</span>),
},
});
<span>return</span> model;
},
<span>inject</span>: [<span>ConfigService</span>],
},
],
})
<span>export</span> <span>class</span> <span>AiModule</span> {}
测试
启动项目，访问接口。
2.5 实现前端录音功能
写html文件
录音这块用的是浏览器原生的 MediaRecorder API，整个流程就四步：拿权限 → 边录边切块 → 拼成 Blob → 上传。不需要第三方库。
当然你可以使用第三方库：xiangyuecn/Recorder
npm i recorder-core
具体使用看官方文档，此处不再赘述。我的原生代码如下：
<!doctype html>
<span><span><<span>html</span> <span>lang</span>=<span>"zh-CN"</span>></span>
<span><<span>head</span>></span>
<span><<span>meta</span> <span>charset</span>=<span>"utf-8"</span> /></span>
<span><<span>meta</span> <span>name</span>=<span>"viewport"</span> <span>content</span>=<span>"width=device-width, initial-scale=1"</span> /></span>
<span><<span>title</span>></span>ASR 录音上传测试<span></<span>title</span>></span>
<span><<span>style</span>></span><span>
<span>body</span> {
<span>font-family</span>:
-apple-system, BlinkMacSystemFont, <span>'Segoe UI'</span>, <span>'Microsoft YaHei'</span>,
sans-serif;
<span>max-width</span>: <span>720px</span>;
<span>margin</span>: <span>40px</span> auto;
<span>padding</span>: <span>0</span> <span>16px</span>;
<span>line-height</span>: <span>1.6</span>;
}
<span>button</span> {
<span>margin-right</span>: <span>8px</span>;
<span>padding</span>: <span>8px</span> <span>16px</span>;
<span>font-size</span>: <span>14px</span>;
<span>cursor</span>: pointer;
}
<span>button</span><span>:disabled</span> {
<span>cursor</span>: not-allowed;
<span>opacity</span>: <span>0.5</span>;
}
<span>.status</span> {
<span>color</span>: <span>#666</span>;
}
pre {
<span>background</span>: <span>#f6f8fa</span>;
<span>border-radius</span>: <span>6px</span>;
<span>padding</span>: <span>12px</span>;
<span>white-space</span>: pre-wrap;
<span>word-break</span>: break-all;
}
</span><span></<span>style</span>></span>
<span></<span>head</span>></span>
<span><<span>body</span>></span>
<span><<span>h1</span>></span>ASR 录音上传测试<span></<span>h1</span>></span>
<span><<span>p</span>></span>
点击「开始录音」并允许麦克风权限，点击「停止录音」后将录音上传到后端进行语音识别。
<span></<span>p</span>></span>
<span><<span>button</span> <span>id</span>=<span>"startBtn"</span>></span>开始录音<span></<span>button</span>></span>
<span><<span>button</span> <span>id</span>=<span>"stopBtn"</span> <span>disabled</span>></span>停止录音<span></<span>button</span>></span>
<span><<span>p</span> <span>class</span>=<span>"status"</span> <span>id</span>=<span>"status"</span>></span>状态：空闲<span></<span>p</span>></span>
<span><<span>h3</span>></span>识别结果<span></<span>h3</span>></span>
<span><<span>pre</span> <span>id</span>=<span>"result"</span>></span>（暂无）<span></<span>pre</span>></span>
<span><<span>script</span>></span><span>
<span>const</span> startBtn = <span>document</span>.<span>getElementById</span>(<span>'startBtn'</span>);
<span>const</span> stopBtn = <span>document</span>.<span>getElementById</span>(<span>'stopBtn'</span>);
<span>const</span> statusEl = <span>document</span>.<span>getElementById</span>(<span>'status'</span>);
<span>const</span> resultEl = <span>document</span>.<span>getElementById</span>(<span>'result'</span>);
<span>let</span> mediaRecorder = <span>null</span>;
<span>let</span> audioChunks = [];
<span>let</span> mediaStream = <span>null</span>;
<span>const</span> fileName = <span>'record.ogg'</span>;
<span>function</span> <span>setStatus</span>(<span>text</span>) {
statusEl.<span>textContent</span> = <span>'状态：'</span> + text;
}
<span>async</span> <span>function</span> <span>startRecord</span>(<span></span>) {
<span>try</span> {
audioChunks = [];
<span>setStatus</span>(<span>'正在请求麦克风权限...'</span>);
mediaStream = <span>await</span> navigator.<span>mediaDevices</span>.<span>getUserMedia</span>({
<span>audio</span>: <span>true</span>,
});
mediaRecorder = <span>new</span> <span>MediaRecorder</span>(mediaStream);
mediaRecorder.<span>ondataavailable</span> = <span>(<span>event</span>) =></span> {
<span>if</span> (event.<span>data</span> && event.<span>data</span>.<span>size</span> > <span>0</span>) {
audioChunks.<span>push</span>(event.<span>data</span>);
}
};
mediaRecorder.<span>onstop</span> = <span>async</span> () => {
<span>setStatus</span>(<span>'录音结束，正在上传...'</span>);
<span>const</span> blob = <span>new</span> <span>Blob</span>(audioChunks, {
<span>type</span>: mediaRecorder.<span>mimeType</span> || <span>'audio/ogg'</span>,
});
audioChunks = [];
<span>await</span> <span>uploadAudio</span>(blob);
};
mediaRecorder.<span>start</span>(<span>250</span>);
<span>setStatus</span>(<span>'录音中...'</span>);
startBtn.<span>disabled</span> = <span>true</span>;
stopBtn.<span>disabled</span> = <span>false</span>;
} <span>catch</span> (err) {
<span>setStatus</span>(<span>'录音失败：'</span> + err.<span>message</span>);
<span>stopRecordStream</span>();
}
}
<span>function</span> <span>stopRecord</span>(<span></span>) {
<span>if</span> (mediaRecorder && mediaRecorder.<span>state</span> !== <span>'inactive'</span>) {
mediaRecorder.<span>stop</span>();
}
startBtn.<span>disabled</span> = <span>false</span>;
stopBtn.<span>disabled</span> = <span>true</span>;
}
<span>function</span> <span>stopRecordStream</span>(<span></span>) {
<span>if</span> (mediaStream) {
mediaStream.<span>getTracks</span>().<span>forEach</span>(<span>(<span>track</span>) =></span> track.<span>stop</span>());
mediaStream = <span>null</span>;
}
startBtn.<span>disabled</span> = <span>false</span>;
stopBtn.<span>disabled</span> = <span>true</span>;
}
<span>async</span> <span>function</span> <span>uploadAudio</span>(<span>blob</span>) {
<span>try</span> {
<span>const</span> formData = <span>new</span> <span>FormData</span>();
formData.<span>append</span>(<span>'audio'</span>, blob, fileName);
<span>const</span> response = <span>await</span> <span>fetch</span>(<span>'/asr/asr'</span>, {
<span>method</span>: <span>'POST'</span>,
<span>body</span>: formData,
});
<span>if</span> (!response.<span>ok</span>) {
<span>throw</span> <span>new</span> <span>Error</span>(<span>'上传失败：HTTP '</span> + response.<span>status</span>);
}
<span>const</span> data = <span>await</span> response.<span>json</span>();
<span>console</span>.<span>log</span>(data, <span>6777</span>);
resultEl.<span>textContent</span> = data.<span>data</span> || <span>'（空结果）'</span>;
<span>setStatus</span>(<span>'识别完成'</span>);
} <span>catch</span> (err) {
<span>setStatus</span>(<span>'识别失败：'</span> + err.<span>message</span>);
resultEl.<span>textContent</span> = <span>'错误：'</span> + err.<span>message</span>;
} <span>finally</span> {
<span>stopRecordStream</span>();
}
}
startBtn.<span>addEventListener</span>(<span>'click'</span>, startRecord);
stopBtn.<span>addEventListener</span>(<span>'click'</span>, stopRecord);
</span><span></<span>script</span>></span>
<span></<span>body</span>></span>
<span></<span>html</span>></span></span>
html访问接口
启动html后，有多种方式连接调用接口。
1.用live-server启动，启动以后（npx live-server public\asr.html ），地址变成了：http://127.0.0.1:8080/
此时，接口地址是：http://localhost:3000/ai/chat/stream?query=你说毛主席是谁
你会发现前后端url的端口号不一样，这样前端访问后端的时候就会有跨域的报错，解决办法：在nest项目的mian.ts里面添加 app.enableCors()。
而且你访问的接口地址要写全，不然会找不到。
2.利用@nestjs/serve-static，他的好处就是：
- 前后端用一个端口号，都是3000，不用再解决跨域问题。
- 而且访问接口的时候，我们写相对路径就好了，不用再带域名和端口号。
- nest项目启动以后，直接访问http://localhost:3000/index.html 就能进入前端页面，不需要在用live-server
我们使用@nestjs/serve-static
安装包
pnpm install @nestjs/serve-<span>static</span>
在app.module.ts里面注册一下
启动项目访问页面
npm run <span>start</span>:dev
连接asr
之前的asr读取MP3文件是写死的，现在我们有了前端代码，要他来读取录音文件。
我们修改一下asr.controller.ts,因为从前端录音过来的文件有可能是乱七八糟的东西，所以要用FileInterceptor处理下。
FileInterceptor 就是文件上传的"拆包工" 。
@UseInterceptors(FileInterceptor('audio')) 里面的audio和前端的 formData.append('audio', blob, fileName);的第一个参数保持一致，才能处理这个文件。
现在修改asr接口，之前的asr是get接口，现在需要改成post接口，修改地方如下：
注释掉的就是老代码
测试
启动项目
npm run <span>start</span>:dev
访问地址：http://localhost:3000/asr.html
2.6 调用大模型
修改前后端代码
想法很简单，就是当我拿到录音转成的文本以后，再调用大模型，让大模型回复我的问题。
主要添加代码如下：
JS代码
测试
2.7 大模型生成的东西用语音播放
页面很简单如图所示：
大模型回复生成以后，我点一下按钮，就能够给我朗读。
之前的“固定文本合成语音-tts”是将合成的文本写成MP3文件，存到了目录里面进行播放。
现在是点击按钮，直接播放。所以，前端需要一个播放器
将生成的接收到的信息传给player的src就好了。具体代码如下：
实现 html
实现voice.controller.ts
测试
点击语音播放，自动播放大模型搜出来的答案。
2.8 连生成文本带播放
上面这个例子是点击了小喇叭才能阅读，现在我想要的是，大模型流式输出多少，就能读多少。
连生成带读的效果。
先看看tts-stream.controller.ts里面,他的入参是一个数组，也就是我们只需要将接口返回的数据收集成数组就好了，但是阅读的时候是一句一句读的，生成的时候却不是，如图所示
从图中看到，他是一些没有规律的字符串，我们不能直接读，得等，等一句话成形以后，再读。所以在前端代码里面我们要去搜索标点符号。生成一句话，请求一次tts-stream接口。一共请求了4次。
实现html
代码块里面有一些注释的代码，是没有处理直接调用tts-stream接口的代码，和后面处理后一句一句读的进行对比。
<!doctype html>
<span><span><<span>html</span> <span>lang</span>=<span>"zh-CN"</span>></span>
<span><<span>head</span>></span>
<span><<span>meta</span> <span>charset</span>=<span>"utf-8"</span> /></span>
<span><<span>meta</span> <span>name</span>=<span>"viewport"</span> <span>content</span>=<span>"width=device-width, initial-scale=1"</span> /></span>
<span><<span>title</span>></span>ASR 录音上传测试<span></<span>title</span>></span>
<span><<span>style</span>></span><span>
<span>body</span> {
<span>font-family</span>:
-apple-system, BlinkMacSystemFont, <span>'Segoe UI'</span>, <span>'Microsoft YaHei'</span>,
sans-serif;
<span>max-width</span>: <span>720px</span>;
<span>margin</span>: <span>40px</span> auto;
<span>padding</span>: <span>0</span> <span>16px</span>;
<span>line-height</span>: <span>1.6</span>;
}
<span>button</span> {
<span>margin-right</span>: <span>8px</span>;
<span>padding</span>: <span>8px</span> <span>16px</span>;
<span>font-size</span>: <span>14px</span>;
<span>cursor</span>: pointer;
}
<span>button</span><span>:disabled</span> {
<span>cursor</span>: not-allowed;
<span>opacity</span>: <span>0.5</span>;
}
<span>.status</span> {
<span>color</span>: <span>#666</span>;
}
pre {
<span>background</span>: <span>#f6f8fa</span>;
<span>border-radius</span>: <span>6px</span>;
<span>padding</span>: <span>12px</span>;
<span>white-space</span>: pre-wrap;
<span>word-break</span>: break-all;
}
</span><span></<span>style</span>></span>
<span></<span>head</span>></span>
<span><<span>body</span>></span>
<span><<span>h1</span>></span>ASR 录音上传测试<span></<span>h1</span>></span>
<span><<span>p</span>></span>
点击「开始录音」并允许麦克风权限，点击「停止录音」后将录音上传到后端进行语音识别。
<span></<span>p</span>></span>
<span><<span>button</span> <span>id</span>=<span>"startBtn"</span>></span>开始录音<span></<span>button</span>></span>
<span><<span>button</span> <span>id</span>=<span>"stopBtn"</span> <span>disabled</span>></span>停止录音<span></<span>button</span>></span>
<span><<span>button</span> <span>id</span>=<span>"clearBtn"</span>></span>清理聊天记录<span></<span>button</span>></span>
<span><<span>p</span> <span>class</span>=<span>"status"</span> <span>id</span>=<span>"status"</span>></span>状态：空闲<span></<span>p</span>></span>
<span><<span>h3</span>></span>识别结果<span></<span>h3</span>></span>
<span><<span>p</span>></span>先识别出结果，然后再调用大模型回答你的问题<span></<span>p</span>></span>
<span><<span>pre</span> <span>id</span>=<span>"result"</span>></span>（暂无）<span></<span>pre</span>></span>
<span><<span>h3</span> <span>style</span>=<span>"display: inline-block"</span>></span>大模型的回复<span></<span>h3</span>></span>
<span><<span>button</span> <span>id</span>=<span>"voiceBtn"</span>></span>语音播放<span></<span>button</span>></span>
<span><<span>span</span> <span>style</span>=<span>"color: red"</span>></span>生成好以后才能播放呀<span></<span>span</span>></span>
<span><<span>audio</span> <span>id</span>=<span>"player"</span> <span>hidden</span>></span><span></<span>audio</span>></span>
<span><<span>pre</span> <span>id</span>=<span>"response"</span>></span>（暂无）<span></<span>pre</span>></span>
<span><<span>script</span>></span><span>
<span>const</span> startBtn = <span>document</span>.<span>getElementById</span>(<span>'startBtn'</span>);
<span>const</span> stopBtn = <span>document</span>.<span>getElementById</span>(<span>'stopBtn'</span>);
<span>const</span> clearBtn = <span>document</span>.<span>getElementById</span>(<span>'clearBtn'</span>);
<span>const</span> voiceBtn = <span>document</span>.<span>getElementById</span>(<span>'voiceBtn'</span>);
<span>const</span> player = <span>document</span>.<span>getElementById</span>(<span>'player'</span>);
<span>const</span> statusEl = <span>document</span>.<span>getElementById</span>(<span>'status'</span>);
<span>const</span> resultEl = <span>document</span>.<span>getElementById</span>(<span>'result'</span>);
<span>const</span> responseE1 = <span>document</span>.<span>getElementById</span>(<span>'response'</span>);
<span>let</span> mediaRecorder = <span>null</span>;
<span>let</span> audioChunks = [];
<span>let</span> mediaStream = <span>null</span>;
<span>const</span> fileName = <span>'record.ogg'</span>;
<span>function</span> <span>setStatus</span>(<span>text</span>) {
statusEl.<span>textContent</span> = <span>'状态：'</span> + text;
}
<span>//开启录音功能</span>
<span>async</span> <span>function</span> <span>startRecord</span>(<span></span>) {
<span>try</span> {
audioChunks = [];
<span>setStatus</span>(<span>'正在请求麦克风权限...'</span>);
mediaStream = <span>await</span> navigator.<span>mediaDevices</span>.<span>getUserMedia</span>({
<span>audio</span>: <span>true</span>,
});
mediaRecorder = <span>new</span> <span>MediaRecorder</span>(mediaStream);
mediaRecorder.<span>ondataavailable</span> = <span>(<span>event</span>) =></span> {
<span>if</span> (event.<span>data</span> && event.<span>data</span>.<span>size</span> > <span>0</span>) {
audioChunks.<span>push</span>(event.<span>data</span>);
}
};
mediaRecorder.<span>onstop</span> = <span>async</span> () => {
<span>setStatus</span>(<span>'录音结束，正在上传...'</span>);
<span>const</span> blob = <span>new</span> <span>Blob</span>(audioChunks, {
<span>type</span>: mediaRecorder.<span>mimeType</span> || <span>'audio/ogg'</span>,
});
audioChunks = [];
<span>const</span> text = <span>await</span> <span>uploadAudio</span>(blob);
<span>await</span> <span>askQuestion</span>(text);
};
mediaRecorder.<span>start</span>(<span>250</span>);
<span>setStatus</span>(<span>'录音中...'</span>);
startBtn.<span>disabled</span> = <span>true</span>;
stopBtn.<span>disabled</span> = <span>false</span>;
} <span>catch</span> (err) {
<span>setStatus</span>(<span>'录音失败：'</span> + err.<span>message</span>);
<span>stopRecordStream</span>();
}
}
<span>function</span> <span>stopRecord</span>(<span></span>) {
<span>if</span> (mediaRecorder && mediaRecorder.<span>state</span> !== <span>'inactive'</span>) {
mediaRecorder.<span>stop</span>();
}
startBtn.<span>disabled</span> = <span>false</span>;
stopBtn.<span>disabled</span> = <span>true</span>;
}
<span>function</span> <span>stopRecordStream</span>(<span></span>) {
<span>if</span> (mediaStream) {
mediaStream.<span>getTracks</span>().<span>forEach</span>(<span>(<span>track</span>) =></span> track.<span>stop</span>());
mediaStream = <span>null</span>;
}
startBtn.<span>disabled</span> = <span>false</span>;
stopBtn.<span>disabled</span> = <span>true</span>;
}
<span>async</span> <span>function</span> <span>uploadAudio</span>(<span>blob</span>) {
<span>try</span> {
<span>const</span> formData = <span>new</span> <span>FormData</span>();
formData.<span>append</span>(<span>'audio'</span>, blob, fileName);
<span>const</span> response = <span>await</span> <span>fetch</span>(<span>'/asr/asr'</span>, {
<span>method</span>: <span>'POST'</span>,
<span>body</span>: formData,
});
<span>if</span> (!response.<span>ok</span>) {
<span>throw</span> <span>new</span> <span>Error</span>(<span>'上传失败：HTTP '</span> + response.<span>status</span>);
}
<span>const</span> data = <span>await</span> response.<span>json</span>();
resultEl.<span>textContent</span> = data.<span>data</span> || <span>'（空结果）'</span>;
<span>setStatus</span>(<span>'识别完成'</span>);
<span>return</span> data.<span>data</span> || <span>''</span>;
} <span>catch</span> (err) {
<span>setStatus</span>(<span>'识别失败：'</span> + err.<span>message</span>);
resultEl.<span>textContent</span> = <span>'错误：'</span> + err.<span>message</span>;
} <span>finally</span> {
<span>stopRecordStream</span>();
}
}
<span>// async function autoTtsStream(text) {</span>
<span>// const res = await fetch('/tts/synthesize', {</span>
<span>// method: 'POST',</span>
<span>// headers: { 'Content-Type': 'application/json' },</span>
<span>// body: JSON.stringify({ texts: text }),</span>
<span>// });</span>
<span>// const blob = await res.blob();</span>
<span>// player.src = URL.createObjectURL(blob);</span>
<span>// await player.play();</span>
<span>// }</span>
<span>// function askQuestion(text) {</span>
<span>// // 参数只能走 URL query，EventSource 发不了 body</span>
<span>// const url = `/ai/chat/stream?query=${encodeURIComponent(text)}`;</span>
<span>// const es = new EventSource(url);</span>
<span>// const response = [];</span>
<span>// es.onmessage = (e) => {</span>
<span>// const chunk = JSON.parse(e.data); // 后端 map 出来是什么结构就怎么取</span>
<span>// console.log(chunk.content, 7777);</span>
<span>// if (chunk.content) {</span>
<span>// responseE1.textContent += chunk.content;</span>
<span>// response.push(chunk.content);</span>
<span>// autoTtsStream(response);</span>
<span>// } // 追加到页面</span>
<span>// if (chunk.done) es.close();</span>
<span>// };</span>
<span>// es.onerror = (e) => {</span>
<span>// console.error('SSE 出错或已结束', e);</span>
<span>// es.close(); // 必须关，否则浏览器会不断重连</span>
<span>// };</span>
<span>// }</span>
<span>//上述代码的知名问题是不断重复读，把浏览器干死</span>
<span>// ========== 朗读状态 ==========</span>
<span>let</span> readBuffer = <span>''</span>; <span>// 攒文字</span>
<span>let</span> readQueue = []; <span>// 待播句子队列</span>
<span>let</span> isReading = <span>false</span>; <span>// 是否正在播（防并发）</span>
<span>let</span> sseDone = <span>false</span>; <span>// 大模型是否说完</span>
<span>let</span> currentEs = <span>null</span>; <span>// 当前 SSE 连接</span>
<span>/** 把新文字攒起来，凑够一句就切出来 */</span>
<span>function</span> <span>feedForRead</span>(<span>text</span>) {
readBuffer += text;
<span>// 找最后一个句末标点</span>
<span>let</span> idx = -<span>1</span>;
<span>for</span> (<span>const</span> p <span>of</span> [<span>'。'</span>, <span>'！'</span>, <span>'？'</span>, <span>'；'</span>, <span>'\n'</span>, <span>'!'</span>, <span>'?'</span>]) {
<span>const</span> i = readBuffer.<span>lastIndexOf</span>(p);
<span>if</span> (i > idx) idx = i;
}
<span>if</span> (idx === -<span>1</span>) <span>return</span>; <span>// 还没凑够一句</span>
<span>const</span> sentence = readBuffer.<span>slice</span>(<span>0</span>, idx + <span>1</span>).<span>trim</span>();
readBuffer = readBuffer.<span>slice</span>(idx + <span>1</span>);
<span>if</span> (sentence) {
<span>console</span>.<span>log</span>(<span>'切出一句:'</span>, sentence);
readQueue.<span>push</span>(sentence);
<span>pumpQueue</span>();
}
}
<span>/** 播放队列：一次只播一段，播完自动取下一句 */</span>
<span>async</span> <span>function</span> <span>pumpQueue</span>(<span></span>) {
<span>if</span> (isReading) <span>return</span>; <span>// 正在播，直接返回（靠 finally 递归）</span>
<span>if</span> (readQueue.<span>length</span> === <span>0</span>) <span>return</span>;
isReading = <span>true</span>;
<span>const</span> sentence = readQueue.<span>shift</span>();
<span>let</span> url = <span>null</span>;
<span>try</span> {
<span>setStatus</span>(<span>'朗读中：'</span> + sentence.<span>slice</span>(<span>0</span>, <span>15</span>) + <span>'...'</span>);
<span>const</span> res = <span>await</span> <span>fetch</span>(<span>'/tts/synthesize'</span>, {
<span>method</span>: <span>'POST'</span>,
<span>headers</span>: { <span>'Content-Type'</span>: <span>'application/json'</span> },
<span>body</span>: <span>JSON</span>.<span>stringify</span>({ <span>texts</span>: [sentence], <span>codec</span>: <span>'mp3'</span> }), <span>// ← 只传一句</span>
});
<span>if</span> (!res.<span>ok</span>) <span>throw</span> <span>new</span> <span>Error</span>(<span>'TTS HTTP '</span> + res.<span>status</span>);
<span>const</span> blob = <span>await</span> res.<span>blob</span>();
url = <span>URL</span>.<span>createObjectURL</span>(blob);
<span>await</span> <span>playOne</span>(url);
} <span>catch</span> (err) {
<span>console</span>.<span>warn</span>(<span>'朗读失败:'</span>, sentence, err);
} <span>finally</span> {
<span>if</span> (url) <span>URL</span>.<span>revokeObjectURL</span>(url); <span>// 释放内存</span>
isReading = <span>false</span>;
<span>pumpQueue</span>(); <span>// 播完继续下一句</span>
}
}
<span>/** 播放单段，返回 Promise */</span>
<span>function</span> <span>playOne</span>(<span>url</span>) {
<span>return</span> <span>new</span> <span>Promise</span>(<span>(<span>resolve</span>) =></span> {
player.<span>onended</span> = resolve;
player.<span>onerror</span> = resolve; <span>// 出错也继续，别卡死队列</span>
player.<span>src</span> = url;
player.<span>play</span>().<span>catch</span>(resolve);
});
}
<span>/** 收尾：把残留的半句也播掉 */</span>
<span>function</span> <span>finishReading</span>(<span></span>) {
<span>const</span> rest = readBuffer.<span>trim</span>();
readBuffer = <span>''</span>;
sseDone = <span>true</span>;
<span>if</span> (rest) {
readQueue.<span>push</span>(rest);
<span>pumpQueue</span>();
}
}
<span>/** 停止朗读 */</span>
<span>function</span> <span>stopReading</span>(<span></span>) {
readQueue.<span>length</span> = <span>0</span>;
readBuffer = <span>''</span>;
sseDone = <span>false</span>;
<span>try</span> {
player.<span>pause</span>();
player.<span>removeAttribute</span>(<span>'src'</span>);
} <span>catch</span> {}
<span>if</span> (currentEs) currentEs.<span>close</span>();
}
<span>// ========== SSE 接收 ==========</span>
<span>function</span> <span>askQuestion</span>(<span>text</span>) {
<span>if</span> (!text) <span>return</span>;
responseE1.<span>textContent</span> = <span>''</span>;
<span>stopReading</span>(); <span>// 清掉上一轮</span>
<span>const</span> url = <span>`/ai/chat/stream?query=<span>${<span>encodeURIComponent</span>(text)}</span>`</span>;
<span>const</span> es = <span>new</span> <span>EventSource</span>(url);
currentEs = es;
es.<span>onmessage</span> = <span>(<span>e</span>) =></span> {
<span>let</span> chunk;
<span>try</span> {
chunk = <span>JSON</span>.<span>parse</span>(e.<span>data</span>);
} <span>catch</span> {
<span>return</span>;
}
<span>if</span> (!chunk.<span>content</span>) <span>return</span>;
responseE1.<span>textContent</span> += chunk.<span>content</span>; <span>// 文字照常逐字显示</span>
<span>feedForRead</span>(chunk.<span>content</span>); <span>// 同时喂给朗读器</span>
<span>if</span> (chunk.<span>done</span>) {
es.<span>close</span>();
<span>finishReading</span>();
}
};
es.<span>onerror</span> = <span>() =></span> {
es.<span>close</span>(); <span>// 必须关，否则自动重连</span>
<span>if</span> (responseE1.<span>textContent</span>.<span>trim</span>()) <span>finishReading</span>(); <span>// 后端不推 done 也能收尾</span>
};
}
<span>// clearRecord 里加停止朗读</span>
<span>function</span> <span>clearRecord</span>(<span></span>) {
<span>stopReading</span>(); <span>// ← 加这行</span>
resultEl.<span>textContent</span> = <span>'（暂无）'</span>;
responseE1.<span>textContent</span> = <span>'（暂无）'</span>;
<span>setStatus</span>(<span>'空闲'</span>); <span>// ← 原来是 '状态：空闲'，会显示两遍"状态："</span>
}
<span>async</span> <span>function</span> <span>playTts</span>(<span></span>) {
<span>const</span> text = responseE1.<span>textContent</span>.<span>trim</span>();
<span>if</span> (!text || text === <span>'（暂无）'</span>) {
<span>setStatus</span>(<span>'还没有内容可以播'</span>);
<span>return</span>;
}
voiceBtn.<span>disabled</span> = <span>true</span>;
<span>setStatus</span>(<span>'正在生成语音...'</span>);
<span>// 加时间戳防止浏览器缓存旧音频</span>
player.<span>src</span> = <span>`/voice/tts?text=<span>${<span>encodeURIComponent</span>(text)}</span>&t=<span>${<span>Date</span>.now()}</span>`</span>;
<span>try</span> {
<span>await</span> player.<span>play</span>();
<span>setStatus</span>(<span>'播放中'</span>);
} <span>catch</span> (err) {
<span>setStatus</span>(<span>'播放失败：'</span> + err.<span>message</span>);
} <span>finally</span> {
voiceBtn.<span>disabled</span> = <span>false</span>;
}
}
startBtn.<span>addEventListener</span>(<span>'click'</span>, startRecord);
stopBtn.<span>addEventListener</span>(<span>'click'</span>, stopRecord);
clearBtn.<span>addEventListener</span>(<span>'click'</span>, clearRecord);
voiceBtn.<span>addEventListener</span>(<span>'click'</span>, playTts);
</span><span></<span>script</span>></span>
<span></<span>body</span>></span>
<span></<span>html</span>></span></span>
实现tts-stream.controller.ts
把原来的get接口改成post接口，然后把原来存数音频的部分删掉。
<span>import</span> { <span>Body</span>, <span>Controller</span>, <span>Post</span>, <span>Res</span> } <span>from</span> <span>'@nestjs/common'</span>;
<span>import</span> * <span>as</span> path <span>from</span> <span>'path'</span>;
<span>import</span> { <span>TtsService</span> } <span>from</span> <span>'./tts-stream.service.js'</span>;
<span>import</span> { <span>ConfigService</span> } <span>from</span> <span>'@nestjs/config'</span>;
interface <span>SynthesizeDto</span> {
<span>texts</span>: string[]; <span>// 必须以 。！？； 结尾</span>
codec?: <span>'pcm'</span> | <span>'mp3'</span>; <span>// 可选，默认 mp3</span>
filename?: string; <span>// 可选，默认 demo</span>
}
@<span>Controller</span>(<span>'tts'</span>)
<span>export</span> <span>class</span> <span>TtsController</span> {
<span>constructor</span>(<span>
private readonly ttsService: TtsService,
private readonly configService: ConfigService,
</span>) {}
@<span>Post</span>(<span>'synthesize'</span>)
<span>async</span> <span>synthesize</span>(<span>@Body() body: SynthesizeDto, @Res() res: any</span>) {
<span>// ===== 参数定死 =====</span>
<span>// const texts = [</span>
<span>// '你好，欢迎使用流式语音合成。',</span>
<span>// '我是你的语音小助手，你有任何需要都可以给我说',</span>
<span>// ]; // 必须以 。！？； 结尾</span>
<span>const</span> texts = body?.<span>texts</span>;
<span>if</span> (!<span>Array</span>.<span>isArray</span>(texts) || texts.<span>length</span> === <span>0</span>) {
<span>return</span> res
.<span>status</span>(<span>400</span>)
.<span>json</span>({ <span>message</span>: <span>'texts 不能为空，且必须是字符串数组'</span> });
}
<span>const</span> <span>codec</span>: <span>'pcm'</span> | <span>'mp3'</span> = body.<span>codec</span> ?? <span>'mp3'</span>;
<span>const</span> filename = body.<span>filename</span> ?? <span>'demo'</span>;
<span>const</span> options = {
<span>appId</span>: <span>Number</span>(<span>this</span>.<span>configService</span>.<span>get</span>(<span>'TENCENT_APP_ID'</span>)),
<span>secretId</span>: <span>this</span>.<span>configService</span>.<span>get</span>(<span>'TENCENTCLOUD_SECRET_ID'</span>),
<span>secretKey</span>: <span>this</span>.<span>configService</span>.<span>get</span>(<span>'TENCENTCLOUD_SECRET_KEY'</span>),
codec,
};
<span>const</span> ext = codec === <span>'mp3'</span> ? <span>'mp3'</span> : <span>'wav'</span>;
<span>//const outputPath = path.join(process.cwd(), 'output', `${filename}.${ext}`);</span>
res.<span>setHeader</span>(<span>'Content-Type'</span>, codec === <span>'mp3'</span> ? <span>'audio/mpeg'</span> : <span>'audio/pcm'</span>);
res.<span>setHeader</span>(
<span>'Content-Disposition'</span>,
<span>`inline; filename="<span>${filename}</span>.<span>${ext}</span>"`</span>,
);
res.<span>setHeader</span>(<span>'Cache-Control'</span>, <span>'no-store'</span>); <span>// 防止浏览器缓存，改代码后能立刻生效</span>
<span>try</span> {
<span>// await this.ttsService.synthesize(</span>
<span>// options,</span>
<span>// texts,</span>
<span>// {</span>
<span>// onAudio: (chunk: any) => res.write(chunk),</span>
<span>// },</span>
<span>// outputPath,</span>
<span>// );</span>
<span>//outputPath会存数据到磁盘，不写就不存入</span>
<span>await</span> <span>this</span>.<span>ttsService</span>.<span>synthesize</span>(options, texts, {
<span>onAudio</span>: <span>(<span>chunk: any</span>) =></span> res.<span>write</span>(chunk),
});
res.<span>end</span>();
} <span>catch</span> (<span>err</span>: any) {
<span>if</span> (!res.<span>headersSent</span>) res.<span>status</span>(<span>500</span>).<span>end</span>(err.<span>message</span>);
<span>else</span> res.<span>destroy</span>();
}
}
}
测试
点击停止录音，数据进入大模型开始回答问题，实现边输出答案，边阅读的功能。
2.9 是否需要使用@nestjs/event-emitter
@nestjs/event-emitter 是 NestJS 官方的事件总线，让代码之间可以"喊一嗓子"，而不用互相认识。
现在大模型的输出用的是SSE，实现了打印机效果。
流式文本合成语音使用的又是websocket。此时为了解决这个问题。
如果我们在一个接口里面，完成上述两个问题，那么就需要使用@nestjs/event-emitter，去调度事件。
文章提供了从零搭建AI语音交互的完整代码实践，涵盖腾讯云API调用、前后端集成及流式处理，适合开发者快速实现语音对话助手。