Stream preview #157

longnguyen2004 · 2025-02-26T09:48:58Z

Only took 2 months since #133 was made...

Should keyframe extraction be in this repo though, or it should be in https://github.com/Discord-RE/FFmpeg-input-premade?

pkg-pr-new · 2025-02-26T09:49:25Z

Open in Stackblitz

npm i https://pkg.pr.new/Discord-RE/Discord-video-stream/@dank074/discord-video-stream@157

commit: b66da92

aiko-chan-ai · 2025-02-26T10:06:31Z

The function to set stream preview is now available in the library
https://github.com/aiko-chan-ai/discord.js-selfbot-v13/blob/main/src/structures/VoiceState.js#L317
Example:

guild.members.me.voice.postPreview(base64img)

To be more precise, it has been available for several versions already

longnguyen2004 · 2025-02-26T11:59:13Z

Does it work with this library handling the voice connection? If yes then I'll close

aiko-chan-ai · 2025-02-26T12:24:31Z

It will work with this library, but it requires users to clearly specify which server's voice channel they are connecting to (for example, if they are connecting to server A, they will need to retrieve their own VoiceState). Alternatively, you can use this library's APIRequest to create a more secure request (there might be a typing error for ts, I will fix it to be a public property later)
And capturing a frame from the video is still very important

longnguyen2004 · 2025-02-27T08:32:05Z

Alright so we have a bit of a problem regarding keyframe extraction. There are 3 options, with varying level of annoyance

In-process decoding: Requires no changes from user side, but since upstream libav.js doesn't include decoders for H.26x, we'll need to build our own version and publish it on npm (might be annoying but probably doable). Speed is also a potential issue, and we might need to bring back multithreading (yet again...) and enable SIMD
Extra keyframe video stream inside the existing mkv stream: Not supported by fluent-ffmpeg (only 1 audio stream and 1 video stream per output)
Extra mkv stream containing the keyframe stream: Requires 2 demuxer instances (eww), pollutes all the API since you need to pass this keyframe stream around

Since option 1 changes nothing API-wise, I'll try to build libav.js and see how difficult it is to work with Emscripten

longnguyen2004 · 2025-02-27T14:00:52Z

Not too hard I'd say https://www.npmjs.com/package/@lng2004/libav.js-variant-webcodecs-avf-with-decoders
Now for the speed test...

dank074 · 2025-02-27T14:17:23Z

A little late since you've already done it with libav.js

Extra keyframe video stream inside the existing mkv stream: Not supported by fluent-ffmpeg (only 1 audio stream and 1 video stream per output)

You can also use https://github.com/dank074/fluent-ffmpeg-multistream-ts for multiple outputs/inputs

longnguyen2004 · 2025-02-27T14:27:13Z

That is option 3 above (extra mkv stream containing the decoded keyframes). I wanted to avoid it since that means running 2 demuxer instances, which I don't like, and it also requires us to pass an extra stream around, which is not ideal.

dank074 · 2025-02-27T17:11:44Z

You're right, that's actually option 3, sorry misunderstood at first glance

I screwed up the last build, let's try again

longnguyen2004 · 2025-02-28T05:15:34Z

It works!

For performance: Decoding, resizing and encode to jpg a 1920x1080 H.264/5 keyframe on an i3-330M takes ~1s, and increases the CPU usage to 30%, which is actually better than I expected, considering that there are no SIMD optimizations. Since we only decode once every 5s (which is overkill even, since Discord doesn't load the preview that often), this is acceptable for me.

longnguyen2004 · 2025-02-28T08:44:42Z

I love Deno and Bun

longnguyen2004 · 2025-02-28T08:50:41Z

!!!!! memory leak

Let's see if this is the source of the issue

longnguyen2004 · 2025-02-28T15:26:51Z

Leak is gone, I used the wrong free function... (note to self: if there's a function x and a function x_js, it's for a good reason, use the right one)

This version enables `O3` optimization level

longnguyen2004 · 2025-03-01T07:37:39Z

With O3, decoding a somewhat complex H.264 keyframe at 1080p 10000kbps takes ~200ms on an i3-330M, which is more than acceptable. Decoding interval can be increased further since Discord doesn't update preview often. Base memory usage is ~500MB for 1920x1080, higher quality will obviously consume more.

Again, big disclaimer that this doesn't work on Deno or Bun, and trying to enable it anyway will result in catastrophic failure.

dank074

Sorry about the delay, somehow my comment stayed as pending and was never actually submitted

Just one comment

dank074 · 2025-03-03T17:40:46Z

src/client/Streamer.ts

+        const { streamKey } = this.voiceConnection.streamConnection;
+        const data = `data:image/jpeg;base64,${image.toString("base64")}`;
+
+        await fetch(`https://discord.com/api/v9/streams/${streamKey}/preview`, {


Did you find that aiko's library postPreview or APIRequest wouldn't work here? I think it's a good idea to try to stick to it as much as possible to prevent getting possibly detected by Discord anti bot measures since the HTTP headers between requests might potentially be different (e.g. user-agent)

Is it exported yet? I'll have a look later

code snippet

const voiceState = client.guilds.cache.get(server_id).members.me.voice; await voiceState.postPreview('base64image'); // If error code is USER_NOT_STREAMING, try setting the property 'streaming' to true // If this error occurs, it may be because the VOICE_STATE_UPDATE event is not cached properly voiceState.streaming = true; await voiceState.postPreview('base64image');

Wouldn't it be simpler to just do

await this.client.user.voice.postPreview(data);

since users can't join multiple voice channels anyway (bots can, as long as they're in separate servers)

How about we expose the APIRequest thing for now? It's kinda weird for us to re-query information that we already have (but then again we already kinda did that in some ways so I'll try the suggestion above)

Wouldn't it be simpler to just do

await this.client.user.voice.postPreview(data);

since users can't join multiple voice channels anyway (bots can, as long as they're in separate servers)

Users can also join multiple voice channels across different servers

Add setStreamPreview function

afff7a8

Start work on keyframe decoder

5eea988

longnguyen2004 added 6 commits February 28, 2025 09:38

Finish the job

a3f3d5f

Fix import

111c8cf

Update libav.js

c09c8ef

I screwed up the last build, let's try again

Edit filter description

ed13d52

Update libav.js

95c0302

Specify filter type

3ab1b2f

longnguyen2004 force-pushed the stream-preview branch from 115711d to 3ab1b2f Compare February 28, 2025 04:14

longnguyen2004 added 3 commits February 28, 2025 11:31

Properly initialize the filter

6c8c74e

Copy codec parameters only once

2742b6c

Replace async IIFE with promise chain

3029f98

Copy serialize logic from libav.js, add decoding failsafe

f82db97

longnguyen2004 added 5 commits February 28, 2025 15:57

Free the filtergraph

5a8f328

Serialize the free, prevent decoding after free

dee6a50

TypeScript fun

a758b78

Copy out codecpar

3e199eb

Update libav.js

e122cc6

longnguyen2004 added 2 commits February 28, 2025 18:40

Separate decoder and demuxer instance

5b28ba3

Let's see if this is the source of the issue

Send fin on free, use the proper free function

8f57a09

longnguyen2004 added 2 commits March 1, 2025 14:13

Add some logging, add disclaimer about Deno and Bun

9327bcf

Update libav.js

fbbe02e

This version enables `O3` optimization level

longnguyen2004 marked this pull request as ready for review March 1, 2025 07:37

longnguyen2004 added 3 commits March 1, 2025 15:03

Remove unused LibavInstance file

f94b274

Add freeing decoder log

56f89ab

Add cleanup guard to prevent double calling

b5d5d47

dank074 reviewed Mar 3, 2025

View reviewed changes

Increase update interval to 15s, tweak scaling parameters

b66da92

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Stream preview #157

Stream preview #157

longnguyen2004 commented Feb 26, 2025

pkg-pr-new bot commented Feb 26, 2025 •

edited

Loading

aiko-chan-ai commented Feb 26, 2025 •

edited

Loading

longnguyen2004 commented Feb 26, 2025

aiko-chan-ai commented Feb 26, 2025 •

edited

Loading

longnguyen2004 commented Feb 27, 2025 •

edited

Loading

longnguyen2004 commented Feb 27, 2025

dank074 commented Feb 27, 2025 •

edited

Loading

longnguyen2004 commented Feb 27, 2025

dank074 commented Feb 27, 2025

longnguyen2004 commented Feb 28, 2025 •

edited

Loading

longnguyen2004 commented Feb 28, 2025

longnguyen2004 commented Feb 28, 2025

longnguyen2004 commented Feb 28, 2025

longnguyen2004 commented Mar 1, 2025 •

edited

Loading

dank074 left a comment

dank074 Mar 3, 2025

longnguyen2004 Mar 4, 2025

aiko-chan-ai Mar 4, 2025

dank074 Mar 4, 2025

longnguyen2004 Mar 5, 2025 •

edited

Loading

aiko-chan-ai Mar 5, 2025

Stream preview #157

Are you sure you want to change the base?

Stream preview #157

Conversation

longnguyen2004 commented Feb 26, 2025

pkg-pr-new bot commented Feb 26, 2025 • edited Loading

aiko-chan-ai commented Feb 26, 2025 • edited Loading

longnguyen2004 commented Feb 26, 2025

aiko-chan-ai commented Feb 26, 2025 • edited Loading

longnguyen2004 commented Feb 27, 2025 • edited Loading

longnguyen2004 commented Feb 27, 2025

dank074 commented Feb 27, 2025 • edited Loading

longnguyen2004 commented Feb 27, 2025

dank074 commented Feb 27, 2025

longnguyen2004 commented Feb 28, 2025 • edited Loading

longnguyen2004 commented Feb 28, 2025

longnguyen2004 commented Feb 28, 2025

longnguyen2004 commented Feb 28, 2025

longnguyen2004 commented Mar 1, 2025 • edited Loading

dank074 left a comment

Choose a reason for hiding this comment

dank074 Mar 3, 2025

Choose a reason for hiding this comment

longnguyen2004 Mar 4, 2025

Choose a reason for hiding this comment

aiko-chan-ai Mar 4, 2025

Choose a reason for hiding this comment

dank074 Mar 4, 2025

Choose a reason for hiding this comment

longnguyen2004 Mar 5, 2025 • edited Loading

Choose a reason for hiding this comment

aiko-chan-ai Mar 5, 2025

Choose a reason for hiding this comment

pkg-pr-new bot commented Feb 26, 2025 •

edited

Loading

aiko-chan-ai commented Feb 26, 2025 •

edited

Loading

aiko-chan-ai commented Feb 26, 2025 •

edited

Loading

longnguyen2004 commented Feb 27, 2025 •

edited

Loading

dank074 commented Feb 27, 2025 •

edited

Loading

longnguyen2004 commented Feb 28, 2025 •

edited

Loading

longnguyen2004 commented Mar 1, 2025 •

edited

Loading

longnguyen2004 Mar 5, 2025 •

edited

Loading