Turn Spectro into a library

[calebj0seph]

At the moment Spectro just builds to a demo. It should be turned into a library, with the existing demo being moved into a demo or examples folder.

I imagine the API looking something like this:

// Creates a spectrogram attached to the given canvas, provides automatic handling
// of when the canvas is resized
const canvas = document.getElementById('spectrogram-canvas');
const spectrogram = new Spectrogram(canvas);

// Change display parameters of the spectrogram
spectrogram.updateRenderSettings({
    zoom: 1.5,
    minFrequencyHz: 0,
    maxFrequencyHz: 12000,
    scale: 'mel',
    color: /* object representing a color gradient */
});

// Start recording from mic, will prompt for permissions and return a promise once
// recording has started
await spectrogram.recordFromMic();

// Play audio file, can take URL or File object with option to mute output
await spectrogram.playAudioFile(file, mute = true);

// Manual API where audio data is streamed to the spectrogram
const renderAudioBuffer = await spectrogram.streamAudio();
// Call renderAudioBuffer() each time there is an audio buffer to render, returns a
// promise
// Calling renderAudioBuffer() after a call to stop() is an error

// Stop recording from mic or playing audio file
await spectrogram.stop();

// Add event listener to handle state changes
spectrogram.addStateChangeEventListener((state) => {
    // state can be one of 'stopped', 'playing-mic', 'playing-file', or
    // 'playing-custom'
});

[Elvincth]

Really looking forward to this !

[gokgozf]

Hi @calebj0seph
I was looking at the possible libraries around, and I saw your demo, it looks quite good.
I have two questions about making the library issue?

1. do you have a plan to release on npm?
2. do you plan to maintain the repository and accept pull requests?

[rechmbrs]

@calebj0seph

Please have a look at these websites:
https://stackoverflow.com/questions/6663222/doing-fft-in-realtime
https://www.dsprelated.com/showarticle/776.php

They refer to what is called Sliding Window Fourier Transform which is a much faster way to compute the spectrogram. The FFT cost is nLogn where as the SWFT is n.

Using the SWFT should reduce the lag between the sound and the display of the spectrogram.

I'm looking for a Javascript version I had and will send it up if I can find it.

RONC

[calebj0seph]

@rechmbrs Thanks for the links!

From what I understand a sliding DFT is used when you want to compute the DFT for windows which are only 1 sample apart. In this case, yes it makes much more sense to use an O(N) algorithm each window rather than an O(N*log(N)) algorithm since you're computing it every sample.

For a spectrogram though, we don't need to compute the DFT every sample. In Spectro, we only do this every 1024 samples with a window size of 4096 samples. If we do some quick calculations, we end up with (samples / 1024) * (4096 * log(4096)) operations. If we were to replace this with a sliding DFT, instead calculating the DFT for a window every sample, we end up with samples * 4096 operations. This might seem simpler at first, but if you do the math the sliding DFT ends up being orders of magnitude slower since we have to compute it every sample instead of every 1024 samples.

Unless I'm misunderstanding something, it doesn't look like this would help performance at all.

[calebj0seph]

@gokgozf Sorry for the late reply, haven't really touched any of my Github projects since COVID all started. Now that I'm getting back into things, I'm hoping to make Spectro into a reusable library with an npm package while also keeping this repository maintained. I've started by upgrading all of the dependencies, next I'm going to be refactoring the codebase into a library while putting the current demo in a separate directory that uses the new library. No ETA yet as I haven't worked with the code in a year, but hopefully not too long!

[rechmbrs]

@calebj0seph
the sliding DFT can be used to drop m samples and pickup p samples. Most often m = p = 1.

Please contact me at rechmbrs [at] gmail [dot] com to follow on with this. I have a number of other parts to discuss. We can then bring results back here for others.

RONC