Is it possible to do a microphone check?

[zhangchi199133]

Hello! I was wondering whether there is some way I can test the functionality of the microphone? Ideally some features that allows the experiment to proceed only when voice input is detected. Many thanks!

[jodeleeuw]

Hi @zhangchi199133,

We don't currently have any officially released plugins that deal with audio input. @becky-gilbert has been developing an audio-input plugin. You can see more about that here. She's looking for folks willing to test it out. I don't know off the top of my head whether it does a mic check, but that would be useful feedback to have!

[zhangchi199133]

Dear Josh,

Thank you so much for your answer (and for the wonderful online workshop, too)! Indeed I am using Becky's plugin for a structural priming experiment I will put online soon. The microphone check is the first issue I encountered. I am quite a newbie on jsPsych but I am very happy to test out the audio input plugin. So far I found it quite handy.

[annieholtz]

Hi @zhangchi199133,

Just going to slide into this conversation to link to a demo experiment that my colleague and I made which can do audio recording and playback. The playback option could be used to check audio levels at the start of an experiment if you wish. https://experiments.ppls.ed.ac.uk/

[zhangchi199133]

Hi @annieholtz ,

Thank you so much for the interesting plugin! I was wondering whether there is anyway I can get the access to the audio file when I only host the experiment on my local PC? Thanks!

[annieholtz]

Hi @zhangchi199133
Glad you found it interesting! I'm afraid that I'm not very familiar with running jspsych experiments locally and currently the demo is only functional when the experiment is run on a server.

[zhangchi199133]

@annieholtz I will try it on a server then. Thank you!

[becky-gilbert]

That's an interesting question and would be a great feature! Off the top of my head, here are a few ways to address this:

(1) Check that the participant's computer has access to a mic. This seems pretty straightforward, e.g.:
https://www.webrtc-experiment.com/DetectRTC/
https://stackoverflow.com/questions/30047056/is-it-possible-to-check-if-the-user-has-a-camera-and-microphone-and-if-the-permi/30047627
So I'm planning to add this check into the plugin.

(2) Do a test recording where you play the audio back to the participant, and then have them confirm that the recording worked. If not, ask them stop and contact you (and make sure it's clear that they will still be paid for their time if it doesn't work). This should be possible with the current version of the plugin.

(3) Do some online audio processing on the test recording to check that audio input has been detected. I don't think there's an easy way to differentiate a voice from other sounds, but it would confirm that the mic picked up some kind of input. I have no idea how easy/difficult this would be to implement, but I can look into it!

[zhangchi199133]

@becky-gilbert The R code works perfectly. Thank you!

[christinalubinus]

Dear @becky-gilbert,

When looking for a solution to my base64-to-wav conversion problem, I came across this old post of yours. I have a follow-up question to your code, if you don't mind.

In the experiment, I am recording speech data using the audio-audio-plugin and have each trial saved as base64 string. Converting the files to wav caused some problems for me in python, which is why I was happy to find your R code. I am not very familiar with R, so the question might be quite naive. But here goes - how do you load your base64 string? I am asking because I read in the txt file (containing only the base64 string for this trial, no other information), store it in a variable and pass it into base64decode. However, I receive the error I can only decode base64 strings. Then I tried to force my_data to be a string using toString(my_data), that didn't help either. Additionally, I tried different ways to read in the file, none was more successful though.

library(base64enc)
library("readr")

infile = '/path-to-my-file/sub001/whisper_audio_0.txt'
my_data <- read_delim(infile,delim=" ")
y <- base64decode(my_data)
myfile <- file("/path-to-my-file/sub001/whisper_audio_0.wav","wb")
writeBin(y, myfile)
close(myfile)

Do you maybe have an idea what is going wrong here?

Thanks in advance.
All the best,
Christina

[becky-gilbert]

Hi @christinalubinus, here's the R code I've been using for my audio recordings recently:

library(jsonlite)
library(base64enc)
source('parseJSONdata.R')

file_name <- "test1"

# Read in the raw data
data_file <- file(paste(file_name, ".txt", sep=""), open = "r")
raw_data <- readLines(data_file, warn = FALSE)
close(data_file)

# custom function for converting JSON to R dataframe
all_data <- parseJSONdata(raw_data, numComponents=1, isJsonStr=TRUE, id="1", returnResults=TRUE, saveToFile=FALSE)

trial_count <- 0
# loop through the data rows (trials) and convert base 64 data to audio (webm)
for (i in 1:nrow(all_data)) {
  decoded <- base64decode(all_data$audio_data[i])
  if (!is.na(all_data$audio_data[i])) {
    trial_count <- trial_count+1
    aud_file <- file(paste(file_name, "_", trial_count, ".webm", sep=""),"wb")
    writeBin(decoded, aud_file)
    close(aud_file)
  }
}

Sorry about the extra stuff in this code - I'm sharing it all because I'm not sure exactly which bits might be relevant to the error you're getting. I'm not sure whether it matters that I'm using readLines rather than read_delim to read in the raw jsPsych data, and then parsing the JSON (I assume your infile is just the base 64 data and doesn't need to be parsed?). Sorry if this isn't very helpful - not sure what to suggest!

[christinalubinus]

Hi @becky-gilbert !

Thanks for your reply and the code example. It does work for me and creates an output file. However, I am still facing the same problem as when I tried to convert with Python. The audio can be read by VLC player but not by other programs such as Audacity, Praat or Python/Matlab. Since I need to run frequency analysis etc. on the audio, I need the file to be read by Python or Matlab. But I will keep looking for a solution and post it here, if/when I find one.

Best,
Christina

[becky-gilbert]

Hi @christinalubinus, yes I had the same issue with the .webm files not being readable by programs other than VLC. My solution was to batch-convert them to .mp3 using VLC's convert/save feature. I realize this isn't an ideal solution - I haven't had time yet to work on saving the data to other formats like mp3 or wav. BTW, if anyone wants to share their code for doing this (plugin changes and/or custom postprocessing function), I would be very grateful!

[gustavoestivalet]

Hi, guys, I just would like to give my contribution on this topic. I have been working in picture naming tasks and I developed a R script to convert base64 to .wav with fftmpeg conversion, naming each file correctly, and making the file able to open in any software, but specially CheckVocal to track the RT and ACC. One important detail for naming the files, "word" is the name of the picture and may be defined inside "data" in jsPsych, as the example below.

# Example of stimulus in jsPsych/cognition
{stimulus: "<img src='flower.jpg'>", data: {word: "flower"}},

The first block loads the R libraries. The second block opens data directly downloaded from cognition. The third block filters only "response == base64". The fourth block makes the following routine for each file: i) converts base64 to raw; ii) define the input ".webm" file name and iii) define the output ".wav" file name, both using "run_id" and "word" information; iv) the decoded file is written as ".webm"; and the ".webm" file is converted to ".wav" using fftmpeg. The last block deletes all ".webm" files. Have fun!

# Load libraries
library(base64enc)
library(av)

# Open data
data1 <- read.csv("picture-naming.csv", header = TRUE)

# Clean data
data2 <- subset(data1, trial_type == "html-audio-response")

# Decode base64 to .wav
for(i in 1:nrow(data2)) {
  decoded <- base64decode(data2$response[i])
  input_file_name <- paste0(data2$run_id[i], "_", data2$word[i], ".webm")
  output_file_name <- paste0(data2$run_id[i], "_", data2$word[i], ".wav")
  writeBin(decoded, input_file_name)
  av_audio_convert(input_file_name, output_file_name)
}

# Remove .webm files
file.remove(list.files(pattern = "*.webm"))