Upscaling videos guide

[ivabus]

Just finished upscaling a video with Upscayl and here is a result (YouTube (up to 4K), file (big), file (smol)).
Model used: "Digital art" - "realesrgan-x4plus-anime"

Formats

Video: 1280x720 -> 5120x2880 (5K)
Framerate: 30
Length: 3:20 or 6015 frames

Steps

Split video into frames

ffmpeg -i /path/to/video.format -r FRAMERATE /path/to/frames/%08d.png

Extract audio from file (optional)

ffmpeg -i /path/to/video.format -vn -acodec copy /path/to/audio.mkv

Batch Upscayl

Just use Upscayl in the "Batch" mode

Running in parallel on the MacBook Air M2 and MacBook Air M1, it was done in 10 hours. (2.8.6)

Join frames into video

Without audio

ffmpeg -framerate FRAMERATE -i /path/to/upscayled/%08d.png -r FRAMERATE -vf format=yuv420p upscayled.mkv

With audio

ffmpeg -framerate FRAMERATE -i /path/to/upscayled/%08d.png -i /path/to/audio.mkv -r FRAMERATE -vf format=yuv420p upscayled.mkv

[NayamAmarshe]

That's awesome! Thanks for posting this!

[TGS963]

Thanks for posting such a detailed guide!

[Darin755]

This is surprisingly easy (not tried yet)

[jefflomax]

This is great - but I really wish we could pipe the processed results into ffmpeg
https://ffmpeg.org/ffmpeg-all.html#pipe
so we could re-scale without having to store all the frames.
Could be command line wizardry, addition to Upscayl, .NET Core or Python.

[aaronliu0130]

1. We're not introducing Python because it would introduce a ton of dependencies. We're not using .NET either as it's Windows-only.
2. We're a GUI for the CLI tool Real-ESRGAN-ncnn-vulkan. I'm pretty sure piping only works with CLI tools, so you can pipe with that

[NayamAmarshe]

1. We're not introducing Python because it would introduce a ton of dependencies. We're not using .NET either as it's Windows-only.
2. We're a GUI for the CLI tool Real-ESRGAN-ncnn-vulkan. I'm pretty sure piping only works with CLI tools, so you can pipe with that

Well, we're not Real-ESRGAN-ncnn-vulkan anymore :D We're Upscayl NCNN

[StephenCleary]

We're not using .NET either as it's Windows-only.

I'm not arguing you absolutely should use it, but this is a common misconception. .NET has officially supported Linux and macOS for more than 8 years at this point.

[boludoz]

With ffmpeg it is impossible, better use OpenCV with a small buffer so as not to saturate the memory that progressively writes the video in 4k and later adds the sound.

[aaronliu0130]

Why would it be impossible with FFMPEG?

[owen555999]

Did this work with gif?

[aaronliu0130]

Yes

[owen555999]

so what is the best format for gif file?

[aaronliu0130]

Just follow the gudie

[michaelnew]

I put together a quick and dirty bash script that essentially just does what's in the above guide. It tries to figure out the framerate and it puts all the frames into /tmp, so they'll eventually get cleaned up.

You can use it with upscale_video.sh coolvideo.mp4

To set it up:

1. Download and unzip the latest upscayl-ncnn release from here
2. Download the models from here and put them in a models directory alongside the upscayl-bin binary
3. Set UPSCAYL= in the first line of the bash script to the path for upscayl-bin

UPSCAYL=~/upscayl-bin-20240601-103425-linux/upscayl-bin
MODEL=RealESRGAN_General_WDN_x4_v3
# can use any model from upscayl-bin-20240601-103425-linux/models

INPUT=$1
BASE=$(basename $INPUT | cut -d. -f1)
FR=$(ffprobe $INPUT 2>&1 | grep -i "stream.*video" | awk -F'fps' '{print $1}' | awk '{print $NF}')

TD=$(mktemp -d /tmp/XXX)
mkdir $TD/frames
mkdir $TD/upscaled

ffmpeg -i $INPUT -r $FR $TD/frames/%08d.png
$UPSCAYL -i $TD/frames -o $TD/upscaled -n $MODEL
ffmpeg -framerate $FR -i $TD/upscaled/%08d.png -r $FR -vf format=yuv420p ${BASE}_upscaled.mkv

[hardBSDk]

@michaelnew Can you send this script in a PR for the upscayl-ncnn repository?

Also remove any hardcoded values, for the upscayl-ncnn executable you can use the * symbol on the name to automatically detect the executable present on the relative directory.

[aaronliu0130]

1. I don't see what the PR would add to. It's a C++ repo.
2. No values are hardcoded. Everything is specified with variables.

[Xelvoren]

Do you know how can we set a custom with resolution while using that method?

[hardBSDk]

@ivabus Can you send this tutorial to the repository README?

[aaronliu0130]

I've pinned this guide and added it to the wiki.

[duelmas]

we want sub folder batch upscale for clip frames but will give this a try....

[Turranx]

Upscayl 2.11.5 has a memory leak. This means that the process will crash after 1000 to 5000 images depending upon how much RAM is in your system. 5000 images may seem like a lot, but I used ffmpeg to decompose a movie into 210,000+ PNG images. Since manually running individual batches of 5K images is time consuming, I wrote this bash script.

• It moves 20 source files from 'ToBeProcessedLocation' into 'ProcessingLocation'
• Then it processes them with upscayl-bin
• The newly created up-scaled images are created in 'UpsizedLocation'
• The batch of 20 source files are moved out to 'ProcessedLocation'
• The process repeats itself until 'ToBeProcessedLocation' is empty

To use this code, you will need to...

1. Replace '<userID>' with your Linux user ID
2. Create the directories as shown in the first several lines of code
3. Create your own command line arguments for 'upscayl-bin'

To create my own command line arguments for 'upscayl-bin', I played with the Upscayle GUI and with images from the movie until I found results that I liked. Then I upsized a batch of 100 images through Upscayl (GUI). While the batch was processing, I opened a terminal and ran this command:

ps -eo args | grep upscayl-bin

The arguments were displayed on screen and I copied them into the script.

NOTE: I have very little experience writing bash scripts. However, I have decades of experience with PowerShell, T-SQL, and writing technical documentation, so please be kind in your criticisms.

#!/bin/bash

ToBeProcessedLocation='/home/<userID>/Downloads/Upscayl/Frames_to_be_Processed'
ProcessingLocation='/home/<userID>/Downloads/Upscayl/Frames'
UpsizedLocation='/home/<userID>/Downloads/Upscayl/Frames/upscayl_png_remacri_3x'
ProcessedLocation='/home/<userID>/Downloads/Upscayl/Frames_Processed'
BatchSize=20

# Identify Unprocessed Files To Be Moved In
FileBatch=( $(find $ToBeProcessedLocation -maxdepth 1 -type f | head -n $BatchSize) )


while [ ${#FileBatch[*]} -gt 0 ]
do

    # Move New Files In For Processing
    for file in "${FileBatch[@]}"; do
        echo "Moving in $file --> $ProcessingLocation"
        mv $file $ProcessingLocation
    done


    # Run Upscayl
    /opt/upscayl/bin/upscayl-bin -i $ProcessingLocation -o $UpsizedLocation -s 3 -m /opt/<userID>/models -n remacri -f png -c 0


    # Identify Processed Files To Be Moved Out
    FileBatch=( $(find $ProcessingLocation -maxdepth 1 -type f) )


    # Move Processed Files Out
    for file in "${FileBatch[@]}"; do
        echo "Moving out $file --> $ProcessedLocation"
        mv $file $ProcessedLocation
    done


    # Identify Unprocessed Files To Be Moved In
    FileBatch=( $(find $ToBeProcessedLocation -maxdepth 1 -type f | head -n $BatchSize) )
done

echo "< Finished >"

[NayamAmarshe]

Upscayl 2.11.5 has a memory leak. This means that the process will crash after 1000 to 5000 images depending upon how much RAM is in your system. 5000 images may seem like a lot, but I used ffmpeg to decompose a movie into 210,000+ PNG images. Since manually running individual batches of 5K images is time consuming, I wrote this bash script.

We really need to look into this issue but thank you for sharing your solution.

[DuckersMcQuack]

Thank you!

As i have done exactly this with batch upscaling images for just this, but joining it back together to a functional movie/episode, i could not find the ffmpeg commands for! :D

[ivabus]

It's in the last section of the guide

Join frames into video

Without audio

ffmpeg -framerate FRAMERATE -i /path/to/upscayled/%08d.png -r FRAMERATE -vf format=yuv420p upscayled.mkv

With audio

ffmpeg -framerate FRAMERATE -i /path/to/upscayled/%08d.png -i /path/to/audio.mkv -r FRAMERATE -vf format=yuv420p upscayled.mkv

[ljnath]

Does upscayl supports GPU multi threading similar to RealScaler ?