OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

How to process video frames faster in nodejs using ffmpeg

  • Thread starter Thread starter Aviato
  • Start date Start date
A

Aviato

Guest
I am working on a video editing application in which user can upload a video and I want to perform some frame manipulation on top of the video, and thus I want to extract the video frames and draw them over a canvas api in nodejs and edit them accordingly. I used to do video manipulations using opencv and there was a function cv2.VideoCapture(inputFilePath) which would read video frames one at a time and we can perform our manipulations on the frame. The processing was very fast and I had tried processing big video files and it works really fast.

However, I was trying to replicate the same process in nodejs and thought about using ffmpeg to do that, now instead of loading all the frames first in the memory and loop through them, I wanted a way to load each frame one at a time and be able to process them fast, so after researching a lot on the internet, I came up with this solution :-

Code:
const fs = require('fs');
const { spawn } = require('child_process');

const width = 1920;
const height = 1080;
const bytesPerPixel = 3;
const frameSize = width * height * bytesPerPixel;

const spawnProcess = spawn('ffmpeg', [
    '-i', 'test.mp4',
    '-vcodec', 'rawvideo',
    '-pix_fmt', 'rgb24',
    '-f', 'rawvideo',
    '-s', `${width}x${height}`, // size of one frame
    'pipe:1'
]);

let buffer = Buffer.alloc(0);
let frameCount = 0;

spawnProcess.stdout.on('data', (data) => {
    buffer = Buffer.concat([buffer, data]);

    while (buffer.length >= frameSize) {
        const frameBuffer = buffer.slice(0, frameSize);
        buffer = buffer.slice(frameSize);

        const outputPath = `frame-${frameCount++}.png`;
        saveFrameAsPNG(frameBuffer, width, height, outputPath);
    }
});

function saveFrameAsPNG(frameBuffer, width, height, outputPath) {
    const { PNG } = require('pngjs');

    const png = new PNG({
        width: width,
        height: height
    });

    // Copy the frame buffer to the PNG data
    for (let y = 0; y < height; y++) {
        for (let x = 0; x < width; x++) {
            const idx = (y * width + x) * 3;
            const pngIdx = (y * width + x) * 4;

            png.data[pngIdx] = frameBuffer[idx];     // Red
            png.data[pngIdx + 1] = frameBuffer[idx + 1]; // Green
            png.data[pngIdx + 2] = frameBuffer[idx + 2]; // Blue
            png.data[pngIdx + 3] = 255;              // Alpha
        }
    }

    png.pack().pipe(fs.createWriteStream(outputPath));
}

spawnProcess.stderr.on('data', (data) => {
    console.error(`stderr: ${data}`);
});

spawnProcess.on('close', (code) => {
    console.log(`child process exited with code ${code}`);
});

I first input the video in the ffmpeg running in the child processes and stdout the buffer data and used a function to determine the value of one frame and extract that one frame and save that in the filesystem. However, the processing of the frames is extremely slow, it takes about 1 minute to process a 5 second long video with 60fps which is very inefficient as compared to python opencv

I am really confused between which approach should I continue with :-

  1. I could figure out a way to process the video using python opencv and pipe the frames buffer / save them individually in the filesystem and the nodejs would watch for any file save in the directory and keep processing them and deleting them simultaneously
  2. Or I could keep trying to find a way to make the read frame processing faster in the nodejs

Any help with code would be greatly appreciated :)
<p>I am working on a video editing application in which user can upload a video and I want to perform some frame manipulation on top of the video, and thus I want to extract the video frames and draw them over a canvas api in nodejs and edit them accordingly. I used to do video manipulations using <code>opencv</code> and there was a function <code>cv2.VideoCapture(inputFilePath)</code> which would read video frames one at a time and we can perform our manipulations on the frame. The processing was very fast and I had tried processing big video files and it works really fast.</p>
<p>However, I was trying to replicate the same process in nodejs and thought about using ffmpeg to do that, now instead of loading all the frames first in the memory and loop through them, I wanted a way to load each frame one at a time and be able to process them fast, so after researching a lot on the internet, I came up with this solution :-</p>
<pre class="lang-js prettyprint-override"><code>const fs = require('fs');
const { spawn } = require('child_process');

const width = 1920;
const height = 1080;
const bytesPerPixel = 3;
const frameSize = width * height * bytesPerPixel;

const spawnProcess = spawn('ffmpeg', [
'-i', 'test.mp4',
'-vcodec', 'rawvideo',
'-pix_fmt', 'rgb24',
'-f', 'rawvideo',
'-s', `${width}x${height}`, // size of one frame
'pipe:1'
]);

let buffer = Buffer.alloc(0);
let frameCount = 0;

spawnProcess.stdout.on('data', (data) => {
buffer = Buffer.concat([buffer, data]);

while (buffer.length >= frameSize) {
const frameBuffer = buffer.slice(0, frameSize);
buffer = buffer.slice(frameSize);

const outputPath = `frame-${frameCount++}.png`;
saveFrameAsPNG(frameBuffer, width, height, outputPath);
}
});

function saveFrameAsPNG(frameBuffer, width, height, outputPath) {
const { PNG } = require('pngjs');

const png = new PNG({
width: width,
height: height
});

// Copy the frame buffer to the PNG data
for (let y = 0; y < height; y++) {
for (let x = 0; x < width; x++) {
const idx = (y * width + x) * 3;
const pngIdx = (y * width + x) * 4;

png.data[pngIdx] = frameBuffer[idx]; // Red
png.data[pngIdx + 1] = frameBuffer[idx + 1]; // Green
png.data[pngIdx + 2] = frameBuffer[idx + 2]; // Blue
png.data[pngIdx + 3] = 255; // Alpha
}
}

png.pack().pipe(fs.createWriteStream(outputPath));
}

spawnProcess.stderr.on('data', (data) => {
console.error(`stderr: ${data}`);
});

spawnProcess.on('close', (code) => {
console.log(`child process exited with code ${code}`);
});

</code></pre>
<p>I first input the video in the <code>ffmpeg</code> running in the child processes and <code>stdout</code> the buffer data and used a function to determine the value of one frame and extract that one frame and save that in the filesystem.
However, the processing of the frames is extremely slow, it takes about 1 minute to process a 5 second long video with 60fps which is very inefficient as compared to python <code>opencv</code></p>
<p>I am really confused between which approach should I continue with :-</p>
<ol>
<li>I could figure out a way to process the video using python opencv and pipe the frames buffer / save them individually in the filesystem and the nodejs would watch for any file save in the directory and keep processing them and deleting them simultaneously</li>
<li>Or I could keep trying to find a way to make the read frame processing faster in the nodejs</li>
</ol>
<p>Any help with code would be greatly appreciated :)</p>
 
Top