OiO.lk Community platform!

Oio.lk is an excellent forum for developers, providing a wide range of resources, discussions, and support for those in the developer community. Join oio.lk today to connect with like-minded professionals, share insights, and stay updated on the latest trends and technologies in the development field.
  You need to log in or register to access the solved answers to this problem.
  • You have reached the maximum number of guest views allowed
  • Please register below to remove this limitation

How to further improve Node.js server application performance

  • Thread starter Thread starter Schtraded
  • Start date Start date
S

Schtraded

Guest
My Question:

  • What can I do to gain performance for my worker part, it doesn't necessarily need to be interpreted in my code but just in general?
  • Would choosing a different language be in my situation better?

If anything is unclear or you have questions please ask, I will try my best to answer!

Currently I'm working on a Project where my main calculation is very CPU heavy. Therefore I'm using workers which do those calculation, which have reduced my calculation time by a lot, but it still takes a long time. For 8 iteration it took a total of around 30sec, if I want to increase the iteration the total time will increase by so much that it takes hours. Of course I could just let it be but if I made a mistake in my code I would need to do the whole calculation all over.

I know that in web workers using I think they're called transferable Object, like array buffers and such reduces the time, but if I remember correctly worker threads use shared memory. Other than that I don't know much to increase performance, besides avoiding copying large arrays in general. I tried slicing my arrays in my worker to be smaller but that just increased the total time.

At the bottom you can find a Web Debugger overview of my code running.

In the following is my console log of my node.js file.

Code:
    Debugger attached.
    Worker took 3138.933 milliseconds.
    Current Average Worker took 3138.933 milliseconds.
    Expected Total Worker time 25111.464 milliseconds.
    -----------------------------------------------
    Worker took 3273.5117 milliseconds.
    Current Average Worker took 3206.22235 milliseconds.
    Expected Total Worker time 25649.7788 milliseconds.
    -----------------------------------------------
    Worker took 3406.0996000000014 milliseconds.
    Current Average Worker took 3272.8481000000006 milliseconds.
    Expected Total Worker time 26182.784800000005 milliseconds.
    -----------------------------------------------
    Worker took 3417.6726 milliseconds.
    Current Average Worker took 3309.0542250000003 milliseconds.
    Expected Total Worker time 26472.433800000003 milliseconds.
    -----------------------------------------------
    Worker took 3152.6977000000006 milliseconds.
    Current Average Worker took 3277.7829200000006 milliseconds.
    Expected Total Worker time 26222.263360000004 milliseconds.
    -----------------------------------------------
    Worker took 3198.689699999999 milliseconds.
    Current Average Worker took 3264.6007166666673 milliseconds.
    Expected Total Worker time 26116.805733333338 milliseconds.
    -----------------------------------------------
    Worker took 3614.2327000000005 milliseconds.
    Current Average Worker took 3314.5481428571434 milliseconds.
    Expected Total Worker time 26516.385142857147 milliseconds.
    -----------------------------------------------
    Worker took 3366.2328000000016 milliseconds.
    Current Average Worker took 3321.0087250000006 milliseconds.
    Expected Total Worker time 26568.069800000005 milliseconds.
    -----------------------------------------------
    -----------------------------------------------
    Data has been written to the file successfully!
    -----------------------------------------------
    Waiting for the debugger to disconnect...

The following is my Node.js main thread which fetches data from a file (around 300mb large), then creates worker or rather the function ChunkConversionCompleted and the function functionTriggerBuffer is for looping.

Code:
const fs = require('fs');
const {Worker} = require('worker_threads');
const chalk = require('chalk')

fs.readFile('./static/data/CONVERTED_compact.csv', 'utf8', (err, data) => {
    if (err) {
        console.error('Error reading file:', err);
        return;
    }
    const klineDataArray = JSON.parse(data);// very large dataset
    let j = 3;//9   
    let cal_on_every_tick = false;
    let multi = 0.4;//2.2
    let AllCalcComplete = false;
    const numWorkers = 10;//navigator.hardwareConcurrency || 4;
    BigChunk = (+80000 * +numWorkers).toFixed(0);

    const chunkSize = Math.ceil(klineDataArray.length / numWorkers);
    let chunk = [];
    ChunkConversionCompleted(klineDataArray, j, cal_on_every_tick, multi, AllCalcComplete);

    //to Iterate over Values - because hundreds of workers would spawn otherwise
    function functionTriggerBuffer(klineDataArray, j, cal_on_every_tick, multi, AllCalcComplete) {
        let j_max = 4;//9
        let multi_max = 0.6;//2.2
        if (multi === multi_max) {
            if (j === j_max && !cal_on_every_tick) {
                j = 2;//8 (9(changer)-1)
                cal_on_every_tick = true;
            };

            j++;
            multi = 0.2;//2.1 (2.2(multi)-0.1) multi 2.2
        };
        
        multi = +((+multi + 0.2).toFixed(1))

        if (j === j_max && multi === multi_max && cal_on_every_tick) {
            AllCalcComplete = true;
        };

        if (j <= j_max) {
            ChunkConversionCompleted(klineDataArray, j, cal_on_every_tick, multi, AllCalcComplete)
        };
    };
    
    async function ChunkConversionCompleted(klineDataArray, j, cal_on_every_tick, multi, AllCalcComplete) {
        const t0 = performance.now();
        for (let i = 0; i < numWorkers; i++) {
            const start = i * chunkSize;
            const end = start + chunkSize;
            chunk = klineDataArray.slice(start, end);
            const adjustedStart = findStartLine(start, klineDataArray, j); // Adjust start to the correct timestamp
            preparationChunk = klineDataArray.slice(adjustedStart, start);
            conversionChunk = klineDataArray.slice(start, end);
            const worker = new Worker('./static/z_NodeServer/worker_NodeServer.js', {
                workerData: { chunk: chunk, index: i, preparationChunk: preparationChunk, changer: j, cal_on_every_tick: cal_on_every_tick, multi: multi }
            });

            worker.on('message', (message) => {
                const {index, calculateKline, counter, preparationCounter} = message;
                Arrays[index] = calculateKline.Array;

                // Counter
                counters[index] = counter;
                preparationCounters[index] = preparationCounter;

                // Terminate worker to free up memory?
                worker.terminate();
            
                completed2++;
            
                if (completed2 === numWorkers) {
                    // All workers are done, combine results
                    const Array = Arrays.flat();
                    const combinedCounter = counters.flat();
                    const combinedPreparationCounter = preparationCounters.flat();

                    console.log(combinedCounter); //for debugging, not needed anymore
                    console.log(combinedPreparationCounter); //for debugging, not needed anymore
                    const correctedArray = CorrectionsOfArrays(Array)
                    completed2 = 0;
                    totalCounter++;
                    AllC_CompleteArray = Overview(correctedArray, j, cal_on_every_tick, multi); 
                
                    AllC_CompleteArray.sort((a, b) => {
                        return a.anp - b.anp;
                    });

                    const j_diff = 4 - 3 + 1; //j_max - j
                    const multi_diff = +((0.6 - 0.4)/0.2).toFixed(0) + +1; //multi_max - multi
                    const totalWorkerCount = j_diff * multi_diff * 2;

                    const t1 = performance.now();
                    WorkerTimerArray.push(t1 - t0);
                    const sum = WorkerTimerArray.reduce((acc, val) => acc + val, 0);
                    AvgTime = sum / WorkerTimerArray.length;
                    const expectedTotalTime = totalWorkerCount * AvgTime

                    console.log(chalk.blueBright(`Worker took ${t1 - t0} milliseconds.`));
                    console.log(chalk.blueBright(`Current Average Worker took ${AvgTime} milliseconds.`));
                    console.log(chalk.blueBright(`Expected Total Worker time ${expectedTotalTime} milliseconds.`));

                    if (AllCalcComplete) {
                        const filePath = './static/data/Data/AllC_CompleteArray.csv';
                        const jsonArray_AllC_CompleteArray = JSON.stringify(AllC_CompleteArray);
                        // Write to the file
                        fs.writeFile(filePath, jsonArray_AllC_CompleteArray, (err) => {
                            if (err) {
                                console.error('Error writing to file:', err);
                                return;
                            }
                            console.log('-----------------------------------------------');
                            console.log(chalk.greenBright('Data has been written to the file successfully!'));
                            console.log('-----------------------------------------------');
                        });
                    }
                    console.log('-----------------------------------------------');
                    functionTriggerBuffer(klineDataArray, j, cal_on_every_tick, multi, AllCalcComplete)
                };
            });
        };
    };
},
);

This is my worker file, the actual calculation compares each line and creates something with the data. The following code block only has a very small amount of my actual code, since putting the whole calculation into it would only make it more confusing to read and understand. The Array which is being returned is the result of the calculation which I removed. It`s at most 100 elements long per worker

Code:
//-- DECLARATIONS --//
//ARRAYS AND SUCH
let MessageArray = [];
let LastCandlestickArray = [];
let closeValuesArray = [];
let BalanceArray = [];
let CandleSeriesArray = [];
let CloseValueSeriesArray = [];
let Balance = 20
BalanceArray.push(Balance)

//-- CALCULATION --//
function createCandleSeries(candlestick) {
    return {
        time: candlestick.t / 1000,
        open: candlestick.o,
        high: candlestick.h,
        low: candlestick.l,
        close: candlestick.c
};};

function createCloseValueSeries(LastCandlestick, CloseValue) {
    return {
        time: LastCandlestick.t / 1000,
        value: CloseValue
};};


function processCandleData(ActualTimestamp, message, ChunkProcessing, changer, cal_on_every_tick, multi) {
    var candlestick = message.k
    if (ChunkProcessing === true) { // === true not needed but it's there for clarity
        const CandleSeries = createCandleSeries(candlestick);
        CandleSeriesArray.push(CandleSeries);
    }
    //return processedCandleData = processCandleData(klineData)


    //CLOSEVALUE OF CANDLESTICK
    MessageArray.push(candlestick)
    let PreviousCandlestick = undefined;
    if (MessageArray.length >= 2) {
        PreviousCandlestick = MessageArray[MessageArray.length - 2]
    };
    if (PreviousCandlestick && PreviousCandlestick.T < candlestick.t) { //PreviousCandlestick.T < candlestick.t
        LastCandlestickArray.push(PreviousCandlestick)
        LastCandlestick = LastCandlestickArray[LastCandlestickArray.length - 1];
        CloseValue = LastCandlestick.c;
        closeValuesArray.push(CloseValue);
        if (ChunkProcessing === true) { // === true not needed but it's there for clarity
            const CloseValueSeries = createCloseValueSeries(LastCandlestick, CloseValue);
            CloseValueSeriesArray.push(CloseValueSeries);
        }
    };


    return {
        Array,
    };
}

let counter2 = 0;
let preparationCounter = 0;

const { parentPort, workerData } = require('node:worker_threads');
const { chunk, index, preparationChunk, changer, cal_on_every_tick, multi } = workerData;
    let calculateKline;
    preparationChunk.forEach(line => {
        //var message = line;//JSON.parse(line);
        if (line.m === false) {
            processCandleData(line.E, line, false, changer ,cal_on_every_tick, multi)
        };
        preparationCounter++;
    });
    // Iterate over each line and parse as JSON
    chunk.forEach(line => {
        //var message = line;//JSON.parse(line);
        if (line.m === false) {
            calculateKline = processCandleData(line.E, line, true, changer, cal_on_every_tick, multi)
        };
        counter2++;
    });
    parentPort.postMessage({ index: index, calculateKline: calculateKline, counter: counter2, preparationCounter: preparationCounter });

web debugger [
web debugger
]

<p><strong>My Question:</strong></p>
<ul>
<li><strong>What can I do to gain performance for my worker part, it doesn't necessarily need to be interpreted in my code but just in general?</strong></li>
<li><strong>Would choosing a different language be in my situation better?</strong></li>
</ul>
<p>If anything is unclear or you have questions please ask, I will try my best to answer!</p>
<p>Currently I'm working on a Project where my main calculation is very CPU heavy. Therefore I'm using workers which do those calculation, which have reduced my calculation time by a lot, but it still takes a long time. For 8 iteration it took a total of around 30sec, if I want to increase the iteration the total time will increase by so much that it takes hours. Of course I could just let it be but if I made a mistake in my code I would need to do the whole calculation all over.</p>
<p>I know that in web workers using I think they're called transferable Object, like array buffers and such reduces the time, but if I remember correctly worker threads use shared memory. Other than that I don't know much to increase performance, besides avoiding copying large arrays in general. I tried slicing my arrays in my worker to be smaller but that just increased the total time.</p>
<p>At the bottom you can find a Web Debugger overview of my code running.</p>
<p>In the following is my console log of my node.js file.</p>
<pre><code>
Debugger attached.
Worker took 3138.933 milliseconds.
Current Average Worker took 3138.933 milliseconds.
Expected Total Worker time 25111.464 milliseconds.
-----------------------------------------------
Worker took 3273.5117 milliseconds.
Current Average Worker took 3206.22235 milliseconds.
Expected Total Worker time 25649.7788 milliseconds.
-----------------------------------------------
Worker took 3406.0996000000014 milliseconds.
Current Average Worker took 3272.8481000000006 milliseconds.
Expected Total Worker time 26182.784800000005 milliseconds.
-----------------------------------------------
Worker took 3417.6726 milliseconds.
Current Average Worker took 3309.0542250000003 milliseconds.
Expected Total Worker time 26472.433800000003 milliseconds.
-----------------------------------------------
Worker took 3152.6977000000006 milliseconds.
Current Average Worker took 3277.7829200000006 milliseconds.
Expected Total Worker time 26222.263360000004 milliseconds.
-----------------------------------------------
Worker took 3198.689699999999 milliseconds.
Current Average Worker took 3264.6007166666673 milliseconds.
Expected Total Worker time 26116.805733333338 milliseconds.
-----------------------------------------------
Worker took 3614.2327000000005 milliseconds.
Current Average Worker took 3314.5481428571434 milliseconds.
Expected Total Worker time 26516.385142857147 milliseconds.
-----------------------------------------------
Worker took 3366.2328000000016 milliseconds.
Current Average Worker took 3321.0087250000006 milliseconds.
Expected Total Worker time 26568.069800000005 milliseconds.
-----------------------------------------------
-----------------------------------------------
Data has been written to the file successfully!
-----------------------------------------------
Waiting for the debugger to disconnect...
</code></pre>
<p>The following is my Node.js main thread which fetches data from a file (around 300mb large), then creates worker or rather the function ChunkConversionCompleted and the function functionTriggerBuffer is for looping.</p>
<pre><code>
const fs = require('fs');
const {Worker} = require('worker_threads');
const chalk = require('chalk')

fs.readFile('./static/data/CONVERTED_compact.csv', 'utf8', (err, data) => {
if (err) {
console.error('Error reading file:', err);
return;
}
const klineDataArray = JSON.parse(data);// very large dataset
let j = 3;//9
let cal_on_every_tick = false;
let multi = 0.4;//2.2
let AllCalcComplete = false;
const numWorkers = 10;//navigator.hardwareConcurrency || 4;
BigChunk = (+80000 * +numWorkers).toFixed(0);

const chunkSize = Math.ceil(klineDataArray.length / numWorkers);
let chunk = [];
ChunkConversionCompleted(klineDataArray, j, cal_on_every_tick, multi, AllCalcComplete);

//to Iterate over Values - because hundreds of workers would spawn otherwise
function functionTriggerBuffer(klineDataArray, j, cal_on_every_tick, multi, AllCalcComplete) {
let j_max = 4;//9
let multi_max = 0.6;//2.2
if (multi === multi_max) {
if (j === j_max && !cal_on_every_tick) {
j = 2;//8 (9(changer)-1)
cal_on_every_tick = true;
};

j++;
multi = 0.2;//2.1 (2.2(multi)-0.1) multi 2.2
};

multi = +((+multi + 0.2).toFixed(1))

if (j === j_max && multi === multi_max && cal_on_every_tick) {
AllCalcComplete = true;
};

if (j <= j_max) {
ChunkConversionCompleted(klineDataArray, j, cal_on_every_tick, multi, AllCalcComplete)
};
};

async function ChunkConversionCompleted(klineDataArray, j, cal_on_every_tick, multi, AllCalcComplete) {
const t0 = performance.now();
for (let i = 0; i < numWorkers; i++) {
const start = i * chunkSize;
const end = start + chunkSize;
chunk = klineDataArray.slice(start, end);
const adjustedStart = findStartLine(start, klineDataArray, j); // Adjust start to the correct timestamp
preparationChunk = klineDataArray.slice(adjustedStart, start);
conversionChunk = klineDataArray.slice(start, end);
const worker = new Worker('./static/z_NodeServer/worker_NodeServer.js', {
workerData: { chunk: chunk, index: i, preparationChunk: preparationChunk, changer: j, cal_on_every_tick: cal_on_every_tick, multi: multi }
});

worker.on('message', (message) => {
const {index, calculateKline, counter, preparationCounter} = message;
Arrays[index] = calculateKline.Array;

// Counter
counters[index] = counter;
preparationCounters[index] = preparationCounter;

// Terminate worker to free up memory?
worker.terminate();

completed2++;

if (completed2 === numWorkers) {
// All workers are done, combine results
const Array = Arrays.flat();
const combinedCounter = counters.flat();
const combinedPreparationCounter = preparationCounters.flat();

console.log(combinedCounter); //for debugging, not needed anymore
console.log(combinedPreparationCounter); //for debugging, not needed anymore
const correctedArray = CorrectionsOfArrays(Array)
completed2 = 0;
totalCounter++;
AllC_CompleteArray = Overview(correctedArray, j, cal_on_every_tick, multi);

AllC_CompleteArray.sort((a, b) => {
return a.anp - b.anp;
});

const j_diff = 4 - 3 + 1; //j_max - j
const multi_diff = +((0.6 - 0.4)/0.2).toFixed(0) + +1; //multi_max - multi
const totalWorkerCount = j_diff * multi_diff * 2;

const t1 = performance.now();
WorkerTimerArray.push(t1 - t0);
const sum = WorkerTimerArray.reduce((acc, val) => acc + val, 0);
AvgTime = sum / WorkerTimerArray.length;
const expectedTotalTime = totalWorkerCount * AvgTime

console.log(chalk.blueBright(`Worker took ${t1 - t0} milliseconds.`));
console.log(chalk.blueBright(`Current Average Worker took ${AvgTime} milliseconds.`));
console.log(chalk.blueBright(`Expected Total Worker time ${expectedTotalTime} milliseconds.`));

if (AllCalcComplete) {
const filePath = './static/data/Data/AllC_CompleteArray.csv';
const jsonArray_AllC_CompleteArray = JSON.stringify(AllC_CompleteArray);
// Write to the file
fs.writeFile(filePath, jsonArray_AllC_CompleteArray, (err) => {
if (err) {
console.error('Error writing to file:', err);
return;
}
console.log('-----------------------------------------------');
console.log(chalk.greenBright('Data has been written to the file successfully!'));
console.log('-----------------------------------------------');
});
}
console.log('-----------------------------------------------');
functionTriggerBuffer(klineDataArray, j, cal_on_every_tick, multi, AllCalcComplete)
};
});
};
};
},
);
</code></pre>
<p>This is my worker file, the actual calculation compares each line and creates something with the data. The following code block only has a very small amount of my actual code, since putting the whole calculation into it would only make it more confusing to read and understand. The Array which is being returned is the result of the calculation which I removed. It`s at most 100 elements long per worker</p>
<pre><code>//-- DECLARATIONS --//
//ARRAYS AND SUCH
let MessageArray = [];
let LastCandlestickArray = [];
let closeValuesArray = [];
let BalanceArray = [];
let CandleSeriesArray = [];
let CloseValueSeriesArray = [];
let Balance = 20
BalanceArray.push(Balance)

//-- CALCULATION --//
function createCandleSeries(candlestick) {
return {
time: candlestick.t / 1000,
open: candlestick.o,
high: candlestick.h,
low: candlestick.l,
close: candlestick.c
};};

function createCloseValueSeries(LastCandlestick, CloseValue) {
return {
time: LastCandlestick.t / 1000,
value: CloseValue
};};


function processCandleData(ActualTimestamp, message, ChunkProcessing, changer, cal_on_every_tick, multi) {
var candlestick = message.k
if (ChunkProcessing === true) { // === true not needed but it's there for clarity
const CandleSeries = createCandleSeries(candlestick);
CandleSeriesArray.push(CandleSeries);
}
//return processedCandleData = processCandleData(klineData)


//CLOSEVALUE OF CANDLESTICK
MessageArray.push(candlestick)
let PreviousCandlestick = undefined;
if (MessageArray.length >= 2) {
PreviousCandlestick = MessageArray[MessageArray.length - 2]
};
if (PreviousCandlestick && PreviousCandlestick.T < candlestick.t) { //PreviousCandlestick.T < candlestick.t
LastCandlestickArray.push(PreviousCandlestick)
LastCandlestick = LastCandlestickArray[LastCandlestickArray.length - 1];
CloseValue = LastCandlestick.c;
closeValuesArray.push(CloseValue);
if (ChunkProcessing === true) { // === true not needed but it's there for clarity
const CloseValueSeries = createCloseValueSeries(LastCandlestick, CloseValue);
CloseValueSeriesArray.push(CloseValueSeries);
}
};


return {
Array,
};
}

let counter2 = 0;
let preparationCounter = 0;

const { parentPort, workerData } = require('node:worker_threads');
const { chunk, index, preparationChunk, changer, cal_on_every_tick, multi } = workerData;
let calculateKline;
preparationChunk.forEach(line => {
//var message = line;//JSON.parse(line);
if (line.m === false) {
processCandleData(line.E, line, false, changer ,cal_on_every_tick, multi)
};
preparationCounter++;
});
// Iterate over each line and parse as JSON
chunk.forEach(line => {
//var message = line;//JSON.parse(line);
if (line.m === false) {
calculateKline = processCandleData(line.E, line, true, changer, cal_on_every_tick, multi)
};
counter2++;
});
parentPort.postMessage({ index: index, calculateKline: calculateKline, counter: counter2, preparationCounter: preparationCounter });


</code></pre>
<p>web debugger
[<img src="https://i.sstatic.net/eAJniQKv.png" alt="web debugger" />]</p>
 

Latest posts

Top