javascript - Optimize comparing huge amounts of data with itself in NodeJS -
i have big multidimensional object in nodejs
, 50 mb worth of json
. contains variations of biological data. think can sufficiently simplify so:
{ lads : { // lad lad4515643 : { brains : { // brain brain1256251 : { var01 : 'lala', var02 : 'jaja', var99 : 'haha', }, // brain brain3567432 : {}, brain4867321 : {}, brain5145621 : {} // etc }, var01 : 'foo', var02 : 'bar', var99 : 'baz' }, // lad lad4555672 : {}, lad5625627 : {}, lad7457255 : {} // etc } }
i need compare combinations of lads
brains
lads
brains
see ones "better", in order make kind of hierarchy. parent lads
keys weigh in on brains
comparison.
i figured, using iterations on objects using references can assign ids of better ones. take quick glance on code (comments) , see mean:
// iterate on lads (var ladid in obj.lads) { if (obj.lads.hasownproperty(ladid)) { var lad = obj.lads[ladid]; // iterate on brains (var brainid in lad.brains) { if (lad.brains.hasownproperty(brainid)) { var brain = lad.brains[brainid]; // iterate on lads again (var lad2id in obj.lads) { if (obj.lads.hasownproperty(lad2id)) { var lad2 = obj.lads[lad2id]; // iterate on lads' brains (var brain2id in lad2.brains) { if (lad2.brains.hasownproperty(brain2id)) { var brain2 = lad2.brains[brain2id]; // 1 lad+brain combination var drone1 = { lad : lad, brain : brain }; // lad+brain combination var drone2 = { lad : lad2, brain : brain2 ladid : lad2id, // required store reference if better brainid : brain2id // required store reference if better }; // comparison unless comparing ourselves if (brain != brain2) { // objects passed reference, convenient: judge(drone1, drone2); } } } } } } } } } // judge better function judge(drone1, drone2) { // magic compares lad+brain combos if (magic) { // add list of better versions drone1.brain.better = drone1.brain.better || []; // passed reference - can modify original brain object directly drone1.brain.better.push({ ladid : drone2.ladid, brainid : drone2.brainid }); } }
now of course, number of iterations increases exponentially when dataset increases. 3000 brains
in total, there 9 million iterations, magic adds more 10 seconds of execution time.
what optimizations (hugely) beneficial in scenario this, apart using multiple threads?
since judge()
purely math, make difference if convert every single step of iteration callback style? (in imagination, create huge overhead of anonymous functions , memory usage.)
Comments
Post a Comment