Memory consumption for corrected correlations

[TyberiusPrime]

So I'd expect that the output (and time) requirements scale with O(n^2)
for correlating all pairwise correlations.

A 10k*10k matrix of floats should, even if not sparse, be about 750 MB.

BigSurR seems to use at peak about 22 GiB for a 5k feature set
(and 17k cells) (all numbers vmPeak as reported by /proc//status).

At 10k features it peaked at 47.5 GiB.

At 15k features it got oom killed somewhere past 119 GiB, though this machine has 192 GiB + 32 swap.

For now this is just a documenting anecdote.
Might be worth profiling to enable larger datasets though,
I have a suspicion this could be better.

[hsuknowledge]

The extra memory use is likely due to tcrossprod in mcPCC. The actual operations are just inner products between rows. One can implement a pairwise algorithm and just loop over all possible pairs.

[hsuknowledge]

This comment was retracted.

[hsuknowledge]

(What I tried to do was avoid creating the dense ematrix by some rudimentary rowwise lapply code, which, as it turns out, was not the problem originial poster encountered.)

[hsuknowledge]

I wrote some new code in Julia in my repo. For now it only does feature selection and simple calculation of PCC′.