Bulk processing

Bulk processing (both matrix building and various postprocessings)

All the tasks described in Matrix postprocessing may be performed in a single run, generating the matrix (including various types of filtering: by per-sample counts, by counts above a threshold in specified number of samples, by giving a set of k-mers that should be keep exclusively) in various forms, performing normalization, counts postprocessing (including reducting dimensionality), correct p-values.

The exemplary command line is as follows:

./mkmc                                                         (1)
 -k 27 -f fa \                                                 (2)
--ci 2 --cx 1000000000 \                                       (3)
--thr-rat 0.5 --thr 3 \                                        (4)
--flt kmers.fasta \                                            (5)
 -o matrix fa \                                                (6)
 -n freq --save-n \                                            (7)
--corr pearson spearman kendall -p correlation_phenotype.txt \ (8)
--cv --leave 2 \                                               (9)
--diff anova dids snr ttest wrs -c de_phenotype.txt \          (10)
--dids-mode quadratic \                                        (11)
--pval-corr bh \                                               (12)
--max-corrected-pval 0.01 \                                    (13)
--entropy \                                                    (14)
--pca --umap \                                                 (15)
-- samples.txt results tmp                                     (16)

The consecutive lines mean as follows:

1. Run MKMC.

2. Count 27-mers; samples k-mers are extracted from FASTA files (see Building a matrix for non-FASTQ input files).

3. Exclude k-mers for a sample which ones within the sample have count less than 2 or more than $10^9$ (see Filtering k-mers along with building the matrix).

4. Exclude k-mers from the matrix which ones has no counts in a half of samples at least 3 (see Filtering k-mers along with building the matrix).

5. Keep only k-mers which are present in kmers.fasta file (see Filtering k-mers along with building the matrix).

6. Generate a matrix in the text form and place k-mers sequences also in FASTA (see Building a matrix in the user-readable form).

7. Normalize counts with frequency count method and save them to disk (see Normalizing counts).

8. Compute correlation (with 3 methods in a single run) between per-sample counts of every k-mer in the matrix with values in correlation_phenotype.txt file (see Computing correlation for further information about correlation and correlation_phenotype.txt example).

9. Perform cross-validation for every correlation (8) by iteratively treating 2 samples as test ones and computing correlation for another ones (see Generating matrix for cross-validation).

10. Perform differential k-mers analysis (with 5 methods in a single run) between per-sample counts of every k-mer in the matrix with values in de_phenotype.txt file (see Performing differential k-mers analysis for further information about diiferential k-mers analysis and de_phenotype.txt example).

11. Results of differential k-mers analysis with DIDS (10) will be squared instead of rooted by default (see Performing differential k-mers analysis).

12. Correct p-values of ANOVA, T-Test, and WRS with Benjamini-Hochberg method (see Correcting p-values).

13. Treat k-mers with p-value no higher than 0.01 as statistically significant (see Correcting p-values).

14. Compute entropy for k-mers counts (see Determining k-mers counts entropy).

15. Reduce the number of counts dimensions from $(\text{no. of k-mers})\times(\text{no. of samples})$ to $2\times(\text{no. of samples})$ with PCA and UMAP methods in a single run (see Dimensionality reduction).

16. Pass to MKMC the file with samples (samples.txt, see Building a matrix and filtering), an output file name template (results) and a temporary directory (tmp). Note the space after the two dashes.

Passing the parameter --reuse-db (see Building a matrix for later utilization) is also possible.