Implementation of the Spark technique in a matrix distributed computing algorithm
- Ying Wang(corresponding author),
- Tianjin Maritime College,
- Trakya University,
- University of Fujairah
Open access
Abstract
Two analyzes of Spark engine performance strategies to implement the Spark technique in a matrix distributed computational algorithm, the multiplication of a sparse multiplication operational test model. The dimensions of the two input sparse matrices have been fixed to 30,000 × 30,000, and the density of the input matrix have been changed. The experimental results show that when the density reaches about 0.3, the original dense matrix multiplication performance can outperform the sparse-sparse matrix multiplication, which is basically consistent with the relationship between the sparse matrix multiplication implementation in the single-machine sparse matrix test and the computational performance of the local native library. When the density of the fixed sparse matrix is 0.01, the distributed density-sparse matrix multiplication outperforms the same sparsity but uses the density matrix storage, and the acceleration ratio increases from 1.88× to 5.71× with the increase in dimension. The overall performance of distributed operations is improved.
Research Output Information
Output type
Original language
EnglishPages from-to (Number of pages)
Pages 660-671 (12 pages)Journal (Volume, Issue Number)
Journal of Intelligent Systems (Volume 31, Issue 1)Publication milestones
- Published - 01/01/2022
Publication status
ISSN
0334-1860Publication IDs
- Scopus: 85132334976
