Participants: Ke Fan, Sidharth Kumar

PhD student Ke Fan’s research in profiling and visualizing performance data at scale is recognized with a best poster nomination at SC23. “Two-phase IO Enabling Large-scale Performance Introspection” focuses on developing an effective and interactive exploration of the profiles of large-scale parallel programs, which remain a challenge due to the high I/O overheads of profiles and the difficulties in scaling downstream visualization tools.

The poster presents Viveka, a lightweight end-to-end system for profiling and visualizing the performance of MPI-based applications.
- A simple data format for the generated logs that minimizes metadata, leading to a smaller storage footprint and faster load times. (on average 3x more lightweight than Caliper).
- Development of a custom two-phase data aggregation system to scale parallel I/O (of performance data) to high core counts, ensuring minimal overhead.
- A lightweight web-based visualization dashboard that is capable of performing interactive analysis of performance data collected at high process counts.
- A case study on real parallel applications to demonstrate the efficacy of our profiling and visualization system.

Funding in part by NSF RII Track-4 award 2132013, NSF PPoSS planning award 2217036, NSF PPoSS large award 2316157 and, NSF collaborative research award 2221811. Compute hours provided on Theta Supercomputer Argonne Leadership Computing Facility’s Directors Discretionary (DD) program.

Date: November 12, 2023 - November 17, 2023

Document: Two-phase IO Enabling Large-scale Performance Introspection

Related Entries

Directory:

• Ke Fan
• Sidharth Kumar

Papers:

• Two-phase IO Enabling Large-scale Performance Introspection (poster)

Related Categories

• Applications
• Data Science
• High Performance Computing
• Software
• Supercomputing
• Visualization