GridTorrent framework: A High-performance data transfer and data sharing framework for scientific computing

Abstract

Large amount of data that is often stored in many thousands of files is created as part of today’s geographically distributed scientific computation and collaboration environments. Managing and transferring large volumes of data sets present a significant challenge and are often a bottleneck in the scientific computing community. In this paper, we introduce an architecture to manage data distributions in a collaborative fashion through a GridTorrent Framework (GTF) whose data transfer mechanism inspired by Bittorrent. We present performance experiment data that compares our framework to parallel TCP (PTCP) and Bittorrent. Experimental results conducted suggest that using GridTorrent for large data set has significant advantages over parallel TCP in LAN and WAN type of computer networks.