Performance evaluation of compression techniques on scientific dataset

Abstract

Data communication is vital, as the world is getting smaller with the help of Internet. The challenge to improve quality and responsiveness of communication is in the network bandwidth bottleneck. However, with compression technologies, the impact of transferring data can be optimized. There are various compression technologies in the market from different origins both available commercially and public domain. Performance in compression technologies are measured according to required computation power and compression ratios achieved. However, not all data can be compressed effectively, where desired compression rates are achieved. The reason is that most data are obtained from digitizing or converted from analog signals. Examples: audio, photos, graphs plotted by input sensors. An important characteristic of data compression lies in the compression ratio and compression speed of a particular data compression tool. Though most theoretical background of compression tool compresses datasets based on Lempel-ziv’s algorithm, in reality, these tools varied when it comes to compressing a binary file to a text file or a graphical one. This is evidence in the statistically analysis of the file format. This project looks into various data compression technique and when to use them, with specifically focus on evaluating the performance of existing data compression and extraction algorithms that best suit scientific dataset. This project applies various tests on selected range of scientific datasets to ascertain the overall performance against a benchmarking compression technique. The tests are based on a real time network transmission of compression and extraction on a set of scientific datasets over a networked environment. This project proves that a generic compression algorithm fair better compare to a more format specific compression algorithm when use on a scientific datasets. The outcome and procedures used in this project use as a template for choosing a suitable compression tool for any particular format of dataset. This template shall minimise any doubt and confusion of choosing and using a compression techniques.