जर्नल ऑफ़ प्रोटियोमिक्स और बायोइनफार्मेटिक्स
खुला एक्सेस

अमूर्त

SasCsvToolkit: A Versatile Parallel `Bag-of-Tasks` Job Submission Application on Heterogeneous and Homogeneous Platforms for Big Data Analytics such as for Biomedical Informatics

Abhishek Narain Singh*

Background: The need for big data analysis requires being able to process large data which are being held fine-tuned for usage by corporates. It is only very recently that the need for big data has caught attention for low budget corporate groups and academia who typically do not have money and resources to buy expensive licenses of big data analysis platforms such as SAS. The corporates con-tinue to work on SAS data format largely because of systemic organizational his-tory and that the prior codes have been built on them. The data-providers continue to thus provide data in SAS formats. Acute sudden need has arisen because of this gap of data being in SAS format and the coders not having a SAS expertise or training background as the economic and inertial forces acting of having shaped these two classes of people have been different.

Methods: We analyze the differences and thus the need for SasCsvToolkit which helps to generate a CSV file for a SAS format data so that the data scientist can then make use of his skills in other tools that can process CSVs such as R, SPSS, or even Microsoft Excel. At the same time, it also provides conversion of CSV files to SAS format. Apart from this, a SAS database programmer always struggles in finding the right method to do a database search, exact match, sub-string match, except condition, filters, unique values, table joins and data mining for which the toolbox also provides template scripts to modify and use from command line.

Results: The toolkit has been implemented on SLURM scheduler platform as a `bag-of-tasks` algorithm for parallel and distributed workflow though serial version has also been incorporated.

Conclusion: In the age of Big Data where there are way too many file formats and software and analytics environment each having their own semantics to deal with specific file types, SasCsvToolkit will find its functions very handy to a data engineer.