From time-critical, real time computational experimentation to applications which process petabytes of data there is a continuing search for faster, more responsive computing platforms capable of supporting computational experimentation. Weather forecast models, for instance, process gigabytes of data to produce regional (mesoscale) predictions on a short turnaround. Through science gateways, these scientific experiments are made accessible to scientists and users worldwide. The introduction of cloud computing increases the options for computational platforms, putting scientists in the position of choosing the best resources to use that maximize their needs. Historical information recorded in job executions can reveal important information about the patterns and experiences of the users to improve the management of scientific jobs in large-scale systems.

This dissertation makes three key contributions to management of scientific job execution in large-scale systems. First, it proposes user patterns, using knowledge-based approaches, to provision for compute resources in such a way that maximizes cost metrics reducing the impact of startup overheads in cloud computing environments. Second, it proposes and through experimental evaluation establishes the viability of user derived reliability, learned by mining historical job execution information, as a new dimension to consider during resource selections. Experimental studies conducted on a prototype implementation of above two contributions validate the applicability of this approach and the accuracy of prediction models. Third, it proposes and prototypes a light-weight and reliable resource abstraction service to hide the complexities of interacting with multiple resources managers in grids and clouds. The framework has been applied in four different domains, including LEAD II cyberinfrastructure, and its performance validated through empirical analysis of such conditions as scalability in comparison to related approaches.