Melhorando o processamento de dados com Hadoop na nuvem através do uso transparente de instâncias oportunistas com qualidade de serviço.

Abstract

Cloud computing offers the users the ease of resources acquisition through the Internet in a fast, cheap and safe manner. However, these clouds have a lot of idle resources due to resource reservation. Aiming to increase resources usage, cloud providers have created an instance model that uses these idle resources, known as opportunistic instances. These instances are cheaper than the dedicated resources instances, but are volatile and can be destroyed at any time, which makes them unsuitable for some types of application. Data processing, following the trend of other applications, have been migrated to the cloud and can be benefited by the use opportunistic instances, due to its fault tolerant nature, resulting in the creation of clusters at a lower cost compared to instances with dedicated resources. In this work, we propose the use of idle resources to create another model of opportunistic instances. This model aims to create opportunistic instances with quality of service, which are created instances based on a prediction of the state of the cloud. The prediction is made from historical data of resource usage such as CPU and RAM, thus reducing the risk of losing instances before the end of the processing. Even with the existence of a predictor, the risk of losing a machine still exists, and for this case we propose the use of live migration, moving the virtual machine to a different server, thus avoiding the its destruction. With our approach, using only two opportunistic instances during the experiments, we found a decrease in 10% in the data processing time in a cluster with 2 workers and 1 master. Furthermore, when using the migration, we have an improvement of approximately 70% in processing time compared with the case where one instance is lost.