Monitoração da sanidade de web scrapers com OpenTelemetry.

Abstract

Web Scrapers are tools for collecting data from websites, widely used to provide various types of services. However, when you have one or more data extraction services from websites, how do you know the "health" of their operation? There is no compatibility commitment between the site and the scraper because they are generally developed by different teams. Thus, the scraper may inadvertently fail to function due to changes made to the site. In this work, a strategy using OpenTelemetry was developed to emit metrics and operational traces to address this lack of insight into the scraper's health. It is expected that by the end of this work, it will be possible to apply the solution in practice and thus have a single location with information on the operation of Web Scrapers.