Clever Cloud

Clever Cloud addressed their data volume challenge by integrating Kestra into their workflow, automating the process of data offloading to enhance system efficiency and data handling capacity.

Clever Cloud provides a Platform as a Service solution, based in Europe. Clever Cloud exists for one purpose: helping people and companies to deliver software and services faster. Their promise is to ensure that once an app is deployed, it stays up, no matter what (high traffic, security udpates, DDoS , application failure, hardware issues etc..). The PaaS helps development teams to put digital applications and services into production on a reliable infrastructure, with automatic scalability and transparent pricing. With monitoring data reaching 20TB weekly, Clever Cloud needed a robust solution to manage this influx without compromising system performance or storage efficiency.

Technology Solution

They employed Warp10 HFiles for data compression and FoundationDB for data management, and Kestra's for orchestrating the automation of the data offloading process.

• Back-end Logic: Managed by proprietary solutions leveraging high-performance databases like FoundationDB, ensuring rapid data processing.
• Database Management: Using Warp10 for time-series data handling, accommodating hundreds of thousands of data points per second.
• Infrastructure Support: Custom microVM, managed by their internal Orchestrator.

The Role of Kestra

Kestra facilitated the automation of Clever Cloud's data management, from initiating data transfers to optimizing storage through efficient data compression, making the entire process more manageable and reliable.

Core tasks used:

• io.kestra.plugin.fs.http.Request: This task is crucial for initiating HTTP requests within their workflows, serving as the primary method for interacting with external data sources and services.
• io.kestra.core.tasks.flows.Subflow: Enables the execution of sub-workflows within a main workflow, allowing for modular and reusable workflow design.
• io.kestra.core.tasks.flows.EachParallel and io.kestra.core.tasks.flows.EachSequential: These tasks manage the execution of operations either in parallel or sequentially, optimizing workflow execution based on the task's nature and dependencies.

The decision to primarily use the Request task is driven by the bulk of the logic being implemented through mini-scripts in a specific language (Warpscript), executed against their database. This approach allows Clever Cloud to encapsulate complex data processing logic within manageable and efficient scripts.

Workflow Design

• Main Workflow with Monthly Trigger: The principal workflow orchestrates the monthly data offloading process. It sequentially calls other workflows tailored to specific business needs, such as differentiating between infrastructure metrics and client metrics.
• Variable and Parameter Utilization: The workflows leverage variables and parameters to customize and differentiate the execution context for various business requirements. This flexibility allows for the parallelization of the main "archive" workflow launches, effectively managing different data sets and offloading tasks.
• Failure Management: In the event of a failure within the "archive" workflow, an "archive_failure" workflow is triggered. This auxiliary workflow notifies the team on Slack about the failure, ensuring prompt awareness and response to any issues encountered during the data offloading process.

Outcomes

This integration led to significant improvements in handling data volume, maintaining system performance, and optimizing storage use. The success of this project has encouraged further exploration of automating other areas within Clever Cloud's infrastructure with Kestra.