CI/CD for your Kestra Workflows

Embrace software engineering practices for versioning, deploying, and maintaining data workflows

Get started
for your Kestra Workflows
What is CI/CD?
CI/CD stands for Continuous Integration and Continuous Deployment, software engineering practices that involve regularly merging code changes (Continuous Integration) and automated deployment to production (Continuous Deployment). CI/CD practices make the development process more efficient, less error-prone, and better aligned with business needs.

Build your Workflows as Code

Kestra allows building workflows as code. This approach facilitates seamless integration with existing CI/CD processes, Infrastructure as Code tools like Terraform, and version control systems like Git.

Diagram illustrating Kestra's integration with GitHub Actions

Implementing CI/CD for Kestra Flows

You can version-control your Kestra workflows in a Git repository, easily track changes, collaborate with other team members, and roll back to previous versions if necessary.

The CI/CD pipeline automatically validates and deploys the updated workflows to the Kestra environment whenever changes are committed and pushed to a given Git branch.

From Development to Production

Creating data workflows often starts locally and gradually moves towards production. It involves designing data pipelines, setting up tasks, dependencies, and verifying the code's functionality.
.
The typical workflow of moving a data pipeline from development to production starts with a pull request. Once approved, the changes get deployed to the production. .
.
This streamlined approach ensures that the production environment remains stable and secure while allowing for rapid adaptation to changing business needs and requirements.
Picture of development workflow to production deployments
Illustration showing Kestra integration to Hashicorp Terraform provider

Manage DataOps Lifecycle with Kestra and Terraform

Combining Kestra and Terraform provides a reliable foundation for managing the data engineering lifecycle while promoting collaboration between engineers and business users.

The combination of Infrastructure as Code and Workflow as Code enables organizations to embrace engineering best practices, balancing fast feedback loops during development with the benefits of infrastructure as code for production environments.

Fast iteration cycles

Kestra enables both data teams and business stakeholders to participate in workflow creation using a simple declarative configuration.

Kestra UI provides an embedded code editor, versioning, auto-completion, and a topology view of workflow dependencies, simplifying the development process for all stakeholders. This way, Kestra facilitates collaboration and fast iteration cycles.
Image showing a fast feedback loop
Monograme

Apply DevOps best practices in your data engineering lifecycle

Kestra provides a comprehensive solution to streamline the data engineering lifecycle by offering collaborative development and infrastructure as code capabilities. Through its user-friendly interface and integration with tools like Terraform, Kestra enables organizations to develop and deploy data workflows efficiently while maintaining the stability and security of production environments.

Key Features

Workflows as Code

Define and manage data workflows using YAML configuration, which opens data workflows to more people in your organization.

Versioning

Kestra workflows are automatically versioned, helping track changes and rollback to previous versions if needed.

CI/CD Integration

Kestra integrates with Terraform, GitLab, and GitHub Actions to automate the deployment of data workflows to production environments.

Software Engineering Best Practices

Kestra embraces software engineering best practices, promoting high-quality and reliable data pipelines.

Ready to take your data platform to the next level?

Talk to UsGet Started