Continuous Deployment

With kedro pipelines started on the remote Kubeflow Pipelnes clusters, changes in the code require re-building docker images and (sometimes) changing the pipeline structure. To simplify this workflow, Kedro-kubeflow plugin is capable of creating configuration for the most popular CI/CD automation tools.

The autogenerated configuration defines these actions:

  • on any new push to the repository - image is re-built and the pipeline is started using run-once,

  • on merge to master - image is re-built, the pipeline is registered in the Pipelines and scheduled to execute on the daily basis.

The behaviour and parameters (like schedule expression) can be adjusted by editing the genrated files. The configuration assumes that Google Container Registry is used to store the images, but users can freely adapt it to any (private or public) docker images registry.

Github Actions

If the Kedro project is stored on github (either in private or public repository), Github Actions can be used to automate the Continuous Deployment. To configure the repository, go to Settings->Secrets and add there:

  • GKE_PROJECT: ID of the google project.

  • GKE_SA_KEY: service account key, encoded with base64 (this service account must have access to push images into registry),

  • IAP_CLIENT_ID: id of the IAP proxy client to communicate with rest APIs.

Next, re-configure the project using

kedro kubeflow init --with-github-actions https://<endpoint_name>.endpoints.<project-name>.cloud.goog/pipelines

This command will generate Github Actions in .github/workflows directory. Then push the code to any branch and go to “Actions” tab in Github interface.