GCP AI Platform support
Google Cloud’s AI Platform offers couple services that simplify Machine Learning tasks with use of Kubeflow based components.
kedro with AI Platform Notebooks
AI Platform Notebooks provides
an easy way to manage and host JupyterLab based data science workbench environment.
What we’ve found out is that the default images provided by a service cause some
dependency conflicts. To avoid this issues make sure you use isolated virtual
environment, e.g. virtualenv. New virual
environment can be created by simply invoking
python -m virtualenv venv command.
kedro-kubeflow with AI Platform Pipelines
kedro-kubeflow plugin should work with AI Platform Pipelines out of
the box, with the only exception is that it requires authentication. Note that the
variable should point to a dashbard URL generated by AI Platform Pipelines service
just open the dashboard from the service page
and copy url from the browser.
Below is the list of authentication scenarios supported so far:
1. Connecting to AI Pipelines from AI Platform Notebooks
In this scenario authentication works out of the box with default credentials mechanism.
2. Authentication to AI Pipelines from local environment
To interact with AI Platform Pipelines from local environment you can use the
mechanisms provided by Google Cloud SDK. After
installing the SDK run
google cloud application-default login to initialize
default credentials on your local machine.
You can use service account key for authentication as well. To make that work just
GOOGLE_APPLICATION_CREDENTIALS environment variable to the path of where the
service account key file is stored.
3. Authenticating through IAP Proxy
Identity Aware Proxy is a product that allows securing your cloud based applications with Google Identity.
To authenticate with IAP find out which oauth client ID is the proxy configured
with and then save it in
IAP_CLIENT_ID environment variable. The authentication
should work seamlessly assuming identity you are using has been granted access to
The above will work if you are connecting from within GCP VM or locally with specified
service account credentials. It will NOT work for credentials obtained with
google cloud application-default login.
kedro-kubeflow with Vertex AI Pipelines (DEPRECATED)
Vertex AI Pipelines support in
kedro-kubeflow has been deprecated in favour of the
new plugin kedro-vertexai