- \n
- Part 1 here<\/a><\/li>\n<\/ul>\n\n\n
Motivation<\/h2>\n\n\n
The previous article discusses the complexities of the machine learning life cycle<\/a> and how Vertex Pipelines<\/strong>, a managed version of Kubeflow on Google Cloud, can address operationalizing a machine learning model. The next step in the journey is to automate the training and deployment of the system in a manner that facilitates and enables reproducibility and time to market. In traditional software development, this capabilities would be enabled by a system that allows for<\/p>\n\n\n
- \n
- Continuous Integration (CI)<\/strong> , the ability to test and validate checked in code that needs to be merged into a central repository<\/li>\n
- Continuous Deployment (CD)<\/strong> , Creating and deploying the system to an appropriate environment<\/li>\n<\/ul>\n\n\n
Since machine learning models are dependent on the data, it must be accounted for in the CI\/CD process<\/a>. Particularly,<\/p>\n\n\n
- \n
- Data must be tested and validated for quality and training\/serving skew<\/a><\/li>\n
- A model must be continuously trained on new data to account for changes in the world<\/li>\n
- The model must be validated, tested and deployed to a prediction service<\/li>\n<\/ul>\n\n\n
![\"Image](\"https:\/\/liora.io\/app\/uploads\/sites\/9\/2024\/05\/image1-1024x307.png\")
Image from Architecture for MLOps using TensorFlow Extended, Vertex AI Pipelines, and Cloud Build<\/figcaption><\/figure>\n\n\n This also introduces the notion of continuous training. Depending on the use case, we need our model to be trained and updated on a frequency.<\/p>\n\n\n
Purpose<\/h2>\n\n\n
The goal of this article is to serve as a gentle introduction to CI\/CD for the Vertex Pipelines<\/strong> created in the previous article. To address all the components in a comprehensive CI\/CD workflow in the image above would go beyond the scope of a short blog. This article assumes the reader has foundational knowledge on concepts surrounding git, docker, and cloud technologies.<\/p>\n\n\n
What is Cloud Build ?<\/h2>\n\n\n
Cloud build is a fully managed, serverless CI\/CD platform<\/strong>. Other potential include Jenkins, Azure DevOps<\/a> and Github Actions. A CI\/CD platform allows for teams to effectively and iteratively develop and ship software to different environments. This tutorial will use Google Cloud Build alongside a github repository to build and run containers using the code that was pushed to a main branch.<\/p>\n\n\n
A Minimalistic CI\/CD\/CT Workflow<\/h2>\n\n\n
Prerequisites<\/h3>\n\n\n
- \n
- A github account and repository\n
- \n
- This article assumes the reader has some foundational knowledge in git. https:\/\/www.atlassian.com\/git\/tutorials\/setting-up-a-repository<\/a><\/li>\n<\/ul>\n<\/li>\n
- Google cloud environment<\/li>\n
- \n
- \n
- If you are new to Google Cloud, refer to this page<\/a> to get started. In this example, we are going to be developing our pipeline on a Vertex AI Notebook<\/a>.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n
Recall from the previous post, that the goal was to build an end-to-end ML workflow where<\/p>\n\n\n
- \n
- A model is trained\u00a0<\/li>\n
- Evaluated against a metric\u00a0<\/li>\n
- Deployed to a computer endpoint where predictions can be made<\/li>\n<\/ul>\n\n\n
Note that in some instances, the deployment workflow may be separate from the training workflow. For example, you may want your machine learning model to be trained on a weekly basis but go through an extensive quality assurance process before deploying.<\/p>\n\n\n
Rewrite our pipeline workflow<\/h3>\n\n\n
The first thing we will need to do is transform the<\/strong>python notebook file (.ipynb)<\/strong>into a normal python file (.py)<\/strong>.\u00a0 This python file contains all the executable code to recreate a compiled pipeline file that will actually be used to execute our machine learning workflow. This is because python files work much better with text files than with mixtures of text and binaries, which is what notebook files are. Refer to the github repository here for the full code.<\/p>\n\n\n
Browse our training courses<\/a><\/div><\/div>\n\n\nCreate a Dockerfile<\/h3>\n\n\n
The next step is to create a Dockerfile that contains our python file pipeline.py and our requirements.txt. Note that we are creating a separate folder (.\/kpi-cli) within our workspace to house the files for our docker image.<\/p>\n\n\n
\n\t\t\t\t\t%%writefile kfp-cli\/Dockerfile\nFROM gcr.io\/deeplearning-platform-release\/base-cpu\nWORKDIR \/kfp-cli\nADD pipeline.py .\/pipeline.py\nADD requirements.txt .\/requirements.txt\nRUN pip install -r requirements.txt\nCMD python .\/pipeline.py\n\t\t\t\t<\/code><\/pre>\n\n\nBuild a local image and push it to Container Registry<\/h3>\n\n\n
Now that a Dockerfile is defined and the dependencies exist within a directory, create a local image and push it to Artifact Registry<\/a>.<\/p>\n\n\n
\n\t\t\t\t\tIMAGE_NAME='kfp-truiz-mlops-v2'\nTAG='latest'\nIMAGE_URI='gcr.io\/{}\/{}:{}'.format(PROJECT_ID, IMAGE_NAME, TAG)\n!gcloud builds submit --timeout 15m --tag {IMAGE_URI} .\/kfp-cli\n\t\t\t\t<\/code><\/pre>\n\n\nThe cloudbuild product page within the console will contain the information on the build.<\/p>\n\n\n
Build a cloudbuild.yaml file<\/h3>\n\n\n
The next step is to create the cloudbuild.yaml file<\/strong>. This yaml file will contain the steps to execute the build when code gets pushed to the remote repository. For more information on creating the cloudbuild, refer to the documentation<\/a>.<\/p>\n\n\n
In short, this will do a few things<\/p>\n\n\n
- \n
- Build the docker image using the code in the repository. If there are errors in code, this build step will fail<\/li>\n
- Push the most recent version of the docker image back to Artifact Registry<\/li>\n
- Use the image that was created to run the python script in the repository. The output of this execution will be the pipeline yaml file\u00a0<\/li>\n
- The next step is to write the pipeline yaml file to a google cloud storage bucket. This yaml file can then be picked up by a scheduler to trigger the pipeline.<\/li>\n<\/ul>\n\n\n
\n\t\t\t\t\tsteps:\n # Step 1: Build the Docker image\n - name: 'gcr.io\/cloud-builders\/docker'\n args: ['build', '-t', 'gcr.io\/$_PROJECT_ID\/kfp-truiz-mlops-v2:latest', '.\/kfp-cli']\n # Step 2: Push the image back to container registry \n - name: 'gcr.io\/cloud-builders\/docker'\n args: ['push', 'gcr.io\/$_PROJECT_ID\/kfp-truiz-mlops-v2:latest']\n # Step 3: Run the Python file within the container\n - name: 'gcr.io\/$_PROJECT_ID\/kfp-truiz-mlops-v2:latest'\n args: ['python', '.\/kfp-cli\/pipeline.py'] \n env:\n - 'PROJECT_ID=$_PROJECT_ID'\n - 'REGION=$_REGION'\n # Step 4: Write results to Google Cloud Storage\n - name: 'gcr.io\/cloud-builders\/gsutil'\n args: ['cp', '.\/xgb-pipeline.yaml', 'gs:\/\/$_PROJECT_ID-cloudbuild-pipelines'] \noptions:\n logging: CLOUD_LOGGING_ONLY\nsubstitutions:\n _PROJECT_ID: '<PROJECT_ID>'\n _REGION: 'us-central1'\n\t\t\t\t<\/code><\/pre>\n\n\nSubmit a local job to the Cloud Build<\/h3>\n\n\n
Once the docker file is defined and the cloudbuild yaml created, a local build can be submitted using gcloud command line interface. This:<\/p>\n\n\n
\n\t\t\t\t\t!gcloud builds submit .\/kfp-cli --config cloudbuild.yaml\n\t\t\t\t<\/code><\/pre>\n\n\nDiscover our training courses<\/a><\/div><\/div>\n\n\nConnect Google Cloud to our Github Repository<\/h3>\n\n\n
Now that the cloudbuild file is defined, there needs to be an integration between the cloudbuild environment and the git repository. Fortunately this is pretty simple and straightforward to set up via the console<\/a>.<\/p>\n\n\n
The first step is to create a host connection to the git environment<\/strong>. This will in turn handle the authentication mechanism to the repository. You will be redirected to your github repository to allow the connection. An authentication token from GitHub will be created and stored in this project as a Secret Manager secret. You may revoke access to Cloud Build through GitHub at any time.\u00a0<\/p>\n\n\n
The second step is to link the repository<\/strong>. You will need to install the Google Cloud build<\/a> application within Github to allow the integration to the selected repository.<\/p>\n\n\n
![\"Illustration](\"https:\/\/liora.io\/app\/uploads\/sites\/9\/2024\/05\/image3.png\")
<\/figure>\n\n\nAfter that step is complete, you will be able to link the repository to your Google Cloud Build host connection.<\/p>\n\n\n
![\"Illustration](\"https:\/\/liora.io\/app\/uploads\/sites\/9\/2024\/05\/image2.png\")
<\/figure>\n\n\nCreate a Cloud Build Trigger<\/h3>\n\n\n
https:\/\/cloud.google.com\/build\/docs\/automating-builds\/create-manage-triggers<\/a><\/p>\n\n\n
- This article assumes the reader has some foundational knowledge in git. https:\/\/www.atlassian.com\/git\/tutorials\/setting-up-a-repository<\/a><\/li>\n<\/ul>\n<\/li>\n
- Continuous Deployment (CD)<\/strong> , Creating and deploying the system to an appropriate environment<\/li>\n<\/ul>\n\n\n
- Continuous Integration (CI)<\/strong> , the ability to test and validate checked in code that needs to be merged into a central repository<\/li>\n
![\"Illustration](\"https:\/\/liora.io\/app\/uploads\/sites\/9\/2024\/05\/image5.png\")
<\/figure>\n\n\n![\"Illustration](\"https:\/\/liora.io\/app\/uploads\/sites\/9\/2024\/05\/image4.png\")
<\/figure>\n\n\n