Automate Everything 🤖 (and, anything!) is all the rage in IT today.
We are coding in an “extremely iterative” environment than ever before. Code-Build-Test cycles are getting shorter and shorter 💨(to minutes and seconds). Thanks to the CI/CD principles and tooling like GitLab, VSTS etc.
For my personal projects, I extensively use GitLab Community Edition to build and run Automation Pipelines. For those of you who wonder ‘Why GitLab CE?’, I find it as a great open source ecosystem to host your code in Git repos and build/run pipelines entirely on Cloud at absolutely no cost!
Some analogy..
“Redundancy” is good for infrastructure and app deployments (generally speaking). But, “Redundancy” is bad when it comes to building pipelines. You must build your pipeline with a “lite” mindset for it to contain only those instructions that are necessary to get your job done (asap). Anything that you find as “repetitive” or “redundant” must be done only-once, and not on every pipeline run. For example, installing dependencies, SDKs, compilers, runtimes etc which must be done only-once.
TL;DR
In this blog, I want to talk about how I managed to bring down Build times from 4 minute something to 1 minute something without compromising what-i-want-to-automate (my pipelines are now 72% faster 🚀 so I can do “more in less”).
How it works?
Step-1: Write a “manual” job in your pipeline to create a runner image containing pre-installed dependencies, configuration or whatever tasks that are “repetitive”.
stages:
- CREATE_RUNNER
- DEPLOY_APIS
Create Runner Image:
image: docker:latest
services:
- docker:dind
stage: CREATE_RUNNER
script:
- docker login $GIT_REGISTRY_URL -u "$GIT_REGISTRY_USER" -p "$GIT_REGISTRY_TOKEN"
- docker build -t "$GIT_REGISTRY_URL/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$RUNNER_IMAGE_NAME:$RUNNER_IMAGE_TAG" .
- docker push "$GIT_REGISTRY_URL/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$RUNNER_IMAGE_NAME:$RUNNER_IMAGE_TAG"
when: manual
#6 tells GitLab to use Docker’s base image from Docker Hub.
#7, #8 is about starting Docker-In-Docker (dind) service which lets build a new Docker image out of our Dockerfile (all inside your running Docker container).
#12 builds Docker image based on instructions in Dockerfile (see below).
#13 pushes Docker image to GitLab Container Registry.
#14 indicates GitLab not to run this job when the pipeline runs. It is intended to run this job manually (for example, when your dependencies need an upgrade etc)
Go to GitLab ▶ <<your repo>> ▶ Settings ▶ CI / CD ▶ Variables to define:
| GIT_REGISTRY_URL | registry.gitlab.com |
| GIT_REGISTRY_USER | <<your gitlab username>> |
| GIT_REGISTRY_TOKEN | Go to GitLab ▶ <<your logo on top-right>> ▶ Access Tokens ▶ Enter <<name>>, Select <<scope>> as api , Click Create personal access token. Copy/Paste the token value here. |
| RUNNER_IMAGE_NAME | <<give any name for your runner image>> E.g. my-runner-image |
| RUNNER_IMAGE_TAG | <<give any tag for your runner image>> E.g. latest |
Step-2: This step is very specific to your needs, based on what dependencies you have for running your pipelines.
Look at this example Dockerfile which installs utilities (unzip, curl, wget etc) followed by dependent modules and runtimes that I use in my pipelines.
FROM amazonlinux
ARG TERRAFORM_VERSION=0.11.14
ARG TERRAFORM_AWS_PLUGIN_VERSION=1.60.0
ARG ANSIBLE_VERSION=2.8.4
ARG SERVERLESS_VERSION=1.45.0
ENV PATH=/usr/bin:/usr/sbin:$PATH
RUN yum update -y
#############################################
################## UTILS ####################
#############################################
RUN yum install -y wget unzip tar git curl
RUN wget http://stedolan.github.io/jq/download/linux64/jq && \
chmod +x ./jq && \
mv jq /usr/bin
#############################################
######### NODE 8.16.0, NPM 6.9.0 ############
#############################################
RUN curl --silent --location https://rpm.nodesource.com/setup_8.x | bash - && \
yum install -y nodejs && \
npm update -g && \
node -v && \
npm -v
#############################################
############ SERVERLESS 1.45.0 ##############
#############################################
RUN npm install -g serverless@````$SERVERLESS_VERSION````
#############################################
############ TERRAFORM 0.11.14 ##############
#############################################
RUN mkdir -p /opt/software/terraform
WORKDIR /opt/software/terraform
RUN wget --no-check-certificate https://releases.hashicorp.com/terraform/````$TERRAFORM_VERSION````/terraform_````$TERRAFORM_VERSION````_linux_amd64.zip && \
unzip terraform_````$TERRAFORM_VERSION````_linux_amd64.zip && \
rm terraform_````$TERRAFORM_VERSION````_linux_amd64.zip && \
mv terraform /usr/bin/ && \
wget --no-check-certificate https://releases.hashicorp.com/terraform-provider-aws/````$TERRAFORM_AWS_PLUGIN_VERSION````/terraform-provider-aws_````$TERRAFORM_AWS_PLUGIN_VERSION````_linux_amd64.zip && \
unzip terraform-provider-aws_````$TERRAFORM_AWS_PLUGIN_VERSION````_linux_amd64.zip && \
rm terraform-provider-aws_````$TERRAFORM_AWS_PLUGIN_VERSION````_linux_amd64.zip && \
mv terraform* /usr/bin
#############################################
############### PYTHON 3.7.0 ################
#############################################
RUN mkdir -p /opt/software/python
WORKDIR /opt/software/python
RUN yum -y install python37
RUN update-alternatives --verbose --install /usr/bin/python python /usr/bin/python3.7 1
RUN sed -i "s/python/python2/g" /usr/bin/yum
RUN cat /usr/bin/yum
RUN curl -O https://bootstrap.pypa.io/get-pip.py && \
python get-pip.py
RUN pip install --upgrade virtualenv j2cli j2cli[yaml] && \
python --version
#############################################
############## ANSIBLE 2.8.4 ################
#############################################
RUN pip install ansible==````$ANSIBLE_VERSION````
RUN yum clean all && rm -rf /var/cache/yum
ENV PATH=/usr/bin:/usr/sbin:$PATH
ENTRYPOINT ["/bin/bash", "-c"]
Step-3: Go to GitLab ▶ CI/CD ▶ Pipelines and run Setup Runner Image job. If everything goes as planned, you will see this.
Step-4: Go to GitLab ▶ <<your repo>> ▶ Packages ▶ Container Registry and check if your docker image is available here.
Step-5: Voila! You can now use your newly built runner image to run your pipelines.
variables:
GIT_RUNNER_IMAGE: "$GIT_REGISTRY_URL/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$RUNNER_IMAGE_NAME:$RUNNER_IMAGE_TAG"
stages:
- CREATE_RUNNER
- DEPLOY_APIS
Deploy API - xxx:
image: $GIT_RUNNER_IMAGE
stage: DEPLOY_APIS
script:
- echo "Deploying API xxx..."
Hopefully, you are still reading this <eof>.. 🤓
