Automate Everything ๐ค (and, anything!) is all the rage in IT today.
We are coding in an “extremely iterative” environment than ever before. Code-Build-Test cycles are getting shorter and shorter ๐จ(to minutes and seconds). Thanks to the CI/CD principles and tooling like GitLab, VSTS etc.
For my personal projects, I extensively use GitLab Community Edition to build and run Automation Pipelines. For those of you who wonder ‘Why GitLab CE?’, I find it as a great open source ecosystem to host your code in Git repos and build/run pipelines entirely on Cloud at absolutely no cost!
Some analogy..
“Redundancy” is good for infrastructure and app deployments (generally speaking). But, “Redundancy” is bad when it comes to building pipelines. You must build your pipeline with a “lite” mindset for it to contain only those instructions that are necessary to get your job done (asap). Anything that you find as “repetitive” or “redundant” must be done only-once, and not on every pipeline run. For example, installing dependencies, SDKs, compilers, runtimes etc which must be done only-once.
TL;DR
In this blog, I want to talk about how I managed to bring down Build times from 4 minute something to 1 minute something without compromising what-i-want-to-automate (my pipelines are now 72% faster ๐ so I can do “more in less”).

How it works?

Step-1: Write a “manual” job in your pipeline to create a runner image containing pre-installed dependencies, configuration or whatever tasks that are “repetitive”.
stages: - CREATE_RUNNER - DEPLOY_APIS Create Runner Image: image: docker:latest services: - docker:dind stage: CREATE_RUNNER script: - docker login $GIT_REGISTRY_URL -u "$GIT_REGISTRY_USER" -p "$GIT_REGISTRY_TOKEN" - docker build -t "$GIT_REGISTRY_URL/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$RUNNER_IMAGE_NAME:$RUNNER_IMAGE_TAG" . - docker push "$GIT_REGISTRY_URL/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$RUNNER_IMAGE_NAME:$RUNNER_IMAGE_TAG" when: manual
#6 tells GitLab to use Docker’s base image from Docker Hub.
#7, #8 is about starting Docker-In-Docker (dind) service which lets build a new Docker image out of our Dockerfile (all inside your running Docker container).
#12 builds Docker image based on instructions in Dockerfile (see below).
#13 pushes Docker image to GitLab Container Registry.
#14 indicates GitLab not to run this job when the pipeline runs. It is intended to run this job manually (for example, when your dependencies need an upgrade etc)
Go to GitLab โถ <<your repo>> โถ Settings โถ CI / CD โถ Variables to define:
GIT_REGISTRY_URL | registry.gitlab.com |
GIT_REGISTRY_USER | <<your gitlab username>> |
GIT_REGISTRY_TOKEN | Go to GitLab โถ <<your logo on top-right>> โถ Access Tokens โถ Enter <<name>>, Select <<scope>> as api , Click Create personal access token. Copy/Paste the token value here. |
RUNNER_IMAGE_NAME | <<give any name for your runner image>> E.g. my-runner-image |
RUNNER_IMAGE_TAG | <<give any tag for your runner image>> E.g. latest |
Step-2: This step is very specific to your needs, based on what dependencies you have for running your pipelines.
Look at this example Dockerfile which installs utilities (unzip, curl, wget etc) followed by dependent modules and runtimes that I use in my pipelines.
FROM amazonlinux ARG TERRAFORM_VERSION=0.11.14 ARG TERRAFORM_AWS_PLUGIN_VERSION=1.60.0 ARG ANSIBLE_VERSION=2.8.4 ARG SERVERLESS_VERSION=1.45.0 ENV PATH=/usr/bin:/usr/sbin:$PATH RUN yum update -y ############################################# ################## UTILS #################### ############################################# RUN yum install -y wget unzip tar git curl RUN wget http://stedolan.github.io/jq/download/linux64/jq && \ chmod +x ./jq && \ mv jq /usr/bin ############################################# ######### NODE 8.16.0, NPM 6.9.0 ############ ############################################# RUN curl --silent --location https://rpm.nodesource.com/setup_8.x | bash - && \ yum install -y nodejs && \ npm update -g && \ node -v && \ npm -v ############################################# ############ SERVERLESS 1.45.0 ############## ############################################# RUN npm install -g serverless@````$SERVERLESS_VERSION```` ############################################# ############ TERRAFORM 0.11.14 ############## ############################################# RUN mkdir -p /opt/software/terraform WORKDIR /opt/software/terraform RUN wget --no-check-certificate https://releases.hashicorp.com/terraform/````$TERRAFORM_VERSION````/terraform_````$TERRAFORM_VERSION````_linux_amd64.zip && \ unzip terraform_````$TERRAFORM_VERSION````_linux_amd64.zip && \ rm terraform_````$TERRAFORM_VERSION````_linux_amd64.zip && \ mv terraform /usr/bin/ && \ wget --no-check-certificate https://releases.hashicorp.com/terraform-provider-aws/````$TERRAFORM_AWS_PLUGIN_VERSION````/terraform-provider-aws_````$TERRAFORM_AWS_PLUGIN_VERSION````_linux_amd64.zip && \ unzip terraform-provider-aws_````$TERRAFORM_AWS_PLUGIN_VERSION````_linux_amd64.zip && \ rm terraform-provider-aws_````$TERRAFORM_AWS_PLUGIN_VERSION````_linux_amd64.zip && \ mv terraform* /usr/bin ############################################# ############### PYTHON 3.7.0 ################ ############################################# RUN mkdir -p /opt/software/python WORKDIR /opt/software/python RUN yum -y install python37 RUN update-alternatives --verbose --install /usr/bin/python python /usr/bin/python3.7 1 RUN sed -i "s/python/python2/g" /usr/bin/yum RUN cat /usr/bin/yum RUN curl -O https://bootstrap.pypa.io/get-pip.py && \ python get-pip.py RUN pip install --upgrade virtualenv j2cli j2cli[yaml] && \ python --version ############################################# ############## ANSIBLE 2.8.4 ################ ############################################# RUN pip install ansible==````$ANSIBLE_VERSION```` RUN yum clean all && rm -rf /var/cache/yum ENV PATH=/usr/bin:/usr/sbin:$PATH ENTRYPOINT ["/bin/bash", "-c"]
Step-3: Go to GitLab โถ CI/CD โถ Pipelines and run Setup Runner Image job. If everything goes as planned, you will see this.

Step-4: Go to GitLab โถ <<your repo>> โถ Packages โถ Container Registry and check if your docker image is available here.

Step-5: Voila! You can now use your newly built runner image to run your pipelines.
variables: GIT_RUNNER_IMAGE: "$GIT_REGISTRY_URL/$CI_PROJECT_NAMESPACE/$CI_PROJECT_NAME/$RUNNER_IMAGE_NAME:$RUNNER_IMAGE_TAG" stages: - CREATE_RUNNER - DEPLOY_APIS Deploy API - xxx: image: $GIT_RUNNER_IMAGE stage: DEPLOY_APIS script: - echo "Deploying API xxx..."
Hopefully, you are still reading this <eof>.. ๐ค