Anatomy of a CI/CD Pipeline Automation

A "Continuous Integration" or "CI" is where developers like you work on a codebase, making sure that it's working as expected and you do that continuously, so that it is nonstop. Every time there's new code, you either integrate it or test it or build it. Then you deploy it to production. And that's actually how we continuously push new product out to the customers.

For every application, this process is going to be different. But "Continuous Delivery" remains the same. It's getting it into the hands of the end users. And continuous delivery means we're wanting to do that very frequently.

So you see...? The purpose of a CI/CD pipeline?

It is to automate every part of the Software Development Life Cycle or SDLC.

Let’s look at what is it like without a CI/CD pipeline. So before we had CI/CD pipelines, You'd be writing code, put it on the development server and test it on that development server. This is how you are adding features or fixing bugs, and you're doing all of this work on the same development server.

Now, usually this is a whole team of developers working on an application and they are all putting the code on that development server, running it and testing and see if the features working.

So now what happens is that when you need to go to production, there will be another person, usually on the operation side or the system side, and he is going to go in and take what's on the development server, and then he'll copy it, copy the differences that he can, package it up, and then move it over to another server which is Staging.

Now, staging is supposed to be very similar in size and configuration as production. But once you put this application into staging, you're going to be making sure that it is working. That means that the system user, the one that's deploying this code, doesn't really know what the developers did on the development server. They might have had to make some tweaks and changes in the configuration of the server to make the code work.

So now again the person in the operation team is now going to try to find and make as many of those changes in staging as are required to make the application work. Once they have a working version, they're going to do that same again on production. They copy down and write down everything they're doing in the staging environment. And then they're going to go ahead and do that same thing again in production.

So you could see just from the explanation here, there's a lot of time going into people having to do these things and to change configuration files or server configurations or all of these different things, that will take alot of manual works to actually help this application get all the way into production. Sometimes just going from the development to production will take weeks and sometimes even months for very large applications because of all of the details required to make it work.

Now, Let's take a look at what it's like with CI/CD pipelines.

So you will write the code and then put the code into a repository. Now once the code is in the repository, it triggers a pipeline, when I say pipeline, think of this as more of an orchestrator. And so what you do in this pipeline is you're going to be doing all of those things that the developers had to do on the development environment, everything that the systems or operations team had to do on the staging and production environment.

You're going to have that all in code. And it's going to go ahead and do all of those things. So as soon as you write and commit the code, you put it to development. But before that, you have to package it, compile it, test it, and then you're going to automatically put it in the development environment so the developers can test and see if it works. Then once that has been done, the pipeline will be able to promote all of those changes it did automatically onto the staging server. Now, once staging is signed off and has an approval. You will be able to move that new code through applying those same changes onto the production environment.

Now you see, you are the only user in this scenario because the pipeline is taking care of all of the aspects that goes into changing configurations or making packages. Compiling all this stuff is going to be done automatically and it's going to put it right into the right environment.

This is how a pipelines help us. This is what we use them for. And in a very simple terms, this is what we want it to do. We want automated flow of code all the way through all of the environments.

Now there are some components that goes into a CI/CD pipeline.

1. Version Control System (e.g: git)

   Every time there's code change that is going to save a revision of the file. So you could see, how is the last commit compared to the previous one, or how does this change compared to a month ago? And how is the code changed over time or what lines were changed? All of this stuff, you can actually tell the differences between every time something has changed in that version control repository.

2. Pipeline

   Think of the pipeline as the orchestrator. All it really is doing is it's making everything work together. When a repository triggers a pipeline. The pipeline is going to trigger all the various things that needs to run. So the state of this code going to production and it is going to be controlled and managed through this pipeline.

3. Build Server

   A build server is what we use to do some of the compiling, packaging, maybe even testing. Instead of manually chaning things on each server, we do those changes on a build server. And that's kind of what its purpose is to make any changes needs to be done. It's going to be compiling and packaging everything up.

Some of the key principles of the CI/CD pipeline.

1. Automation - We want everything to be automated.

2. Everything as Code

   You might have heard terms like "Infrastructure as Code" or IaC. We want everything to be in code. The application, build instructions, code testing, deployment etc all shoul be in code. So we want end to end to be in code. And we honestly even put the pipeline itself in code. So that way we know when anything has changed.

3. Testing

   The only way to be confident that a pipeline works successfully is when it is tested thoroughly. So we want to be able to make tests throughout the pipeline to make sure that we don't introduce anything that will harm the production deployment. So we want to implement tests as frequently as we can to verify that the application is in a good sound state for deployment.

4. Consistency

   We want to be sure that staging is very consistent with how production is set up. So that way, when we go to staging, we know what to expect when we go to production. We want consistency in how we build, compile, package, and most importantly, how we deploy the service to the various environments.

5. Frequent Integration

   We want to continually integrate when we have a lot of code changes, and we're talking about months of work that we're going to be all of a sudden putting together. Now that's going to be potentially a very hard change to make, because that's a lot of changes over the course of a month that now we have to make sure that all of that code changes doesn't break anything.

   Whereas if we're integrating frequently, there might be five lines of code that changes. And, that's a lot safer change because we know what those five lines of code are doing. But over a month, we might have thousands of lines of code. And now finding out what caused the issue becomes that much harder because we have to find it in all of those lines of code. So integrating frequently is very important and actually will help speed up the life cycle of your software development life cycle, because we're actually able to go to production quicker, and we're able to make more releases.