What is a CI pipeline?
Pipeline is a fancy word used by the CI community to define a sequence of steps your build needs to go through. At Fire CI we believe in configuration as code and we've created a YAML format to allow you to easily define your build steps.
The only thing you need to do to tell Fire CI how to run your build is create a file called .fire.yml at the root of your repository.
How do I define build steps in YAML?
Let's get right into it with the example below:
version: 2 pipeline: steps: - step: phase: install image: node cache: - package.json - yarn.lock workdir: somefolder # optional commands: - yarn - step: phase: build name: build workdir: somefolder # optional commands: - yarn build - step: phase: test name: unit tests commands: - yarn lint - yarn test
This looks fairly simple. Because it is. A pipeline definition is just a serie of steps where you define which commands need to be run.
Each step is defined by its phase in the build. The phases can be:
- install: This is the setup of your project. You define which Docker base image to use, which commands to run and the caching conditions. See the next section for more details.
- build: You are building the code in order to run tests on it afterwards.
- test: You are running some testing, whether this is unit or integration or end to end testing.
- deploy: Coming soon. You are deploying your built artifacts somewhere (Docker registry, Heroku, AWS, Azure, etc.)
Each step needs to have at least a phase, name and one command. The install step has the 'cache' property which is described later in the section about caching.
The version property at the top of the file needs to be 2.
The integration of YAML steps with GitHub looks like this:
Each step can have an optional workdir property to point to a subfolder of the repository in which the commands for the step will be executed.
Sharing data between steps
All the steps of your pipeline run in the same container and thus share data by design. If you need outputs of one step in another step you can just rely on the file structure in the container as you would running the commands locally in your cloned working copy.
Install step and smart caching
Your pipeline can contain only 1 install step and it is important to define it right so your builds are faster.
The install phase is used to build the base image to run your build. All the project dependencies will be installed during this phase and will be cached until they change. To detect upon which conditions the dependencies cache needs to be rebuilt you need to specify the files used by the installation process. It will usually be package.json and yarn.lock for JS based projects, and potentially other installation scripts if you are doing funny things with yout project.
A typical install step would look as follows:
- step: phase: install image: node cache: - package.json - yarn.lock commands: - yarnThe step above tells Fire CI to use the Docker image 'node' as a starting point. Then the files defined in the cache property are copied into the build container and then the commands are run.
At the end of the install phase the image will be tagged fireci-YOUR_REPO_URL-base and used to run build and test command phases.
Fire CI uses smart caching mechanisms to optimize subsequent builds: if the files listed in the cache property do not change in content then the step won't be rerun and the tagged image will be used as base to run the following steps. If these files change in content though, the step will be rerun and the fireci-YOUR_REPO_URL-base will be updated.
This is a huge boost in performance! Pulling dependencies for your project is by far the longest operation on most tech stacks. The list of dependencies changes far less often that you push your code. Caching this step makes your builds way faster.
A few more words about caching
The install step actually installs all the dependencies of your project into the container file system. You need to specify in the cache all the files that are needed to do so. For example let's imagine a React/yarn project that uses package.json and yarn.lock to identify dependencies, but the 'yarn' command to run refers to some scripts that are in an 'internals' folder in your repo. The install step needs to look like this:
- step: phase: install image: node cache: - package.json - yarn.lock - internals/ commands: - yarn
Another useful caching feature is on demand images configuration. Let's assume no image from the Docker public registry fits your build needs. You could create such an image and publish it, which is a big maintenance and versioning overhead. Instead you can add commands to your install step to tailor your image to what you want it to be. The example below is an install step using the 'alpine' Docker image and installing yarn since it is not available out of the box:
- step: phase: install image: alpine setup: cache: - package.json - yarn.lock - internals/config.js - internals/scripts commands: - echo -e 'http://dl-cdn.alpinelinux.org/alpine/edge/main\nhttp://dl-cdn.alpinelinux.org/alpine/edge/community\nhttp://dl-cdn.alpinelinux.org/alpine/edge/testing' > /etc/apk/repositories - apk add --no-cache nodejs-current nodejs-npm yarn - yarnThe smart caching will also kick in for that case so subsequent builds do not need to configure the image until your project dependencies change. This is way more handy in maintenance and versioning than publishing your own Docker images to the public or a private registry.
Docker ignore to speed up the build
To speed up your build and depending on the template, the build agent will automatically create a .dockerignore file to speed up the sending of the build context to the docker daemon. To override the ignores just add a "dockerignore" section at the top of your YAML file. It is highly recommended to remove the .git folder and any folder that has heavy binary dependencies that can be reinstalled through the install phase.
version: 1 dockerignore: - .git - node_modules pipeline: steps: - step: ... - step: ...
The creation of the .dockerignore file is skipped if you already have a .dockerignore file in your repository.
If you think the YAML format does not suit your need or already have some builds defined in Docker that you want to use, checkout the other ways you can configure your builds:
If you have questions or needs of more features on the YAML files do not hesitate to contact us at firstname.lastname@example.org.