The advent of Agile development means developers can deliver software products tailored to meet customer design specifications and requirements. In an Agile development environment, smaller groups in a team often work on different aspects of a larger project and commit regularly. The frequent addition of smaller chunks of code to the codebase through version control ensures that previous work is recoverable. However, this frequent addition of code to the codebase can break the existing codebase without proper communication and monitoring. To minimize this manual scrutiny and redundant communication, we need to invest in CI/CD processes. In this article, we will walk through an end-to-end workflow of automating a CI/CD process for a Flask application via Travis CI and Heroku.
CI/CD
Continuous integration places much emphasis on automating test procedures to check that the application is not broken when new commit are made to the main branch. In CI, developers commits their changes back to the main branch as frequently as possible. For each commit, a build is created and the automated tests are run against it to validate it before it moves to the next stage in the CI/CD pipeline. This prevents the difficulties resulting from developers having to wait until the software release date to merge their changes.
With Agile development, software can be released at any time. If you can’t release new changes to your customers as quickly as possible, then it’s not Agile. Continuous delivery is responsible for ensuring that the software can be released at any time. A successful exit from the continuous integration stage moves the continuous delivery stage where a series of automated and manual tests from the QA team verifies that the product meets production requirements.
Continuous delivery ensures that every change to the main branch that passes all the stages of the production pipeline gets deployed to the production environment — except when a failed test occurs within the pipeline, in which case, a previous version of the software gets released instead.
What we are building
We are going to build a modern CI/CD pipeline for a simple Flask application for question-answering, a simple NLP project. And in this tutorial, we will go through all the processes involved in a modern CI/CD workflow (illustrated below):
Source: volansys.com
In this tutorial, we’ll specifically do the following:
- Create a GitHub repository for our project.
- Code our Flask application with the Python Flask Framework.
- Push our Flask application to GitHub.
- Add automated tests to our Flask application
- unittest and pytest for testing code logic
- Flake8 for testing code style
- (We won’t do integration tests in this tutorial.)
- Set up Travis CI to run our automated tests for each commit
- Create a Docker image to package our Flask application
- Push Docker image to DockerHub.
- Deploy Docker image to Heroku.
- Set up Travis CI to automatically run all tests, create a Docker image, push the image to DockerHub, and Deploy Docker to live servers on Heroku. This process will be done automatically for each commit to our GitHub repository.
- Creating a GitHub repository for our project
Start by creating a GitHub (or any version control) repository for our application. This step is necessary for any CI/CD, because Travis CI will directly retrieve and build our application from the GitHub repository.
GitHub repository setup for our Flask application
- Let’s code our simple Flask application (askme)
We will be building a simple NLP project for question-answering with a web interface using the Flask framework. Askme will take a question and magically output an answer — However, this is not the focus of the tutorial, so you can use your own application to follow along.
Interface of the askme Flask application
This is how the source code of our project currently looks from VS Code:
Source code of project from VS Code
Once you have your project working on your local server, push to GitHub. Now that we have a working version on GitHub, let’s add automated tests to our application.
- Add automated tests to our Flask application
We will write two tests for this application. The first test checks that the /askme endpoint returns a 200 status code. And the second test simply checks that the number of lines in the answers.txt file is 2609 (which is true). So far, your project folder directory should look something like this:
Project folder directory
Next, run your tests locally, and make sure your code passes all test cases. To run the tests, install pytest and run the following command in your root project directory: python -m pytest -v tests/test_class.py. When your code passes all test cases, commit and push to GitHub. To check our code style with flake8, we will make Travis CI run that for us. We will skip integration tests for this tutorial, but it’s important to have these tests in your project.
Content of test_class.py and results from running tests
- Set up Travis CI to run automated builds and tests
Travis CI offers continuous integration services, and is extremely easy to set up. Proceed to travis-ci.org, and sign in with your GitHub account. After signing in, you will find a list of all your public GitHub repositories. To enable Travis CI, track your repository, and build your project for each push, flip the switch next to your project repository:
Enable tracking for your GitHub repo on Travis CI
After activating Travis CI to track your project repository, add a .travis.yml file to the root of your project directory and add the following lines to it:
install: - pip install -r requirements.txt # command to run tests script: # unit test - python -m pytest -v # linting test - flake
Additionally, create a requirements.txt file in the root of your project directory, and list all packages that are required to be installed before your project can run. List the following packages:
- pytest
- flake8
- pandas
- flask
- nltk
- pathlib
- Create a Docker image to package our Flask application
Your project directory should look something like this:
Commit all your changes and push to GitHub, and watch Travis CI take it up, build it and run all automated tests in your Travis CI dashboard.
Part of build for project on Travis CI dashboard
Docker is a software platform designed to make it easier to create, deploy, and run applications by using containers. It will contain all the parts needed for our application to run as a single package. The importance of this is that we can run our application on any environment without worrying about dependencies breaking our code.
To Dockerize our application, first download and install Docker and make sure that it is running on your computer. The command docker –version should output the version of Docker you are using in your terminal to indicate a successful installation.
Once you have docker running locally, add a file called Dockerfile to the root directory of your project, and add the following lines to it:
FROM ubuntu:latest MAINTAINER <your name> <your email> RUN apt-get update -y RUN apt-get install -y python-pip python-dev build-essential COPY . /app WORKDIR /app RUN pip install -r requirements.txt ENTRYPOINT ["python"] CMD ["qa/app.py"]
These lines will instruct Docker to pick the latest Ubuntu image, update its repository, install python-pip, python-dev, and build-essential and launch our Flask application when the container is ready.
These lines in the Dockerfile build the image locally. We need to deploy it to DockerHub after each build, to make it shareable on multiple environments, and we can automate the build and deployment to DockerHub from Travis CI.
- Push the Docker image to DockerHub.
To enable Travis CI to orchestrate this build and deployment to DockerHub, create a directory called .travis in your root directory and add a file called deploy_dockerhub.sh. Add the following lines to the created file:
docker login --username $DOCKER_USER --password $DOCKER_PASS if [ "$TRAVIS_BRANCH" = "master" ]; then TAG="latest" else TAG="$TRAVIS_BRANCH" fi docker build -f Dockerfile -t $TRAVIS_REPO_SLUG:$TAG . docker tag $TRAVIS_REPO_SLUG $DOCKER_REPO docker push $DOCKER_REPO
Under the Settings view of your repository on Travis CI, set the environment variables used in the script as shown:
Finally, in your .travis.yml file, add the following lines to it:
after_success: - sh .travis/deploy_dockerhub.sh
Then, add the following lines to the beginning of your .travis.yml file:
sudo: required services - docker
Project directory structure and .travis.yml content from VS Code
Commit all your changes and push to GitHub, and once again watch Travis CI orchestrate this process on your behalf. Travis CI will build your project, run all your automated tests, install Docker, build the Docker image, launch the image when the container is ready, and deploy it to DockerHub. In the event of any failure, you will be notified through your email.
Docker digest pushed to DockerHub from Travis CI
Now, let’s try to launch the built image of our Flask application on our local computer from our container at DockerHub. Run:
docker run -p 5000:5000 --rm -it ci-cd-with-travis
from the terminal on your local computer and see the log of the application running from DockerHub. The beauty of pushing to DockerHub is that we can now launch our application from DockerHub from any environment.
Log of application running from our Docker container at DockerHub from our local computer
This output shows that everything from our pipeline is working perfectly so far. The next step is to make Travis CI deploy our Docker image from DockerHub to a live server in Heroku, so that we can access our application on the Web.
- Deploy Docker image to Heroku
Heroku is a hosting platform for applications. It also offers a free service for lightweight applications, so we will host our project with Heroku. But we will first need to create an account at heroku.com.
Additionally, we will use the Heroku CLI from our local terminal to create our application, so download and install the Heroku Command Line tool from Heroku Dev Center. Make sure Heroku can be invoked via your computer’s terminal, and run the following to create the application:
heroku login heroku create
Creating app… done, desolate-dawn-14339
https://desolate-dawn-14339.herokuapp.com/ | https://git.heroku.com/desolate-dawn-14339.git
This last command creates your application and assigns it a name and a URL address, which you can use to access your application from the Web.
Now, to allow Travis CI to pick up each successful build of our Docker image from DockerHub and deploy to Heroku, add a file called deploy_heroku.sh in our .travis directory, and add the following lines to it:
wget -qO- https://toolbelt.heroku.com/install-ubuntu.sh | sh heroku plugins:install @heroku-cli/plugin-container-registry docker login --username _ --password=$HEROKU_API_KEY registry.heroku.com heroku container:push web --app $HEROKU_APP_NAME heroku container:release web --app $HEROKU_APP_NAME
Set the environment variables in the Settings view of your repository in your Travis CI dashboard.
All environment variables from Travis CI dashboard
The Heroku app name is the name of application generated when you run the heroku create command from your terminal. And the Heroku API Key can be obtained under the account settings from your Heroku account. Finally, add the following line to your .travis.yml file under the after_success section:
This command checks whether the repository is a master branch, and then deploys to Heroku which can then be assessed via the URL address that was generated when you ran the heroku create command above.
Content of deploy_heroku.sh file from VS Code
Commit all changes and push to GitHub. And then watch as Travis CI once again builds the project, runs tests, builds the Docker image and pushes it to DockerHub — and then finally deploys the image to Heroku. After this process is successful, we can then view our live application from our application URL address.
Application after successful deployment from Travis CI
Now that our CI/CD pipeline is complete, let’s add a modification to our application and push and watch what happens. Let’s change the background color of our application to yellow and push our changes, and see whether our application changes automatically after a successful deployment by Travis CI.
And boom!! It worked!
Output after changing background color to yellow and pushing to GitHub
So our CI/CD pipeline seems to be in good shape. Anytime you make changes and commit them, Travis CI will go through the same process, and will finally deploy changes to Heroku when no error exists. However, when a process fails, Travis CI will alert you via email and stop the process, and a previous version of your application will be released instead.
Access the final version of our project source code from GitHub.
References
https://www.atlassian.com/continuous-delivery/continuous-integration-tutorial
https://medium.com/bettercode/how-to-build-a-modern-ci-cd-pipeline-5faa01891a5b
https://runnable.com/docker/python/dockerize-your-flask-application
