Jenkins is an open source continuous integration tool used to shorten the development lifecycle by automating the test and build process. We built Jenkins into Project Diaper’s infrastructure to automate away some of the tedious tests previously given to the infrastructure team when new builds were being deployed.
Current Jenkins server:
Hosted at https://jenkins.diaper.cf/ with the help of nginx. Please see the nginx config file at the following location for more information.
/etc/nginx/nginx.conf
Administrators login using the username and password stored in the login secrets folder on Box.
Current CI/CD Architecture
The current Jenkins configuration consists of 3 pipelines: one for frontend dashboard, one for backend dashboard, and one for backend mobile. The pipelines are configured to automatically build, test (with poor coverage, as no proper unit tests existed at the time of writing), and then deploy all code pushed to the master branches of each Github repository for the three pipelines. Finally an email with all output from each pipeline job run and the results of the job are sent to the diaperjenkins@gmail.com email address (password set to the password stored in the login secrets file on Box).
Each pipeline is currently hosted here infant-nutrition-project/jenkins-pipelines in the for of three separate Jenkinsfiles which specify the main behavior of the pipelines.
How to add on
If you want to build a new Jenkins pipeline, you start by navigating to the Jenkins dashboard, clicking "New Item," then enter a name and click "Pipeline," followed by "Ok." Then configure the pipeline with the following options:
You can then study previously created JenkinsFiles for the format of the new JenkinsFile needed to create the new pipeline and the plugins used (most importantly the "Publish Over SSH" plugin). You will likely need to SSH into a machine to deploy and for that you need to also configure the SSH credentials for that server in the "Publish Over SSH" options under "Configure System" on the "Manage Jenkins" page as exemplified here:
Then you'll update the Jenkinsfile for you pipeline (likely initially created as a copy of an existing Jenkinsfile) with the name you gave the server in the previous step everywhere there is a "shh Publisher" step in the Jenkinsfile. You also need to change the Git checkout step in the file to pull from the desired repository you are deploying with the new pipeline. Finally, you need to add web hooks to that very repository with the following configuration:
Troubleshooting
The first step to diagnose issues with the Jenkins server itself is to ssh into the Jenkins EC2 instance and run the following command:
$ sudo systemctl status jenkins
It should say Active: active (running). If you see some other status besides that you need to restart the server with the following command:
$ sudo systemctl restart jenkins
If you find Jenkins often changes status to Active: active (exited) that is probably due to the out of memory killer killing the Jenkins process. We believed we had the issue resolved at the point of writing, but if it comes up in the future consider cloning the EC2 instance to a machine with more memory or else just scheduling restarts of Jenkins at regular intervals. Alternatively if you can figure out how to reduce JVM memory usage that could also solve the problem.
For other issues with the Jenkins server itself consider making sure its SSL cert is not expired or checking the status of the nginx server.
If Jenkins is up and running and anything goes wrong with a build you should consult the errors either sent to the email or stored in the failed job report in Jenkins. You can find this by navigating to jenkins.diaper.cf and signing in with the username and password stored in the Cornell Box and clicking on the relevant job.
Restoring the Jenkins Server:
In the incredibly unlikely event that the Jenkins server fails, we backed up our EC2 instance to avoid repeating the many hours of work that went into the current configuration. All you need to do is navigate to the EC2 management console, click "Launch an instance", then click on "My AMIs" and select the most recently stored "Jenkins Server" Linux AMI. Next select T3 micro as the instance type. Then click the configure instance header at the top of the screen and then set the following settings:
Finally go into "Configure Security Group" click "Select an existing security group" and then check the box next to Production SG. Finally click "Review and Launch" followed by "Launch." Next select to use an existing key pair from the dropdown menu that appears, select DIAPER-production-key as the key, and click "Launch Instances." A new AWS EC2 server should now begin running with all configurations as of the end of Fall 2021 stored. I recommend backing up the server after any significant changes to the Jenkins EC2 instance configuration.
Installing Jenkins:
While you will likely never need to setup Jenkins yourself, as in the case of a failure you should just restore the EC2 instance. In case that is not possible we created this tutorial to make it easier to reconfigure Jenkins on a new EC2 instance if the need were to arise so you can reuse the pipeline code stored in GitHub and avoid starting a new pipeline from scratch. Before beginning this tutorial make sure you are comfortable interacting with our AWS EC2 instances via SSH.
Generally you just need to follow the tutorial here, and troubleshooting info for issues I ran into during the initial installation are provided below:
https://www.jenkins.io/doc/tutorials/tutorial-for-installing-jenkins-on-AWS/
Errors
curl: (60) SSL certificate problem: certificate has expired More details here: https://curl.haxx.se/docs/sslcerts.html curl failed to verify the legitimacy of the server and therefore could not establish a secure connection to it. To learn more about this situation and how to fix it, please visit the web page mentioned above. error: https://pkg.jenkins.io/redhat-stable/jenkins.io.key: import read failed(2).
Try running
$ yum upgrade ca-certificates
and if that doesn’t work try
$ sudo yum upgrade ca-certificates --disablerepo=jenkins
If you get the following error:
Error: Package: jenkins-2.319.1-1.1.noarch (jenkins) Requires: daemonize You could try using --skip-broken to work around the problem You could try running: rpm -Va --nofiles --nodigest
run
$ sudo amazon-linux-extras install epel -y $ sudo yum install daemonize -y $ sudo yum install jenkins java-1.8.0-openjdk-devel -y
Additionally following the installation you need to change the default user Jenkins runs as to "ec2-user". You can follow instructions given here: http://blog.manula.org/2013/03/running-jenkins-under-different-user-in.html
You will also need to install the Publish Over SSH plugin and add the private key of the Jenkins server to the Publish Over SSH options under the "Configure System" subpage of "Manage Jenkins." You'll also need to configure a key pair for accessing Github using "Manage Credentials" under "Manage Jenkins." Information on how to configure these plugins is widely available on StackOverflow and Jenkins own docs page.