• Created by Unknown User (jpb347), last modified on Dec 15, 2021

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 5 Next »

Jenkins is an open source continuous integration tool used to shorten the development lifecycle by automating the test and build process. We built Jenkins into Project Diaper’s infrastructure to automate away some of the tedious tests previously given to the infrastructure team when new builds were being deployed.


Current Jenkins server:

Hosted at https://jenkins.diaper.cf/ with the help of nginx. Please see the nginx config file at the following location for more information.

/etc/nginx/nginx.conf


Administrators login using the username and password stored in the login secrets folder on Box.


Installing Jenkins:

While you will likely never need to setup Jenkins yourself, in case there are any issues we created this tutorial to make it easier to reconfigure Jenkins on a new EC2 instance if the need were to arise. Before beginning this tutorial make sure you are comfortable interacting with our AWS EC2 instances via SSH.


Generally you just need to follow the tutorial here, and troubleshooting info for issues I ran into during the initial installation are provided below: 

https://www.jenkins.io/doc/tutorials/tutorial-for-installing-jenkins-on-AWS/

Errors


curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.
error: https://pkg.jenkins.io/redhat-stable/jenkins.io.key: import read failed(2).


Try running

$ yum upgrade ca-certificates

and if that doesn’t work try

$ sudo yum upgrade ca-certificates --disablerepo=jenkins


If you get the following error:


Error: Package: jenkins-2.319.1-1.1.noarch (jenkins)
           Requires: daemonize
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

run

$ sudo amazon-linux-extras install epel -y
$ sudo yum install daemonize -y
$ sudo yum install jenkins java-1.8.0-openjdk-devel -y

Current CI/CD Architecture

The current Jenkins configuration consists of 3 pipelines: one for frontend dashboard, one for backend dashboard, and one for backend mobile. The pipelines are configured to automatically build, test (with poor coverage, as no proper unit tests existed at the time of writing), and then deploy all code pushed to the master branches of each Github repository for the three pipelines. Finally an email with all output from each pipeline job run and the results of the job are sent to the diaperjenkins@gmail.com email address (password set to the  password stored in the login secrets file on Box). 

Each pipeline is currently hosted here infant-nutrition-project/jenkins-pipelines in the for of three separate Jenkinsfiles which specify the main behavior of the pipelines.


Troubleshooting

The first step to diagnose issues with the Jenkins server itself is to ssh into the Jenkins EC2 instance and run the following command:

$ sudo systemctl status jenkins

It should say Active: active (running). If you see some other status besides that you need to restart the server with the following command:

$ sudo systemctl restart jenkins

If you find Jenkins often changes status to Active: active (exited) that is probably due to the out of memory killer killing the Jenkins process. We believed we had the issue resolved at the point of writing, but if it comes up in the future consider cloning the EC2 instance to a machine with more memory or else just scheduling restarts of Jenkins at regular intervals. Alternatively if you can figure out how to reduce JVM memory usage that could also solve the problem.

For other issues with the Jenkins server itself consider making sure its SSL cert is not expired or checking the status of the nginx server.

If Jenkins is up and running and anything goes wrong with a build you should consult the errors either sent to the email or stored in the failed job report in Jenkins. You can find this by navigating to jenkins.diaper.cf and signing in with the username and password stored in the Cornell Box and clicking on the relevant job.

  • No labels