When you have a Jenkins instance that has been running for quite a while it is likely that the day will come when you will face the task of moving it to another server. In my case it was to move it from an old debian version to a recent one.
The jenkins instance I was working with was quite big. It hosted around 1000 jobs and its home directory (/var/lib/jenkins) contained around 16GB of data). I wanted the migration to be as smooth as possible, hence I wanted to limit downtime to a minimum.
How to transfer Jenkins including all of its settings, installed plugins and jobs
This one was quite easy, as jenkins stores all of its data in its home directory (by default /var/lib/jenkins) everything will be preservered as long as you copy the entire home directory to the new server.
How do I get the data transferred without having to shut down jenkins for a long time.
Rsync is your friend. I decided to use rsync to copy the contents of the jenkins home directory to the target server. That way I was able to copy all of jenkins’ data while it was still running, and run rsync a second time after shutting it down. The second run would of course be much faster as it only has to deal with the changes since the first run.
In order to preserve all existing file ownerships, you need to be logged in as root on the source host and transfer the data to the “root” user.
This is the command I used:
time rsync -av --delete -e "ssh -i /root/.ssh/<keyfile>" /var/lib/jenkins/* <targethost>:/var/lib/jenkins/
The time command in the beginning I only used to get the time the entire rsync command took.
The first time I ran rsync it seemed to take forever. The reason was that I have the config history plugin installed which creates thousands of small files over time. Some jobs had a history of more than 500 changes, which results in 500 files. I adjusted the plugin so that it would only keep a maximum of 100 entries. In the general configuration of jenkins you can adjust this:
Just be aware that jenkins won’t start removing config history for every job just by changing this value. Only after you hit the save button on the configuration page of a job this will be done. As a workaround I used the configuration slicing plugin. I went to the SCM Timer Trigger Slicer as this is disabled on all my jobs. I changed the value from “(Disabled)” to some fantasy value like “3 3 3 3 3” and clicked “Save”. That triggered a config change on all jobs, thus removing all the config history as configured above. Afterwardsprocess drastically.
The migration step by step:
- Create the new Jenkins host machine. Install Jenkins and make sure it runs. In my case I also made sure the /etc/default/jenkins file is copied as it contained some specific configuration. We are using chef so I made sure this file was managed via chef. Also I made sure the apache virtual host configuration file for jenkins was managed.
- Stop Jenkins service on the new host.
- Run Rsync as described above while the old Jenkins is still running
- Start Jenkins service on the new host but put it in “quiteDown” mode to prevent and Jobs from being processed. As the old Jenkins is still active it could be problematic if the same jenkins jobs are being triggered in parallel. To put jenkins into “quiteDown” mode just call the following url: https://<your_jenkins>/quietDown. This step is to verify that jenkins correctly starts on the new host. See if it can connect to all the slaves (if you have any).
When done you can stop jenkins on the new host again
- Stop Jenkins on the old host (Now the downtime begins)
- Run Rsync for a second time. This should be quite fast. In my case it took around a minute.
- When Rsync is finished start jenkins on the new host (End of downtime)
- Adjust DNS Record so that it points to the new Jenkins host
Et voilà! Your jenkins should now be available on the new host.