Moving Jenkins from one server to another

pablo.png

When you have a Jenkins instance that has been running for quite a while it is likely that the day will come when you will face the task of moving it to another server. In my case it was to move it from an old debian version to a recent one.

The jenkins instance I was working with was quite big. It hosted around 1000 jobs and its home directory (/var/lib/jenkins) contained around 16GB of data). I wanted the migration to be as smooth as possible, hence I wanted to limit downtime to a minimum.

The Concerns

How to transfer Jenkins including all of its settings, installed plugins and jobs

This one was quite easy, as jenkins stores all of its data in its home directory (by default /var/lib/jenkins) everything will be preservered as long as you copy the entire home directory to the new server.

How do I get the data transferred without having to shut down jenkins for a long time.

Rsync is your friend. I decided to use rsync to copy the contents of the jenkins home directory to the target server. That way I was able to copy all of jenkins’ data while it was still running, and run rsync a second time after shutting it down. The second run would of course be much faster as it only has to deal with the changes since the first run.

In order to preserve all existing file ownerships, you need to be logged in as root on the source host and transfer the data to the “root” user.

This is the command I used:

time rsync -av --delete -e "ssh -i /root/.ssh/<keyfile>" /var/lib/jenkins/* <targethost>:/var/lib/jenkins/

The time command in the beginning I only used to get the time the entire rsync command took.

The first time I ran rsync it seemed to take forever. The reason was that I have the config history plugin installed which creates thousands of small files over time. Some jobs had a history of more than 500 changes, which results in 500 files. I adjusted the plugin so that it would only keep a maximum of 100 entries. In the general configuration of jenkins you can adjust this:

Auswahl_136.png

Just be aware that jenkins won’t start removing config history for every job just by changing this value. Only after you hit the save button on the configuration page of a job this will be done. As a workaround I used the configuration slicing plugin. I went to the SCM Timer Trigger Slicer as this is disabled on all my jobs. I changed the value from “(Disabled)” to some fantasy value like “3 3 3 3 3” and clicked “Save”. That triggered a config change on all jobs, thus removing all the config history as configured above. Afterwardsprocess drastically.

The migration step by step:

  1. Create the new Jenkins host machine. Install Jenkins and make sure it runs. In my case I also made sure the /etc/default/jenkins file is copied as it contained some specific configuration. We are using chef so I made sure this file was managed via chef. Also I made sure the apache virtual host configuration file for jenkins was managed.
  2. Stop Jenkins service on the new host.
  3. Run Rsync as described above while the old Jenkins is still running
  4. Start Jenkins service on the new host but put it in “quiteDown” mode to prevent and Jobs from being processed. As the old Jenkins is still active it could be problematic if the same jenkins jobs are being triggered in parallel. To put jenkins into “quiteDown” mode just call the following url: https://<your_jenkins>/quietDown. This step is to verify that jenkins correctly starts on the new host. See if it can connect to all the slaves (if you have any).
    When done you can stop jenkins on the new host again
  5. Stop Jenkins on the old host (Now the downtime begins)
  6. Run Rsync for a second time. This should be quite fast. In my case it took around a minute.
  7. When Rsync is finished start jenkins on the new host (End of downtime)
  8. Adjust DNS Record so that it points to the new Jenkins host

Et voilà! Your jenkins should now be available on the new host.

Tagged , , , ,

Chef: How to insert or append a line to a file

When using chef, I have often faced the problem of having to insert a line into an existing file.
The way I solved this before, was to use bash.
bash "insert_line" do
  user "root"
  code <<-EOS
  echo "172.22.0.47    www.example.com" >> /etc/hosts
  EOS
  not_if "grep -q www.example.com /etc/hosts"
end
The Chef (or better the Ruby) way to do this is the following:
ruby_block "insert_line" do
  block do
    file = Chef::Util::FileEdit.new("/etc/hosts")
    file.insert_line_if_no_match("/www.example.com/", "www.example.com")
    file.write_file
  end
end
Now this big question is, which is better or more readible. I guess this depends if you are more familiar with shell or ruby scripting.
One advantage of the ruby way is that the Chef::Util::FileEdit class offers a couple of other possibilities to manipulate files.
For example:
  • insert_line_if_no_match
  • search_file_delete
  • search_file_delete_line
  • search_file_replace
  • search_file_replace_line
Tagged , ,

Deployment Automation

Everyone is talking about Continuous Delivery and DevOps these days. An elemental part of this is the move towards automation of recurring tasks that have been performed manually in many businesses in the past. Especially comprehensive deployment automation is one of the most desirable goals in this methodology.

The software build process is widely automized nowadays thanks to tools like Maven or Ant and Continuous Integration servers like Jenkins, TeamCity or GO. The Internet provides plenty of good technical instructions about how to setup and run these tools. But when it comes to the last mile, the actual deployment process, most literature doesn’t go into much detail about best practices and tools to use.

Currently I am trying to dig deeper into this field, and I’d like to share my learnings. The tools that support automated deployments are also called command dispatching frameworks.  The ones I run into the most are (this list is not supposed to be complete):

Within the next days I will try to gather more information about these tools. I will not be capable to go into detail with all of them, but I will share the experiences I’ve made.

Tagged , ,