Maintaining Maintenance

Sometimes well-intentioned features have unintended side effects. Case in point: WordPress’ maintenance mode. Whenever you update plugins WP will automatically enter maintenance mode, which displays a nice message to your visitors that the site will be back online shortly. It will automatically go out of maintenance once the updates are done.

Well, sometimes unexpected things happen: you are stuck in maintenance mode. WP will effectively lock you out … even the admin section will not be accessible. *ugh* This is the moment you start panicking … luckily if you wait 10 minutes or delete the .maintenance file manually you’ll be able to access your site again. *phew*

Just went though that whole cycle. m(

Custom CAs everywhere

I recently finished introducing custom CA infrastructure in two instances. Each having two sub CAs, two Servers and a bunch of users. The “create your own CA” part was quite easy after I found  a dated but still accurate tutorial. In hindsight it is quite silly why I didn’t do this before.

On the server side I had to make it work on:

  • Apache
  • Nginx
  • Postfix
  • Cyrus
  • Dovecot

Each expecting its own Format/Packing of certificates, keys and certificate chains. :/

On the client side I had to produce installation and configuration howtos for Windows and OS X and a bunch of popular browsers and email clients. Then there is the “user education” part … this is still in progress, but its looking good.

All in all, I’m happy with the result. 🙂

Fixing Borked UTF-8 Data in MySQL

While updating ownCloud to version 4 it reencoded my already UTF-8-encoded data and left me with borked strings.

I thought about trying to do a bunch of find and replace operations, but I knew this was error prone.

A little internet research produced a very simple simple solution for fixing double-encoded data in MySQL:

mysqldump -u DB_USER -p DB_PASSWORD --opt --quote-names --skip-set-charset --default-character-set=latin1 DB_NAME > DB_NAME-latin1-dump.sql
mysql -u DB_USER -p DB_PASSWORD --default-character-set=utf8 DB_NAME < DB_NAME-latin1-dump.sql

Just replace DB_USER, DB_PASSWORD and DB_NAME with the appropriate values and your good to go. 😀

Howto Renew Your SSL Certificates

So my IMAP server certificate expired today … so I needed to renew it. I use self-signed certificates for services I run myself.

First you need your config file. If you don’t have this you will be prompted to do so. (the Ubuntu Wiki has a nice introduction)

[ req ]
default_bits = 2048
encrypt_key = yes
distinguished_name = req_dn
x509_extensions = cert_type
prompt = no

[ req_dn ]
C=DE
ST=HB
L=Bremen
O=IMAP-Server
OU=Automatically-generated IMAP SSL key
CN=imap.your-domain.tld
emailAddress=postmaster@yourdomain.tld

[ cert_type ]
nsCertType = server

Then you generate the new certificate (expiring in 365 days) using the config file from above (imapd.cnf) and have it save it into imapd.pem.

openssl req -x509 -days 365 -nodes -newkey rsa:2048 -config imapd.cnf -keyout imapd.pem -out imapd.pem

Restart your service.

If you need to check the new key’s fingerprint you can get it with the following command.

openssl x509 -in imapd.pem -fingerprint

Debugging SASL

If you are using Cyrus SASL with your Postfix you might feel the need to debug what SASL does in the background. But SASL does not log into /var/log/mail.*. 🙁

So after some research I fount a way …

/etc/init.d/saslauthd stop

Stop the SASL daemon and start it by hand:

saslauthd -d -a pam -r -c -m /var/spool/postfix/var/run/saslauthd

Consult the MECHANISMS and OPTIONS settings in /etc/defaults/saslauthd for which options to use in your case.
But the most important option is -d. It will run the daemon in the foreground and make it more verbose.

Now it will show you everything it does. 😀

Don’t forget to start the actual daemon once you are done debugging:

/etc/init.d/saslauthd start

How to to set up Gitlab on Debian

Update: This howto is outdated. GitLab has changed a lot since it was written and a lot of it is not applicable anymore (e.g. since GitLab 5.0 it doesn’t depend on Gitolite any more and only needs one system user to be setup). So you are probably better off using the official installation guide. 🙂


If you want to install Gitlab on Debian you can easily follow their installation steps for Ubuntu. But be careful there are a few gotchas nobody is talking about.

The following steps will assume you are root.

Preparations

First make sure you have all the latest updates installed.

aptitude update
aptitude full-upgrade

Then we have to install a few packages.

aptitude install git-core wget curl gcc checkinstall libxml2-dev libxslt-dev sqlite3 libsqlite3-dev libcurl4-openssl-dev libc6-dev libssl-dev libmysql++-dev make build-essential zlib1g-dev libicu-dev redis-server sudo

Install Ruby

If you have not installed ruby you might want to consider using RVM.

Install it with

bash -s stable < <(curl -s https://raw.github.com/wayneeseguin/rvm/master/binscripts/rvm-installer)

It will be installed into /usr/local/rvm.

Ask it for the requirements for installing MRI and install them.

rvm requirements
aptitude install build-essentials ...

Install ruby and make it the default.

rvm install 1.9.3
rvm --default use 1.9.3

You should install a minimum set of gems. Add “passenger” if you are running Apache as your web server or “thin” if you are using Nginx.

gem install bundler

Install Gitolite

First of all we want to create a dedicated user for Gitolite and Gitlab. This will also be the user the Rails processes will be running in (this is important later).

adduser \
  --system \
 --shell /bin/sh \
 --gecos 'Git Version Control' \
 --group \
 --disabled-password \
 --home /home/git \
 git

Configure git for the new user.

sudo -u git -H git config --global user.email "git@your-server.tld"
sudo -u git -H git config --global user.name "Gitlab Admin"

Generate the ssh key for the git user. It will be saved in /home/git/.ssh/id_rsa. We will run Gitlab as the git user so it will use this key to authenticate against Gitolite.

sudo -u git -H ssh-keygen -t rsa -b 2048

Copy the public part of the key for later use when we setup Gitolite.

sudo -u git -H cp /home/git/.ssh/id_rsa.pub /home/git/rails.pub

After that we install Gitolite. In contrast to the Gitlab documentation I installed it from the Debian repositories.

aptitude install gitolite

It will not be fully installed as it will tell you something like:

No adminkey given – not initializing gitolite in /var/lib/gitolite.

So we do this by using dpkg-reconfigure and using our previously prepared account.
When prompted, answer as follows:

  • Gitolite user: git
  • repositories directory: /home/git
  • admin key: /home/git/rails.pub
dpkg-reconfigure gitolite

Now you should have Gitolite set up in the /home/git directory. But we will still have to tweak it a little.

Edit /home/git/.gitolite.rc and find the line that reads “REPO_UMASK = 0077;” and change it to “REPO_UMASK = 0007;” (i.e. three zeros).

You now need to change the directory privileges on the /repositories directory so Gitlab can use them

sudo chmod -R g+rwX /home/git/repositories/
sudo chown -R git:git /home/git/repositories/

Gitolite should be ready now.

You can test it by cloning the admin repository:

sudo -u git -H git clone git@localhost:gitolite-admin /tmp/gitolite-admin
rm -rf /tmp/gitolite-admin

Install Gitlab

Install a few prerequisites.

aptitude install python-dev python-pip redis-server libicu-dev
sudo pip install pygments

Clone Gitlab

git clone git://github.com/gitlabhq/gitlabhq.git gitlab
cd gitlab

We create a gemset for Gitlab to not pollute the global gemset. To automate this we will use a .rmvrc inside the Gitlab directory. RVM will make sure it will be loaded automatically whenever you enter the directory.

echo "rvm use 1.9.3@gitlab --create" > .rvmrc

cd into directory to make rvm use the .rvmrc and accept with “y”.

cd .. && cd gitlab

Check your current gemset with

rvm current

It should show something like “ruby-1.9.3-p0@gitlab”.

Now you might need to update Gitlab’s Gemfile (e.g. add the mysql2 gem for MySQL databases).

Now install the gems necessary for running Gitlab.

bundle install --deployment

You may need to run “bundle install –no-deployment” to pick up changes to the Gemfile and rerun the previous command.

Edit config/gitlab.yml to configure Gitlab. If you have followed this howto you should only need to update the “email” section and the “host” option in the “git_host” section.

You might want to edit config/application.rb and update the time zone and locale configurations.

Edit config/database.yml and set up your database configuration.

Now set up and initialize your database.

bundle exec rake db:setup RAILS_ENV=production
bundle exec rake db:seed_fu RAILS_ENV=production

Install with Passenger + Apache

(todo)

Install with Thin + Nginx

(todo)

Result

😀

Catching bugs in the cloud

I encountered am error when syncing my mobile phone’s address book with ownCloud:

PHP Fatal error: Call to undefined method OCContactsVCard::deleteFromDAVData() in /path_to_owncloud/apps/contacts/lib/connector_sabre.php on line 194″ while reading response header from upstream, client: 0.0.0.0, server: your-server.tld, request: “DELETE /owncloud/apps/contacts/carddav.php/addressbooks/user/default/number.vcf HTTP/1.1”

It turns out there is a slight misnomer in the code. It has been fixed in the development version, but the fix is not yet part of an official release.

I documented my findings on ownCloud’s issue tracker with step by step instructions on how to fix this until there is a fixed release available.