GitLab, Postgresql and problems on vservers

After working on a vserver, the little guy needed a reboot. And - who would have thought, after that reboot, GitLab didn’t start correct.

Without caring about the reasons, I decided to do GitLab a pleasure spending it an update. I should have known better, because I never did an GitLab update without trouble…

The problem

Even after the update, GitLab didn’t want to work. A subset of its processes was up and running but navigating to gitlab in my browser, presented me an error 502 and later on an error 500.

The problem was the combination of postgresql and the vserver, but one step after another:

Updating GitLab on Ubuntu 14.04

Let’s start with step one and spend GitLab the latest update. I am using binary Unbuntu packages for some time.

Only two lines in the terminal, and the everything will be installed rather quickly. It’s a good advice stoping all GitLab instances before sudo service gitlab stop or sudo gitlab-ctl stop, depending on the version.

Now we can start the installation:

curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
sudo apt-get install gitlab-ce

Don’t be surprised, at first, the gitlab package will be uninstalled. You’ll see a lot of information scrolling along your screen. On my first try, I got an error message, the package couldn’t be installed properly. Don’t ask why, but at my second attempt it worked.

After the installation we have to reconfigure GitLab to transfer the old settings:

sudo gitlab-ctl reconfigure

Done! Or are we?

Error 502 - Getting Postgresql to work

After the installation I got an error 502 when looking in the browser. Having a look at gitlab-ctl status told me, everything is fine.

I didn’t trust the status, so I started the GitLab selftest:

sudo gitlab-rake gitlab:app:check

At first everything looked fine, but then the test was aborted due a fatal error. Postgresql couldn’t be accessed because there wasn’t any socket file under tmp. So, to the Googles! I found some advices, one should create symbolic links etc. But then I found the real problem.

Postgresql, VServer and RAM

It seems like Postgresql is reserving 25% of the system memory by default. This will probably fail, running on a vserver. Looking at the logfile (/var/log/gitlab/postgresql/current) confirmed this theory.

GitLab comes with its own instance of Postgresql and so it also have an own config, which is included in the main GitLab config at /etc/gitlab/gitlab.rb.

Depending on the documentation of GitLab you have to set postgresql['shared_buffers'] = "100MB". I did, but Postgresql didn’t care. The logs said something about shmmax. So I searched within the config and found something. I set the value down to: postgresql['shmmax'] = 10179869184

After changes on that config, you have to reconfigure GitLab:

sudo gitlab-ctl reconfigure

After that I restarted Gitlab, too, just to make sure Postgresql gets the changes.

sudo gitlab-ctl restart

And here we are! A socket file at /tmp. Hurray! Launching my browser, heading towards GitLab… and… Error 500. Great.

Error 500

Because everything was up and running I started the selftest again, hoping I will get some more information this time.

sudo gitlab-rake gitlab:app:check

And tada! The datebase has to be migrated.

sudo gitlab-rake db:migrate RAILS_ENV=production

Running the test again confirms: everything is find. So let’s restart GitLab again:

sudo gitlab-ctl restart

Starting the browser and we’ll see a beautiful login screen. Nice! We can also use git to push and pull.

Let’s hope the next update will be easier.

Update

After restarting the server, the problem occured again. Great. This Blogpost helped.

Sources:

https://gitlab.com/gitlab-org/omnibus-gitlab/tree/master#postgres-error-39-fatal-could-not-create-shared-memory-segment-cannot-allocate-memory-39

https://gitlab.com/gitlab-org/gitlab-ce/issues/719

What you could do now

If you (don't) like this post, you can comment, write about it elsewhere, or share it. If you want to read more posts like this, you can follow me via RSS or ActivityPub, or you can view similar posts.