GitLab, Postgresql and problems on vservers
After working on a vserver, the little guy needed a reboot. And - who would have thought, after that reboot, GitLab didn’t start correct.
Without caring about the reasons, I decided to do GitLab a pleasure spending it an update. I should have known better, because I never did an GitLab update without trouble…
The problem
Even after the update, GitLab didn’t want to work. A subset of its processes was up and running but navigating to gitlab in my browser, presented me an error 502 and later on an error 500.
The problem was the combination of postgresql and the vserver, but one step after another:
Updating GitLab on Ubuntu 14.04
Let’s start with step one and spend GitLab the latest update. I am using binary Unbuntu packages for some time.
Only two lines in the terminal, and the everything will be installed rather quickly. It’s a good advice stoping all GitLab instances before sudo service gitlab stop or sudo gitlab-ctl stop, depending on the version.
Now we can start the installation:
curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash
sudo apt-get install gitlab-ceDon’t be surprised, at first, the gitlab package will be uninstalled. You’ll see a lot of information scrolling along your screen. On my first try, I got an error message, the package couldn’t be installed properly. Don’t ask why, but at my second attempt it worked.
After the installation we have to reconfigure GitLab to transfer the old settings:
sudo gitlab-ctl reconfigureDone! Or are we?
Error 502 - Getting Postgresql to work
After the installation I got an error 502 when looking in the browser. Having a look at gitlab-ctl status told me, everything is fine.
I didn’t trust the status, so I started the GitLab selftest:
sudo gitlab-rake gitlab:app:checkAt first everything looked fine, but then the test was aborted due a fatal error. Postgresql couldn’t be accessed because there wasn’t any socket file under tmp. So, to the Googles! I found some advices, one should create symbolic links etc. But then I found the real problem.
Postgresql, VServer and RAM
It seems like Postgresql is reserving 25% of the system memory by default. This will probably fail, running on a vserver. Looking at the logfile (/var/log/gitlab/postgresql/current) confirmed this theory.
GitLab comes with its own instance of Postgresql and so it also have an own config, which is included in the main GitLab config at /etc/gitlab/gitlab.rb.
Depending on the documentation of GitLab you have to set postgresql['shared_buffers'] = "100MB". I did, but Postgresql didn’t care. The logs said something about shmmax. So I searched within the config and found something. I set the value down to: postgresql['shmmax'] = 10179869184
After changes on that config, you have to reconfigure GitLab:
sudo gitlab-ctl reconfigureAfter that I restarted Gitlab, too, just to make sure Postgresql gets the changes.
sudo gitlab-ctl restartAnd here we are! A socket file at /tmp. Hurray! Launching my browser, heading towards GitLab… and… Error 500. Great.
Error 500
Because everything was up and running I started the selftest again, hoping I will get some more information this time.
sudo gitlab-rake gitlab:app:checkAnd tada! The datebase has to be migrated.
sudo gitlab-rake db:migrate RAILS_ENV=productionRunning the test again confirms: everything is find. So let’s restart GitLab again:
sudo gitlab-ctl restartStarting the browser and we’ll see a beautiful login screen. Nice! We can also use git to push and pull.
Let’s hope the next update will be easier.
Update
After restarting the server, the problem occured again. Great. This Blogpost helped.