How to Set Up Free Incremental Deployments Over SFTP With Gitlab CI and PHPloy

Update: Gitlab has just announced paid plans, with a monthly limit on CI runner minutes (2,000). For personal projects this likely won’t have a major impact, but something to keep in mind if you frequently deploy or have long builds.

I have been really impressed with Gitlab.

They are constantly cranking out new features and fixes as they release every month like clockwork. What has really attracted me to Gitlab is their integrated CI (Continuous Integration). I’ve used Scrutinizer, Travis and DeployBot for professional projects, but I wanted to create a deployment system for my smaller, personal projects.

I have many small personal sites that I want to be able to easily edit from anywhere without depending on having everything on my local machine. I wanted to only require git and a text editor, and let the deployment tool handle the rest.

Most of these sites don’t make any money so I wasn’t real excited about paying for tools (even though I’d happily do so for my other projects) and it was also a fun challenge. I wanted to bring the structure and ease of deployment I have in my professional projects to my personal projects. I usually avoided making changes to these projects as I could never remember where the servers were, the credentials and what had potentially changed on the server in the meantime.

The Criteria

I needed this system to be free, automated, and host and local machine agnostic.

Free

As mentioned above, I wanted to see whether I could do this with freely available tools. Gitlab CI provides free runners that you can use, or you can run your own on a server you control. For these projects, I am only using the free runners as they are not mission critical applications.

Automated

I could use these tools to deploy straight from my local machine, but I wanted to be able to deploy just by pushing to Gitlab or by clicking a button in the UI.

Local Machine Agnostic

You can use any of the command line deployment tools I mention below to sync local files to another server. This is great, but it would mean that every machine I wanted to deploy from needed to have these tools installed and configured, and I needed to have the login credentials available.

Host Agnostic

My goal here was to have one deployment process that I could use regardless of host. Most the hosts provide SSH access, but not all. So I wanted something that only required STFP to deploy. This ruled out tools like Capistrano for me as SSH access is required.

The Deployment Tool

I originally used Dandelion, a Ruby deployment tool. It worked great, but the Ruby dependency meant I needed a separate container just to install it. The build runs much quicker if only one container is needed, so I sought out a PHP deployment tool.

Enter PHPloy.

PHPloy is an “Incremental Git (S)FTP deployment tool that supports multiple servers, submodules and rollbacks.” Rather than deploying your entire repo every time, it will only deploy what has changed since your last deployment. It does this by writing a .revision file to your remote server that stores the most recent deployment’s git commit hash. PHPloy determines which files have changed compared to that revision and only deploys those.

Some of the features that were great in Dandelion were missing or broken in PHPloy. It took a few PRs and bug reports to get it to the point where I’m comfortable using it for my projects and recommending it. The repository is fairly active and the author regularly merges PRs. I suggest getting involved on Github if this is a tool that would be useful to you.

Configuration

All you need to get started with Gitlab CI is a Gitlab repo with a .gitlab-ci.yml file. Below is an example configuration for deploying a PHP project with PHPloy.

.gitlab-ci.yml

stages:
- deploy

deploy_to_production:
  image: php:5.6-cli
  stage: deploy
  environment: production
  before_script:
    - apt-get update -yqq
    - apt-get install git zip unzip curl wget openssh-client -yqq
    - mkdir -p ~/.ssh
    - echo "$PRIVATE_KEY" | tr -d '\r' > ~/.ssh/deploy_rsa
    - chmod 600 ~/.ssh/deploy_rsa
    - ssh-keyscan -p $PHPLOY_PORT -H "$PHPLOY_HOST" >> ~/.ssh/known_hosts
    - wget --quiet https://github.com/banago/PHPloy/archive/master.zip
    - unzip -qq master.zip
    - mv ./PHPloy-master/dist/phploy.phar phploy
  script:
    - php phploy -s production
  when: manual

This file creates one stage called deploy that is run manually when triggered from the Gitlab UI.

Uses a PHP5.6-cli image which matches our production environment. You can choose any docker image.
Installs dependencies needed for deployment.
Copies the private key needed for server authentication to the container and adds the destination server to known_hosts.
Downloads the latest version of PHPloy
Runs the production PHPloy configuration to deploy.

phploy.ini

; NOTE: If non-alphanumeric characters are present, enclose in value in quotes.
[production]
scheme = sftp
path = /home/site/public_html
privkey = '~/.ssh/deploy_rsa'
branch = master
exclude[] = '.env.example'
exclude[] = '.gitignore'
exclude[] = '.gitlab-ci.yml'
exclude[] = 'phploy.ini'
exclude[] = 'README.md'

This file:

Sets the scheme, you’re not limited to SFTP.
The remote server deployment path.
The private key file path.
The branch to deploy.
Files to exclude

For SSH deployments you can use pre and post deploy hooks to run commands to backup or flush cache on deploy.

The only other configuration you need is to define four environment variables in Gitlab: PHPLOY_HOST, PHPLOY_PORT, PHPLOY_USER, and PRIVATE_KEY and create an environment.

For my example here, I created an environment named production which matches the environment in the .gitlab-ci.yml file.

Once this is configured, on your next push you should see a pipeline trigger in Gitlab and provide the option to run your deploy_to_production stage.

What’s missing?

Some of the downsides:

This isn’t a generic deployment tool. It works best for PHP projects that use git.
No easy way to include untracked files. This is something I love with DeployBot’s Build Tools. Any new files created as part of the build process are deployed to your server, as long as they aren’t .gitignored. I’m still looking for a good option here.
It takes longer to set up than an off-the-shelf deployment tool.

Future plans:

Use composer to pull down PHP dependencies and WordPress plugins.
Determine how to include untracked files (like composer dependencies).
Run unit tests and build scripts on the container before deploying.

There’s still more I can do here and may in the future, but for most of my small projects this works great. How do you deploy your hobby projects?

Update: I’ve edited the .gitlab-ci.yml file to reflect the new location of the phploy phar file.

Sources: Pushing to Dokku from Gitlab CI

How to Automatically Add Hosts to Vagrant Without a Password on OS X

UPDATE 2016-07-27: I do not believe this is working with the current version of vagrant-hostmanager 1.2.0. Will update if I find a resolution.

There’s an excellent plugin for vagrant called vagrant-hostmanager which will automatically add entries to your hosts file for domain aliases used in your vagrant instance. However, you’ll get a password prompt every time it runs as editing /etc/hosts requires elevated privileges. The instructions below allow you to run the hostupdater without having to enter your password every time.

The vagrant-hostmanager repo provides these instructions, but I’ve added additional information if you haven’t dealt with visudo before.

Be super careful when editing the sudoers file because editing it incorrectly can lock you out of your computer and prevent you from editing files!

Open Terminal
Check your $EDITOR env variable: echo $EDITOR
If it’s subl -w (for Sublime users) or anything that’s not nano, vi, or vim, you will need to use the longer version of the command below.
Short Version: sudo visudo
Long Version: sudo EDITOR=nano visudo
This opens the sudoers file for editing, which should look like this: http://opensource.apple.com/source/sudo/sudo-60/src/sudoers
If the file opens in Sublime Text or is empty, stop what you’re doing, otherwise proceed.
Near the bottom of the file, add these two lines, replacing <YOUR_USERNAME> with your OS X username:

Cmnd_Alias VAGRANT_HOSTMANAGER_UPDATE = /bin/cp /home/<YOUR_USERNAME>/.vagrant.d/tmp/hosts.local /etc/hosts
%sudo ALL=(root) NOPASSWD: VAGRANT_HOSTMANAGER_UPDATE

The next time you vagrant up or vagrant halt you shouldn’t be asked to provide your password. It will work for both command line vagrant use and a tool like Vagrant Manager.

Useful OS X Tools: Glui

One thing I rarely go a day without doing, is taking a screenshot.

It’s not something you’d immediately associate as a developer tool, but I use it constantly. How many times have you gone back and forth with a client trying to describe an issue, when all could be resolved with a simple screenshot?

OS X has screenshot capabilities built right in, but I find the easy annotation features well worth the price. Explaining UI issues is much easier to do with annotations than just trying to describe what’s happening.

Along with the annotations, you can hold down the option key to drag the image into any app that support drag and drop uploads, or click to upload directly to Dropbox and share publicly.

I used to use a free app called Skitch. They were bought by Evernote and then once Skitch was integrated with Evernote it became unusably slow and clunky. Then Glui came along and brought back all of the best parts of Skitch and none of the crap.

The app costs $6.99, but is well worth the price for something I use constantly.

$6.99 | Buy from the Mac App Store

Useful OS X Tools: Gas Mask

Gas Mask is a free and open-source hosts file manager for OS X. This is a small, yet very useful utility for web developers.

Normally, you’d have to manually edit your hosts file each time you need to make a change, and then flush your local DNS cache, to force the change, by running:

dscacheutil --flushcache

Now it’s as simple as picking from a dropdown in your menubar to switch between hosts file setups for different sites. Gas Mask handles the DNS cache flush for you, and the switch is instantaneous.

I initially started using this app when I would run local servers using the same domain as the live site. This made swapping WordPress databases between local and production much easier as I wouldn’t have to deal with find/replace on serialized data. Since that time, I’ve started using newer tools like WP Migrate DB Pro and Interconnect/it’s Search and Replace script which make migrating databases much easier and running a local server on the live domain is less important. I still use this to manage my hosts file when switching between sites running on MAMP Pro.

A few things to watch out for

In the screenshot above, you’ll see I have my computer’s hostname Joshs-MacBook-Air listed after the initial localhost declaration. The default hosts file content in Gas Mask won’t include this, but I found that I had trouble using hosts files without my machine’s hostname included there. Also, the editor could use some work, as some of the standard OS X keyboard shortcuts don’t function the same. The project is open source, so anyone who knows Objective-C can hop in and make changes, or make a donation to fund a feature for the project.

Price: Free (open source, donationware)

Download from GitHub | Donate