The Startup CTO’s Guide to Ops (3 of 3): A Minimal Production and Deployment Setup

  • Keep costs as low as possible
  • Have a fast website that handles our target load
  • Make deployments painless
  • Ensure internal and external monitoring
  • Track business metrics
As long as it meets the requirements (art source)

Outline

Production setup

Our application stack

  • Language: Python 2.7
  • Web framework: Pyramid (WSGI)
  • Javascript: our site isn’t front-end heavy, but we use a fair bit of JQuery, JQuery datatables, and Chart.js
  • Web server: Waitress (Pyramid default, works fine)
  • OS: Ubuntu Linux 16.04
  • Database: PostgreSQL 9.5 (extensions: PostGIS, foreign data wrappers for Sqlite and CSV)
  • Load balancer/web server: Nginx

SSD

Codero hosting

Nginx

upstream myservice_prod {
server 127.0.0.1:8000;
server 127.0.0.1:8001;
}
server {
server_name mydomain.com www.mydomain.com;
listen 443 ssl;
ssl on;
ssl_certificate /etc/nginx/ssl/bundle.crt;
ssl_certificate_key /etc/nginx/ssl/star_mydomain_com.key;
location / {
proxy_pass http://myservice_prod;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header X-Forwarded-Proto $scheme;
}
}
server {
listen 80 default_server;
listen [::]:80 default_server;
server_name _;
return 301 https://$host$request_uri;
}

Where things are located, and users

[app:main]
use = egg:myservice
sqlalchemy.url = postgresql+psycopg2://user:pass@localhost/db_name
stripe.api_key = sk_live_stuff
stripe.public_key = pk_live_stuff
...and so on...
[app:main]
use = config:secrets-prod.ini
...

Web server

[server:main]
use = egg:waitress#main
host = %(http_host)s
port = %(http_port)s
url_scheme = https
threads = 8
sudo -u myservice /var/myservice-prod/venv/bin/pserve /var/myservice-prod/venv/config/production.ini http_host=localhost http_port=8000

Systemd

[Unit]
Description=Our Amazing Service, Production
[Service]
ExecStart=/var/myservice-prod/venv/bin/pserve /var/myservice-prod/venv/configs/production.ini http_host=localhost http_port=8000
WorkingDirectory=/var/myservice-prod
User=myservice
Group=myservice
Restart=always
[Install]
WantedBy=multi-user.target
$ sudo systemctl start myserviceprod
$ sudo systemctl stop myserviceprod
Actually, neither you nor my dog wants to exercise.

Logging

Deployment and Versioning

Package and Version Requirements

  • Packages have dependencies: a package should specify dependencies which will be automatically installed as part of deployment.
  • Configurations are treated like code: config files should be managed by the deployment system, either by bundling them as part of an application package (the approach we’ll be taking) or as their own deployable unit.
  • Deployments are versioned: each deployable release candidate is marked with a version like “1.0.32”. This version is visible to the application (it “knows” that it is 1.0.32), and also the version is used as a git tag so we have a clean historical record.
  • Enable pre-package hooks: sometimes your build has preparation tasks like minifying and combining css and js.

Application version

__version__ = '1.0.32'
from version import __version__
#...
def main(global_config, **settings):
#...
config.registry['version'] = __version__
#...
  • The application reports what version it is running, which makes it easy to check what-is-running-where.
  • Many cache keys include the version, so on deployment we bust the cache.
  • We append “...?v=<version>” to static web asset URLs. This forces clients to pick up the latest version after a deployment.

Bump version

  • Get the current git tag, increment it, and set a new tag
  • Update our Python version.py file with the new version
  • Push these changes to origin
  • Get the current tag: git describe --tags --abbrev=0
  • Get the toplevel directory for your git repo: git rev-parse --show-toplevel

Build package

# Pre-processing: minify our JS and CSS
./minify.sh
# Create source distribution
python setup.py sdist
  • There is a list of Python package dependencies. If we add packages or change versions, Pip will install the new packages as part of deployment.
  • We will use our versioning scheme.
  • Our config files are included in the build.
# Load the version (which is updated by our bump version script)
from myservice.version import __version__
requires = [
# List all required python packages here. If you
# add a new package, the deployment process will
# pick it up.
'pyramid',
'pyramid_jinja2',
#...etc...
]
# Add a few things to the setup() method
setup(
#...boilerplate stuff...
# Use our versioning scheme
version=__version__,
# Copy .ini files to venv/configs
data_files=[
('configs', ['staging.ini']),
('configs', ['production.ini']),
],
)

Deployment

ops/deploy.sh [username] [--staging or --production]
# Get the path of the most recent package 
distfile=$(ls -t myservice/dist | head -n 1)
# Copy the bundle to our remote host
scp $distfile $user@$host:/tmp
# Remotely:
# 1) Pip install the new bundle
# 2) Restart the service
# 3) Print the status (sanity check)
if [[ $2 == '--production' ]]
then
ssh -t $user@$host "sudo -u myservice /var/myservice-prod/venv/bin/pip install /tmp/$distfile --no-cache-dir; sudo systemctl restart myserviceprod; sleep 1; sudo systemctl status myserviceprod | cat;"
fi

In Conclusion: Focus on What Matters

  • Our setup is cheap: we spend about $140/month, plus a few annual fees.
  • The site runs well: in the course of a few weeks, we’ve steadily grown revenue and traffic, and easily managed the load when we appeared on Hacker News. Uptime has been 99.99%, and the production box has plenty of capacity.
  • Deployments are easy: we run a command line script which reliably works. Even though it’s rather basic, our system manages package dependencies, application versioning, configuration files, and SCM tags.
  • We have monitoring: there are external status checks and log file monitoring.
  • There are extensive metrics: we use Google Analytics and GrayLog dashboards to get real-time insights about our product.
  • We are not built for scale: but we could scale as needed; I’d start by moving the web servers to two or more virtual machines. The database has years of headroom; and honestly, I’d probably be much more inclined to throw more RAM and SSD at the database than move to a distributed system because having a single relational database keeps our code and system so much simpler.
  • We have single points of failure: but even in a catastrophe we could build a replacement within 2 hours. At our current size the business impact would be tolerable.
  • Our deployment scripts are simplistic: we don’t have clearly defined roles or host configuration management, or a package repository. As we grow, the next steps will be to set up a local Python package repository, and to move our deployment management to Ansible.
This corgi accepts you AND your infrastructure. (source)

Consulting CTO open to projects. I’m a serial entrepreneur, software engineer, and leader at early- and mid-stage companies. https://www.chuckgroom.com

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Milestone Project: How I built my first CLI Gem in Ruby

Common Linked List problems

Sergeys C# blog 301

Using Vanity URLs for Stability and Easier Sharing

Python USB Camera Tutorial for Raspberry Pi 4

DRY Your Rails Code with Singleton Class Methods and Metaprogramming

Upgrade to JetBeans IntelliJ IDEA

AWS Lambda Error Handling

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Chuck Groom

Chuck Groom

Consulting CTO open to projects. I’m a serial entrepreneur, software engineer, and leader at early- and mid-stage companies. https://www.chuckgroom.com

More from Medium

Staying Focused as an Engineer in 2022

How Does 360 DIGITECH process 10,000+ workflow instances per day by Apache DolphinScheduler?

The Introduction to Data Lake Architecture

A Startup CTO’s Guide to the Modern Data Stack