Nick Mitchinson

Aligning Double-Sided Cards in Spite of Cricut Design Space

Sat, 08 Aug 2026 09:00:00 -0700

Cricut Design Space is the required software to use when interacting with a Cricut device. It is also one of the most infuriating, piece of shit softwares I have ever had the necessity to use.

Most of what makes it awful is a licensing decision rather than a technical one. A Cricut is a stepper-driven gantry with a tool head on it. That is a CNC machine. Nearly every other machine in that class — mills, routers, 3D printers, most desktop lasers — speaks some dialect of G-code, which means you can drive it from whatever software you prefer, or from a file you generated yourself. Cricut instead ships a proprietary protocol, an account requirement, and a cloud round-trip for a device sitting eighteen inches from your hands. There is no engineering reason a blade on a gantry needs to phone home before it moves three inches to the left.

The lock-in is the whole point, and they have been fairly open about it. In March 2021 they announced that anyone without a subscription would be capped at twenty uploads of their own artwork per month, and only reversed it after several days of very loud backlash. That is the posture of a company that knows you cannot take your machine elsewhere. The software is as bad as it is because it has never had to be better.

Anyway. Here is one specific way it is broken, and the workaround I ended up being forced to build.

Change Capacity

Mon, 27 Jul 2026 00:00:00 -0700

For most of the time I’ve been writing software, the thing that decided what actually shipped was how fast the team could build it. Engineering capacity — the throughput of design, build, test, and release — was the ceiling. It’s why engineering leadership tended to end up as the de facto gatekeepers of the roadmap: the constraint on what reached customers was the engineering org’s constraint. If you wanted more out the door, you argued for more engineers, or you argued about priorities inside a fixed budget of engineering-weeks.

Agents are quietly removing that ceiling, and I don’t think we’ve fully reckoned with what replaces it.

Using Git Worktrees for Multi-Feature Development with AI Agents

Mon, 13 Oct 2025 00:00:00 -0700

If you’ve ever needed to work on multiple features simultaneously, you’ve likely encountered the friction of context switching between git branches. The traditional approach—stashing changes, switching branches, and checking out different code—disrupts your flow and can be error-prone. Git worktrees offer an elegant solution, and they become even more powerful when you’re working with local AI coding agents. What Are Git Worktrees?# Git worktrees allow you to check out multiple branches of the same repository simultaneously in different directories.

About | Nick Mitchinson

Mon, 19 May 2025 09:00:00 -0800

living in San Diego working on a number of new things Will mostly talk about technical things, primary in the Kubernetes, engineering practices, and Reliability space, maybe with some opinions of mine thrown in the mix. I do technical advising for early-stage startups, primarily in the B2B SaaS space, with a focus on backend systems/product architecture, reliability, observability, cloud operations, security, and preparing teams to have systems in place to facilitate scale.

Consulting | Nick Mitchinson

Mon, 19 May 2025 09:00:00 -0800

I provide consulting services within software engineering and management, primarily focused on small to mid-sized engineering and product teams, with a focus on backend systems/product architecture, reliability, observability, cloud operations, security, and preparing teams to have systems in place to facilitate scale. If you’re looking for advice, you can book an initial conversation via my consultancy, Stratus Studios. Consulting engagements begin with an initial flat-rate evaluation of the problem set, and professional recommendations for future work.

Projects | Nick Mitchinson

Mon, 19 May 2025 09:00:00 -0800

Preview# Preview is a SaaS application for generating and managing dynamic and ephemeral Preview environments. Production-like environments are created to track every Pull Request, or manually for demos, manual testing, or training. MergeDeps# MergeDeps is a Github application for managing Pull Request dependencies and avoiding accidental premature merges. Redirector# Redirector is a hosted application for managing and servicing permanent or temporary redirects from one domain to another. RecruitMe# RecruitMe is a hosted application to filter inbound recruitment emails to only serious opportunities.

Streaming Build Logs

Mon, 11 Jul 2022 00:00:00 -0700

The PreviewHQ product is a hosted ephemeral Preview Environment service. It contains both an internal build and deployment service, and logs from these services need to be available to users. The initial implementation of these log streams was based entirely on Kubernetes pod logs. On a request for logs, the application backend: Queried for the name of the Kubernetes pod that ran that workflow step Created a kubernetes client to tail the logs of that pod Filter out any internal application logs Stream each log to the client While this approach was relatively simple to implement it had a number of core issues:

Debugging Mystery 502 Errors in Kubernetes

Mon, 04 Jul 2022 00:00:00 -0700

For a while, under any real load, one of my services was returning 502 errors at an ~0.5% rate. This is a service that is run in Kubernetes, with an Nginx Ingress Controller, and a Flask python application being run with uWSGI. I was expecting these errors to be fairly easy to identify because I generate Request IDs in nginx (added as an X-Request-Id HTTP header to both the downstream request and upstream response), and add this ID to the request-context logger in the application.

DigitalOcean Connection Pool Metrics

Sat, 25 Jun 2022 08:38:32 -0700

DigitalOcean is a player in the Infrastructure-as-a-Service (IaaS) market, competing with options such as Amazon Web Services (AWS), Google Cloud Platform (GCP), as well as other “low cost” providers such as Linode. Amoung these services, DigitalOcean has their “Managed Database” offering. One of the downsides of this offering is the terribly low connection limits, particularly on Postgres. In order to address this, DigitalOcean offers managed Connection Pools as well. Details of how a postgres connection pool works is outside the scope here, however as a brief summary the connection pool accepts a number of client connections, and then load balances upstream server connections between the clients.

Software Engineer Leveling

Tue, 01 Feb 2022 18:00:00 -0600

Software leveling is a complicated debate. What is a “Junior” Engineer? At what point is someone “Senior”? Every company has a slightly different outlook on this, and this is just my opinion; it’s the mental model I use. Junior Software Engineer# As a Junior Software Engineer, an individual would be primarily getting small-to-medium tasks delegated by their manager (or team lead). The expectation should be that this engineer will require assistance with these tasks as they become familiar with the environment, tools, and grow into their role.

Update: Managing Kubeconfig Files

Sun, 06 Jun 2021 14:34:50 -0500

I while back I wrote about how I managed Kubeconfig files in my environment, in a easy-to-modify way. I’ve changed my approach recently to make this easier to use; requiring a KUBECONFIG_MANUAL environment variable in order to avoid an explicitly-set KUBECONFIG being overwritten was error-prone and ergonomically poor. I’ve recently decided to take a different approach, with the assumption that I will never explicitly set a KUBECONFIG environment variable using multiple files.

Capability vs Task Based Scheduling

Mon, 03 May 2021 17:40:01 -0500

Container (or deployment) orchestration is the automation of taking your application, and placing it on infrastructure (typically a machine or VM) to execute as part of your larger application, while ensuring that the workload continues to run through a variety of potential issues. Scheduling, a subset of orchestration, determines which node (or nodes) in your infrastructure the application should run on. Controlling this scheduling is often necessary in order to ensure that scheduled applications are capable of running.

Introducing Hookshot

Tue, 27 Apr 2021 12:15:23 -0500

As the third project in my 12 Startups in 12 Months year, I’m announcing Hookshot. Hookshot is a managed platform for Webhook delivery. It is designed to handle the full lifecycle of a Webhook implementation, from allowing the end-user to create, manage, and inspect events, to ensuring reliable and secure delivery across a variety of potential issues. Implementing a reliable webhook delivery system on your own feels like it should be trivial, however increases in complexity as you grow.

Using Calendly to Get Kittens Adopted

Fri, 23 Apr 2021 16:12:13 -0500

This is the first post in my new “Everyday Automation” series, which explores using free tools and services to make every day life just a little bit easier. As a bit of background, through 2020 I have been fostering kittens. While fun and entertaining, the process of getting them actually adopted during COVID can be time consuming. The program we foster through switched to virtual Meet and Greets, which means when a litter of kittens are ready to be adopted, we have a huge flurry of interest all at once.

Introducing: Everyday Automation Series

Mon, 19 Apr 2021 22:19:07 -0500

Programming (or coding) is an extremely powerful skill that is becoming more important each year. There is a misconception though: that the benefits of these skills are always trapped behind learning a programming language, and writing your own software. Automation, one of the large benefits that programmers are able to utilize, can be accomplished in every day life using readily available tools. The key is to think about the things that you are doing on a regular basis, and finding an already built tool that you can use to solve this problem.

Announcing Recruitme

Sat, 02 Jan 2021 23:13:41 -0600

As the second project in my 12 Startups in 12 Months year, I’m releasing (sort of) RecruitMe. I’m a full time software engineer, and week-to-week I’ll receive up ~5 cold-outreach opportunities from recruiters for new opportunities. Unfortunately the majority of these are clearly not a good fit, are impersonal, and are low-effort outreach from low-quality recruiters. If the person reaching out had taken the time to review me, it would have been clear that I would not be interested.

12 Startups in 12 Months

Wed, 16 Dec 2020 22:56:14 -0600

“12 Startups in 12 Months” has been a relatively popular goal for a little bit now. The idea is to give yourself 12 months to attempt to start 12 unique companies, in a “fail fast” approach, with the goal of finding something that sticks. This is, in my opinion, a good goal. It splits evenly into a year (1 concept per month), and gives you a set amount of time for each attempt.

Building Environment-Friendly React Apps

Mon, 14 Dec 2020 12:00:00 -0600

A core tenant of application promotion is that you are using the same version of your application between environments. If you have a version of your code running on staging, or in a Preview environment, and you want to promote that code to production, you simply deploy same image to production, with the production set of environment variables. With a populate-on-build system like Reacts’, you can not do this. The staging environment variables are baked in to the image, so it can not be deployed to production.

MergeDeps Time-based Blocking

Sat, 12 Dec 2020 13:40:25 -0600

Last month I released MergeDeps in order to allow developers and teams to easily blocked Pull Request merges based on dependencies. Today, I’m announcing that MergeDeps can now block merges based on time. Often times you may not want a PR to be merged before a specific date or time. Especially with a Continuous Deployment pipeline in place, you do not want to merge code into your main branch before you want it deployed.

All Your Code Are Belong to Me

Thu, 10 Dec 2020 08:00:00 -0600

Publishing of this post was postponed until the report vulnerability described here had been corrected. The core risk of software supply chain issues is still a widespread issue. tldr; don’t depend on un-pinned external projects controlled by someone who doesn’t work at your company. Especially as a core part of your payment processing infrastructure. Fun security-related thing that I thought I’d share, just so that people don’t forget the risk.

Product vs Project Management

Tue, 08 Dec 2020 11:34:39 -0600

The last couple days I’ve been involved in a couple unique conversations about the difference between Product and Project Management. As someone who does not work in either of these roles, but often works with people in these roles, these are just my opinions and expectations as an engineering looking in. I don’t expect to be 100% right, and recognize that these roles will have slightly different meanings in different organizations.

Mailchimp deleted my account with no warning or notification - 2020 Edition

Sat, 05 Dec 2020 17:57:08 -0600

I hate doing this. Apparently writing public blog posts and crossing my fingers (🤞🏼) is the best way to get a delete Mailchimp account recovered. This is a follow on from this post from 2018. And from this post in 2019. It sucks that I have to be the one writing the 2020 edition. tldr; Mailchimp deleted my account and all of my data without notification because they felt I wasn’t using their product enough.

Lime, Bird, Lyft: What the F*ck

Fri, 27 Nov 2020 18:58:37 -0600

There were decades of campaigns to get kids to wear helmets while riding bikes. Decades. 2018. 2017. 2015. 2010. 2000. 1990. Then Bird/Lime/Uber/Lyft/whoever all came in and were like ELECTRIC SCOOTERS FOR EVERYONE NO HELMET NEEDED and just threw all of that progress in the trash? I was out for a walk through downtown Austin, and I passed no-less-than 8 couldn’t-be-older-than-13-year-old children flying around on scooters without helmets. 2 of them quite literally could not even see over the handle bars.

Announcing MergeDeps

Fri, 20 Nov 2020 18:43:46 -0600

Today, I am announcing the initial launch of a new project, MergeDeps. In one line, MergeDeps allows a developer to specify a dependency for a Github Pull Request, and MergeDeps will ensure that the Pull Requests are merged in the correct order. These dependencies are specified directly in the Pull Request body, without any necessary changes to your code or workflow. Whether you are a solo developer working on multiple branches, or a larger team needing to ensure dependencies across teams, MergeDeps is designed to easily and seamlessly address your coordination issues.

One Simple Trick for Building Images Faster

Sat, 17 Oct 2020 10:49:15 -0500

First off, I apologize for the clickbait title. It hurt me just writing it. A common step in a Continuous Integration/Continuous Delivery (CI/CD) pipeline is building container images. Fast image building is heavily dependent on being able to use a layer cache, which is many cases happens by default. The layer cache is what allows a docker build to skip a complex or long-running build step, by instead using the already-built layer.

Handling Remote Salaries

Wed, 14 Oct 2020 18:24:46 -0500

Large tech organizations are announcing permanent “work-from-home” policies. Companies are announcing that they will be “remote first”. As employees chose to move out of high cost of living areas, companies are introducing compensation adjustments. This is a very complex topic. Overall, I feel that location-adjusted compensation for location-independent work is, at best, fundamentally unfair to employees, and at worst, discriminatory. Note that I am focusing on remote-within-the-United-States. When looking internationally, there are more variables at play, which makes it a more complex discussion.

State of Continuous Product Review Tools

Mon, 12 Oct 2020 12:37:47 -0500

Disclaimer: I am currently building a product in this space at Preview. As such, I have biased opinions regarding the “correct” way of addressing Continuous Product Review needs. Preview is explicitly not included in this posting. This article is meant to be a summary of existing tools in the space, and not a comparison between tools. Over the last year, there as been an explosion of new tools and products in the Continuous Product Review space.

Working Around Kubernetes Sidecar Shutdowns

Fri, 04 Sep 2020 13:38:57 -0500

Over 2 years ago I wrote a quick Kubernetes controller in order to ensuring that “sidecar” containers were shut down after the “main” container in a pod exited. The issue we were seeing was fairly straight forward: we had a container running in a pod to accomplish some application logic, as well as a number of “sidecar” containers to provide some functionality to the application. Examples of this are log forwarding, sql proxies, networking proxies, or in our case, metric collectors.

Non-Terminating Namespace Removal

Thu, 03 Sep 2020 00:00:00 -0500

Namespaces are a core component of the Kubernetes landscape which are often used as as a base level of resource isolation. As a resource which contains multiple others, the shutdown behaviour associated with terminating namespaces is complex. In effect, namespaces can often get “stuck” in a Terminating state. I ran into this recently with a variety of namespaces in a Digital Ocean Kubernetes cluster. Much advice on the internet will recommend editing the namespace to remove any finalizers on the object.

Yes, Kubernetes "Limits" Are Important

Wed, 02 Sep 2020 19:08:00 -0500

Early today an engineer at Buffer put out a post about removing resource limits from a subset of their Kubernetes deployments in order to “make their services faster”. In summary, they were experiencing a kernel bug in which cpu throttling was being applied to containers which had not yet hit their CPU limits. They determined that this bug did not have an impact if no limits were set. This bug has since been fixed, and backported to many current linux versions, however they are still using a lack of limits to provide greater performance for some of their user-facing features.

Apple's Market Dominance

Thu, 13 Aug 2020 16:43:51 -0500

Note that while Google took similar action, I will be focusing primarily on a discussion of Apple’s market dominance and anti-competitive behaviour, rather than Google’s. This is, honestly, because much of this post was written before Google took action. There are many things at play here, and they are conflated. Early today Epic Games fired the first shots to kick of a war with Apple and Google over their dominance of their respective platforms.

WeChat Executive Order is Not About National Security

Mon, 10 Aug 2020 11:30:59 -0500

On August 6th, 2020, Donald Trump signed an Executive Order prohibiting any transaction related to WeChat within the United States after September 15th, 2020. It was claimed to be in the name of “National Security”, however it feels more like a targetted attack to reduce day-to-day communication between the US and China. This is a follow on from the Executive Order targeted at TikTok. There are similiarities here, but there are some key differences between the two.

Going Political

Sun, 09 Aug 2020 12:02:21 -0500

When I first tried to start writing more, I purposely attempted to avoid political topics. Those days are over. I figured that the political topics were totally disjoint from the technical side of things that I was trying to talk about, and my opinions on political topics were not necessary. I don’t believe that assumption holds any more. Technology is being more and more a political game as it’s influence on the world expands outwards.

Avoiding Log Loss from Short-Lived Job Containers

Mon, 25 May 2020 11:44:03 -0500

In multiple situations and clusters I’ve encounted the problem of logs being lost from short-lived containers; ie, a container which is dynamically spun up to complete a single job, and then exiting. These containers will often only exist for a couple of seconds, which makes effectively collecting and forwarding their logs challenging. There are two main log-collection methods in Kubernetes: using a log-forwarding sidecar (1), or a node-based collector agent watching log locations on the host machine (typically under /var/containers/logs) (2).

The Facebook-Giphy Data Play

Wed, 20 May 2020 00:00:00 -0500

On May 15th, 2020, Facebook announced that it had reached an agreement to purchase Giphy, for $400M. To many, this seems like a staggering amount of money for something that just serves GIFs. Facebook’s main business is in advertising, and GIFs are not inherently valuable for advertising purposes. There may be some possible value, such as Giphy selling “preferred placement” of specific GIF themes in searches, but that risk doesn’t seem to warrant the $400M price tag.

COVID-19 Impact on Restaurant and Bar Industry

Sun, 17 May 2020 20:02:29 -0500

Note that I have no experience in the restaurant industry, and these are my thoughts as pure outsider. If you’re looking for an opinion with hard facts backing it up, you’re not going to find it here. Over the past couple of days, I have been seeing a lot of discussion about how COVID-19 is the beginning of the end for the restaurant and bar industry. It is no secret that currently, even after “reopening”, government enforced social distancing is causing restaurants to struggle.

Instacart and their Unsustainable Model

Sun, 10 Nov 2019 13:15:39 -0600

Personally, I believe that Instacart is attempting to operate on a fundamentally flawed business model, and is doomed to fail unless the underlying model is changed. It is being kept afloat by external money, but will never be economically viable in a socially responsible way. If they can’t make enough margins to actually pay their employees, it’s not a viable business. Note that I have 0 background or knowledge in this specific sector.

CircleCI Tooling Wishlist

Sun, 15 Sep 2019 18:46:18 -0500

CircleCI is a great CI/CI solution for companies who do not want to build an internal expertise on running these systems. For small-to-medium startups (up to ~50 engineers), these is almost always a small tradeoff. Despite some issues I have brought up in the past, I still believe that CircleCI can be effectively used as long as it is setup and administered appropriately for the security concerns of the company.

Risking Private Key Exposure with CircleCI

Tue, 10 Sep 2019 00:00:00 -0400

Note that in order to avoid confusion with this separate issue, the publish date of this post was pushed back. On July 29th, 2019, I discovered what I consider to be a major security risk with the use of the CircleCI CI/CD product that should be unacceptable to any corporation or open source project. This article details how commonly used CircleCI features interact with GitHub security mechanisms in a way that exposes GitHub secret keys to malicious users.

Private Slack Channels

Mon, 01 Apr 2019 00:00:00 -0400

Many of us have been there. We interview at hot new companies, excited to join a small team building out a new product or platform. One of the major draws of joining a startup is being able to have a major contribution to the vision, and to have a level of transparency that you can’t get at a larger organization. You decide on one of these companies after hearing from everyone you spoke with at the company how open and transparent the organization.

Rebase After a Squash

Mon, 11 Feb 2019 20:30:14 -0600

Depending on your development workflow, rebasing may be a common component of your workflow; ie: You branch off master (we’ll call this branch A) You add a bunch of commits, for instance, developing a new feature You branch off again for a small fix (we’ll call this branch B) B1---B2 branch-b / A1---A2---A3 branch-a / M master Now, A gets merged into master. Typically (if keeping a clean git history), A was squashed into a single commit before merging.

Managing kubeconfig Files

Sat, 12 Jan 2019 00:00:00 -0600

Note: There is an updated version of this functionality described in a later post. If you deal with Kubernetes, you are most likely familar with the kubeconfig file, typically located at ~/.kube/config. This file typically contains the clusters, users, and contexts (combinations of clusters and users) that you use in order to connect to our Kubernetes environments (typically with kubectl). As well as using the default file at ~/.kube/config, there is also the option to specify a kubeconfig file using either the --kubeconfig argument, or the KUBECONFIG environment variable.

Linting Your Helm Charts

Thu, 20 Dec 2018 23:11:30 -0500

In any engineering environment, linting is an important tool to help maintain style, and prevent simple mistakes. The same is true of your Helm charts. Chart linting is an easy tool that you can add to your pipeline to ensure your deployments are valid and versioned correctly. Getting linting set up for your custom Helm charts is actually extremely straight-forward thanks to the Helm chart-testing project, which can be found here: https://github.

Identifying Kubernetes Evictions

Wed, 07 Nov 2018 13:28:30 -0500

For personal projects, I run a single node “cluster” on an OVH node, with kubeadm. Over the last couple months, I’ve been having repeated issues with pod evictions, which are typically not an issue, except when affecting critical pods, such as the kube-apiserver and kube-scheduler. Despite these being marked as critical pods, they were still being evicted due to disk pressure. After fighting with pod evictions for a while, it became clear that I wasn’t actually solving the underlying problem; this machine has 2TB of disk space… disk pressure evictions should not be occurring.

Using PGAnalyze

Wed, 23 May 2018 00:28:24 -0500

We recently set up PGAnalyze as a way of doing a quick database health check, and managed to get a couple quick wins out of it. By looking at the Query Performance section, we were able to quickly identify a bad outlying query which was accounting for ~7.5% of our total DB activity. Furthermore, we were able to see that this query was loading ~7.65 GB of data per call, with the majority of the data needing to be loaded from disk (rather than buffer cache), and was taking almost a full minute on each execution.

Swapping ALB Request IDs for Nginx

Wed, 04 Apr 2018 00:00:00 -0600

I’ve recently been making a switch from using Amazon’s Application Load Balancers (ALBs), to using their new Network Load Balancers (NLBs). There are many reasons for this switch (that are not the subject of this post), but there was one glaring short fall: We heavily rely on request log data centered around the ALB provided X-Amzn-Trace-Id Request ID value. When switching to the NLB (which is L4, vs the ALBs’ L7), this HTTP header is not added for us.

Recovering a Kubeadm Cluster

Fri, 23 Mar 2018 00:00:00 -0600

Recently a small cluster I maintain became unresponsive due to a failure of the kube-apiserver. Kubeadm clusters are currently limited to a single master, this meant that any interaction with the cluster was impossible. Due to interaction through the API being (obviously) impossible, troubleshooting and recovery required digging a bit deeper. By ssh -ing to the physical node, I was able to at least determine that the kube-apiserver was attempting to start, and then being killed because it was not responding to a livenessProbe in time.

Why Traffic Manipulation Capabilities Matter

Tue, 14 Nov 2017 20:34:54 -0600

Multi-tenancy systems are very common for a simple reason: it both saves costs, and reduces complexity. Unfortunately, the biggest downside is that in certain situations, a small group of clients can negatively impact the rest of the system. At Cratejoy, we run with a multi-tenant set up which is part SaaS, part Marketplace, and part Website Hosting. These 3 sections of our business have drastically different traffic patterns. Even within out Website Hosting, our individual merchants see extremely different traffic patterns.

Importance of Index Monitoring

Mon, 13 Nov 2017 19:13:34 -0600

Monitoring is a a large, and complex topic, so rightfully is a a lot written about it. One topic that is not covered often however, is effectively monitoring your database indexes. This particular aspect of monitoring is something that had not previously occurred to me as something important, however one incident was rather eye-opening.

Easy "Maintenance Mode" in Kubernetes

Sat, 11 Nov 2017 22:53:38 -0600

Despite our best efforts, systems sometimes require downtime for a variety of reasons. Different systems that I’ve seen have built this capability into different areas of their stacks, mostly in the application itself, or into the web server (such as nginx). Many of these solution still require the application to be running in order to serve an appropriate maintence page. At Cratejoy, this requirement proved problematic.

"PastaPassport": Why You Didn't Win (and why a bot may not have either)

Thu, 14 Sep 2017 20:02:56 -0500

Earlier today, Olive Garden ran their annual PastaPass promotion. I’m going to assume you know what this is (or else why else are you reading this?), but just incase you don’t, during this promotion, the Olive Garden restaurant chain sells (this year) 22000 ‘PastaPasses’, each of which entitle the holder to unlimited soup, breadsticks, and pasta during their ‘Never Ending Pasta Bowl’ season. This year, that time period is 8 weeks long, so if you plan on eating alot of pasta (or live near an Olive Garden and want to save some money on food), it is clear why getting one of these $100 ‘passes’ is a good investment.

Correcting Kops Etcd

Wed, 16 Aug 2017 23:38:46 -0500

Recently, I attempted to downsize a staging cluster from a 3 master HA setup, to a single master. The HA setup was unnecessary, and this was done primarily as a cost saving measure. Unfortunately, this ‘quick cost cut’ ended up taking multiple hours to clean up.

Separating a Monolith while Keeping Commit History

Wed, 31 May 2017 20:55:12 -0500

As small companies grow, and their engineering teams expand, they will often wish to begin splitting a large, monolithic repository into smaller service repositories. There are two naive approaches to this, neither of which we wanted to do: 1. Copy into a new repo# This would simply involve copying the files related to the service into a new repository, and then removing them from the original. This solution will totally work, however you will lose all history in the migration.

Killing a Service with Healthchecks

Fri, 05 May 2017 14:43:41 -0500

If you’ve been reading anything I’ve written lately, you’ll be aware that we at Cratejoy have recently migrated out infrastructure into a new Kubernetes deployment. Throughout this process, we hit a couple rather embarassing snafus, and this was just one of them.

Long story short, we accidently DoS’d some of our own services, and attempting to solve the problem by scaling horizontally only made the problem worse.

Migrating to a VPC

Tue, 28 Mar 2017 19:47:25 -0500

There are many advantages to running your infrastructure inside a VPC on AWS, as opposed to running in general EC2-Classic. While EC2-Classic is basically one giant, shared VPC, isolating your resources into your own VPC gives you greater control over internal communication, networking, and security.

We recently made a full transition from EC2-Classic into a set of VPC’s.

Integrating PayPal - Some Things To Remember

Wed, 01 Mar 2017 00:00:00 -0500

Around August, I began implementing a full PayPal integration for our merchants, allowing any of them to beign accepting PayPal payments, in the exact same way as any of our other payment providers. This project actually went rather smoothly, and much better than my initial expections. Our requirements prevented us from using their newer REST API, however after spending a day getting familiar with their SOAP API, and writing a small wrapper lib for the functionality we needed, it was no longer a concern.

Assumption Verification with "Impossible" Bugs

Sun, 15 Jan 2017 11:35:57 -0600

Recently at Cratejoy, we found ourselves in a situation where we had an “impossible” bug, and a limited time to identify and correct the issue. For a bit of background, one of our merchants was about to begin a relatively large marketing event, and we had developed a new caching solution designed to address the traffic profile we were expecting. The Problem# Unfortunately, a few hours before the event, we started seeing reports of pages returning blank responses.

Effective Code Review

Wed, 28 Sep 2016 12:51:56 -0500

Code reviews are an important tool in every engineering organization, with lists of benefits just a quick Google search away. With that said, simply ‘doing’ code reviews is not always enough. It is important to ensure that teams are focused on conducting effective code reviews, and are not just going through the process in order to check a box.

Handling Timeouts in RQ

Wed, 06 Jul 2016 22:12:23 -0500

An important part of building scalable applications and systems is to offload non-urgent tasks to worker processes. We want to do this in order to limit the time a user is waiting on requests, and prevent our application servers from getting bogged down with non-urgent tasks. Recently I was migrating a selection of long running processes (data export operations to be specific) to background processes, with two main goals: Preventing request timeouts that some of our clients had been seeing Move this load off of our application servers Imagine my surprise when, after moving these tasks completely out of process, I found a JobTimeoutException in our logs.

Deduplication Failures with SQLAlchemy collection_class

Wed, 06 Jul 2016 21:42:23 -0500

As discussed in Failsafing Multitenancy in SQLAlchemy, we use a custom collection_class as a failsafe to protect against cross tenant data leakage in our environment. This was working very well, until we discovered a bug which sometimes caused objects to be dropped incorrectly. Ie, if we had a Customer with 2 addresses, one for merchant_1, and one on merchant_2, a request against merchant_1 would always correctly remove the address for merchant_2, however would sometimes also remove the address for merchant_1.

Failsafing Multitenancy in SQLAlchemy

Wed, 06 Jul 2016 21:42:08 -0500

At Cratejoy, we run a hosted, multi-tenant application, and for multiple reasons, certain data is centralized between our separate merchants. Merchant specific data should never be crossed, for obvious reasons, however our current setup led to simple engineering oversights to lead to exactly this. We were able to trace data leaks back to our use of joinload’s of SQLAlchemy backrefs on centralized models. Ie, we have a centralized model with a child relation to something that is store specific. When we joinload the child model, it includes the children across all merchants, not the specific one we want at the time. While we could manually filter this in most cases, we take these data leakages very seriously, and do not want an oversight by an engineer to so easily lead to data leakage. We took some time to develop a failsafe system that should prevent data leakage between stores, even in the event of missing filters.

Using Pow for Single Page Applications

Fri, 23 Jan 2015 23:38:29 -0500

I recently started building out the front ends of a project I’ve been working on, and spent longer than I’d like to admit getting Pow to serve it properly. Since I’ve built the backend purely as a RESTful API, the front end is a single page application which manages it’s own state. This works perfectly fine when hitting the root route (ie, /), however Pow fails when hitting any other routes. Seeing as the files don’t actually exist this is understandable, however Single Page Applications are becoming more popular, and it seems like this should be an quick fix.

What we need to do here is rewrite all URLs to serve index.html. This seemed simple enough, however getting Pow to do this was not as straightforward as I had hoped (admittedly probably due to a lack of experience with Ruby). I finally got it working, and thought I’d put some straight forward directions here for future reference.

Resume | Nick Mitchinson

Mon, 01 Jan 0001 00:00:00 +0000

Resume | Nick Mitchinson

Mon, 01 Jan 0001 00:00:00 +0000

Resume | Nick Mitchinson

Mon, 01 Jan 0001 00:00:00 +0000

Resume | Nick Mitchinson

Mon, 01 Jan 0001 00:00:00 +0000