Instance Capacity Fallback: A Tested Design for Shortages

TL;DR

Problem: A provider cannot supply the instance type you asked for, and your provisioner has no safe second choice.
Pattern: Walk an ordered list of acceptable pools, cool down failed ones, and create the replacement before destroying the old server.
When to use: Autoscalers, CI runners, and any code that launches or replaces servers on demand.
Trade-off: Fallback can cost more and run on slower hardware, so the list needs limits and alerts.

Instance capacity fallback means your provisioner tries an ordered list of acceptable instance types and zones when the first choice is unavailable. Fall back only on capacity errors, cool down failed pools, cap attempts, and always create the new server before destroying the old one.

— Ashutosh Tiwari, NexGismo

Instance capacity fallback is a resilience pattern for infrastructure code. A launcher holds an ordered list of equivalent compute pools, such as instance type and zone pairs. It moves to the next pool only when the current one reports a capacity shortage.

Instance capacity fallback is the part of a provisioner that nobody tests until a shortage arrives. In October 2026 two public postmortems showed what happens when it fails. A defect in fallback logic blocked alternate instance types, and a rebuild destroyed servers it could not replace. Both teams published their fixes.

This guide uses those two reports, plus AWS documentation, to design a launcher that survives shortages. It includes a tested Python module you can adapt. All code here was run in a sandbox; the results are shown below.

What went wrong in the October 2026 capacity incidents?

Two unrelated teams hit the same class of failure within weeks: a provider could not supply the exact server they asked for, and their own code made it worse.

CircleCI published a report on five job-start incidents between October 1 and 8, 2026. Per the report, three were driven by limited cloud-provider instance supply. On October 8, from 12:48 UTC, the provider lacked the requested instance type. A defect in the fallback logic blocked use of other instance types in the primary region. The team deployed a fix at 14:49, after which the service began requesting other instance types and starting machines in large numbers. Waits had reached up to 50 minutes on Linux machine and remote Docker jobs.

The second report is a public GitHub postmortem from the event-junkie project. It says a production apply meant to add two DNS records instead replaced both servers. Hetzner could not place the replacement server types, so production was rebuilt on different types. The site was down for about 40 minutes, from 11:34 to about 12:14 UTC on October 28, 2026. The monthly bill rises from €33.21 to €91.76 until production returns to the original types.

The common thread: capacity is a runtime condition, not a configuration constant. Code that assumes “the type I want exists” fails the first time that is false.

Why does a naive launcher fail when capacity runs out?

A naive launcher fails because it treats capacity errors as exceptional. It tries one pool, raises on the first refusal, and either gives up or retries the same pool.

AWS documents the error plainly. The EC2 troubleshooting guide says you get InsufficientInstanceCapacity when AWS does not currently have enough On-Demand capacity for your request. Its listed responses are: wait a few minutes and retry because “capacity can shift frequently”, request fewer instances per call, submit the request without an Availability Zone, or use a different instance type.

Three failure patterns follow from ignoring this advice.

Single pool. The code names one type in one zone. Any shortage there is an outage.
Broken fallback. A list exists, but a bug prevents it from being used. CircleCI’s October 8 defect is an example, and nothing exercised the path until a real shortage.
Destroy first. The code removes the old server before the new one exists. The event-junkie rebuild did this, and the postmortem names destroy-before-create ordering as a contributing factor.

How should an instance capacity fallback be designed?

Design it as an ordered list of pools, a rule for when to move on, and a limit on how long to keep trying. Five steps cover it.

Define equivalent pools. List instance type and zone pairs your workload can run on, best first. Include different families and at least two zones.
Classify errors. Fall back only on capacity errors. Authentication, quota, and validation errors must surface, because another pool will not fix them.
Cool down failed pools. After a capacity error, skip that pool for a short time. This stops repeated calls to a pool that just said no.
Cap attempts. Bound the number of real launch calls per request, then fail loudly with the full attempt log.
Create before destroy. When replacing a server, launch the new one first. If no pool has capacity, the old server stays running.

The data path for one launch request looks like this:

request -> launcher.walk(pools in preference order)
             |-- pool cooling down?  -> record "skipped", next pool
             |-- provider.launch(pool)
             |      |-- ok               -> return instance + attempt log
             |      |-- CapacityError    -> start cooldown, next pool
             |      |-- PermanentError   -> raise (no fallback)
             '-- list exhausted or attempt cap hit -> raise NoCapacity(log)

AWS describes the same idea for managed fleets. The AWS Knowledge Center advises that you “be flexible about the instance types that you request and the Availability Zones that you deploy”, and lists the capacity-optimized and price capacity optimized allocation strategies. If you use Auto Scaling groups or Spot Fleet, prefer those built-in strategies over hand-written fallback. Write your own walker where the platform does not do it for you.

What does working capacity fallback code look like?

The module below is Python 3.10+ and uses only the standard library. The provider is a protocol, so you can plug in boto3, a Hetzner client, or a fake.

"""Capacity fallback walker. Python 3.10+, standard library only."""
from __future__ import annotations

import random
import time
from dataclasses import dataclass, field
from typing import Callable, Protocol


class CapacityError(Exception):
    """The pool has no capacity right now. Safe to try another pool."""


class PermanentError(Exception):
    """Bad credentials, quota, invalid request. Another pool will not help."""


@dataclass(frozen=True)
class Pool:
    instance_type: str
    zone: str

    def __str__(self) -> str:
        return f"{self.instance_type}@{self.zone}"


class Provider(Protocol):
    def launch(self, pool: Pool) -> str: ...


class NoCapacity(Exception):
    def __init__(self, attempts: list[tuple[str, str]]):
        super().__init__(f"no capacity in any of {len(attempts)} pools")
        self.attempts = attempts


@dataclass
class FallbackLauncher:
    provider: Provider
    pools: list[Pool]                      # ordered by preference
    cooldown_s: float = 60.0               # how long a failed pool is skipped
    max_attempts: int = 8
    clock: Callable[[], float] = time.monotonic
    _cool: dict[Pool, float] = field(default_factory=dict)

    def launch(self) -> tuple[str, Pool, list[tuple[str, str]]]:
        attempts: list[tuple[str, str]] = []
        tried = 0                          # real launch calls; skips are free
        now = self.clock()
        for pool in self.pools:
            if tried >= self.max_attempts:
                break
            if self._cool.get(pool, 0.0) > now:
                attempts.append((str(pool), "skipped: cooling down"))
                continue
            tried += 1
            try:
                instance_id = self.provider.launch(pool)
            except CapacityError:
                self._cool[pool] = self.clock() + self.cooldown_s
                attempts.append((str(pool), "capacity"))
                continue
            # PermanentError and anything else propagate: do not mask real bugs.
            attempts.append((str(pool), "ok"))
            return instance_id, pool, attempts
        raise NoCapacity(attempts)


def replace_instance(launcher: FallbackLauncher, old_id: str,
                     terminate: Callable[[str], None]) -> tuple[str, Pool]:
    """Create-before-destroy: the old instance dies only after a new one exists."""
    new_id, pool, _ = launcher.launch()   # raises NoCapacity: old instance untouched
    terminate(old_id)
    return new_id, pool

Three details matter. First, only CapacityError triggers fallback; every other exception propagates, so real bugs are never hidden. Second, the attempt cap counts real launch calls, not skipped pools, so a long cooldown list cannot starve the walk. Third, replace_instance calls terminate only after launch returns.

If a provider call can time out after succeeding, use an idempotency token. NexGismo’s idempotency keys guide covers the pattern, and the retry storm prevention post explains how to bound retries so they do not add load.

How do you test that the fallback really works?

Test it with fault injection, because production shortages are too rare to rely on. A fake provider raises CapacityError for chosen pools. The test then asserts which pools were tried and in what order.

import itertools, random, unittest
from capacity_fallback import *

class Fake:
    def __init__(self, available, permanent=()):
        self.available, self.permanent, self.calls = set(available), set(permanent), []
    def launch(self, pool):
        self.calls.append(pool)
        if pool in self.permanent: raise PermanentError("denied")
        if pool not in self.available: raise CapacityError()
        return f"i-{len(self.calls)}"

A = Pool("m6i.large", "a"); B = Pool("m6i.large", "b")
C = Pool("m5.large", "a");  D = Pool("c6i.large", "b")
ORDER = [A, B, C, D]

class T(unittest.TestCase):
    def test_prefers_first_pool(self):
        f = Fake(ORDER); _, pool, _ = FallbackLauncher(f, ORDER).launch()
        self.assertEqual(pool, A); self.assertEqual(f.calls, [A])

    def test_falls_through_every_pool_in_order(self):
        f = Fake([D]); _, pool, att = FallbackLauncher(f, ORDER).launch()
        self.assertEqual(pool, D); self.assertEqual(f.calls, ORDER)  # none skipped

    def test_exhaustion_raises_with_full_attempt_log(self):
        with self.assertRaises(NoCapacity) as cm:
            FallbackLauncher(Fake([]), ORDER).launch()
        self.assertEqual(len(cm.exception.attempts), 4)

    def test_permanent_error_is_not_masked(self):
        f = Fake([B], permanent=[A])
        with self.assertRaises(PermanentError): FallbackLauncher(f, ORDER).launch()
        self.assertEqual(f.calls, [A])

    def test_cooldown_skips_then_expires(self):
        t = [0.0]; f = Fake([B]); L = FallbackLauncher(f, ORDER, cooldown_s=60, clock=lambda: t[0])
        L.launch(); f.calls.clear(); t[0] = 10
        _, _, att = L.launch()
        self.assertEqual(f.calls, [B]); self.assertEqual(att[0][1], "skipped: cooling down")
        f.calls.clear(); t[0] = 61; L.launch(); self.assertEqual(f.calls[0], A)

    def test_replace_keeps_old_instance_when_no_capacity(self):
        killed = []
        with self.assertRaises(NoCapacity):
            replace_instance(FallbackLauncher(Fake([]), ORDER), "i-old", killed.append)
        self.assertEqual(killed, [])

    def test_replace_terminates_after_new_exists(self):
        killed = []; new, pool = replace_instance(FallbackLauncher(Fake([C]), ORDER), "i-old", killed.append)
        self.assertEqual(killed, ["i-old"]); self.assertEqual(pool, C)

    def test_property_any_available_pool_is_found(self):
        # exhaustive: all 16 availability combinations
        for mask in itertools.product([0, 1], repeat=4):
            avail = [p for p, m in zip(ORDER, mask) if m]
            L = FallbackLauncher(Fake(avail), ORDER)
            if avail:
                _, pool, _ = L.launch(); self.assertEqual(pool, avail[0])
            else:
                with self.assertRaises(NoCapacity): L.launch()

if __name__ == "__main__": unittest.main(verbosity=2)

We ran this suite with Python 3.13 in a sandbox: python3 -m unittest test_capacity_fallback. Result: Ran 8 tests ... OK. The property test walks all 16 availability combinations of four pools. That kind of exhaustive check is what catches a CircleCI-style bug, where one pool in the list is silently never tried.

Add an alert whenever a launch succeeds on anything other than the first pool, so a silent downgrade becomes visible. A staging game day that forces a refusal confirms the fallback fires.

What are the trade-offs of instance capacity fallback?

Fallback trades predictability for availability. Each choice below needs an explicit decision.

Cost. A fallback type can cost more. The event-junkie bill difference above is a real example of a forced type change.
No guarantee. AWS’s re:Post guidance states that flexibility does not guarantee capacity for the alternative type. It also says larger alternatives can raise costs and that the approach suits workloads that tolerate type changes.
Thundering herd. If many launchers fail together, they all move to the same second pool. Randomize order slightly or share cooldown state.
Stale cooldowns. A long cooldown wastes recovered capacity. Start with tens of seconds (an assumption, not a sourced figure) and tune it.

When should you not use instance capacity fallback?

Skip instance capacity fallback when the workload needs specific hardware features. The AWS re:Post article lists such workloads as a poor fit. Cluster placement groups are another case, since the EC2 guide names them as a cause of capacity errors. For those, reserve capacity ahead of time instead.

What is a hypothetical example of the fallback in action?

Consider a hypothetical CI runner service. This example is illustrative and not from a real company. It prefers m6i.large in zone a, then zone b, then m5.large in zone a, then c6i.large in zone b. A shortage hits m6i.large in both zones.

The launcher records a capacity error for each, starts two cooldowns, and launches m5.large in zone a. A second request seconds later skips both cooled pools without calling the provider. The on-call alert fires because the launch used the third pool, not the first.

How do the common approaches compare?

Approaches to handling instance capacity shortages
Approach	Handles shortage?	Main risk	Best for
Single type, retry only	Only if the shortage clears quickly	Outage when the pool stays empty	Prototypes
Custom ordered fallback (this post)	Yes, within your pool list	Bugs in rarely run code	CI runners, custom provisioners
Managed allocation strategies (capacity-optimized, price capacity optimized)	Yes, chosen by the platform	Spot-style interruptions	Auto Scaling groups, Spot Fleet
Reserved capacity	Yes, by prior commitment	Paying for idle capacity	Hardware-bound workloads

Interview questions this topic prepares you for

How would you design a provisioner that survives a capacity shortage?

Use an ordered list of equivalent pools across types and zones. Fall back only on capacity errors, cool down failed pools, and cap attempts. Replace servers create-before-destroy. Log every attempt and alert on launches that use a non-preferred pool. Mention that managed allocation strategies may cover part of this already.

Follow-up they may ask: How do you stop many launchers moving to the same fallback pool at once?

Why must a fallback only trigger on capacity errors?

Other errors, such as invalid credentials or exceeded quotas, are not pool-specific. Trying another pool repeats the same failure and hides the real problem. Classifying errors keeps the fallback honest and keeps incident logs clear. Unknown exceptions should propagate rather than being treated as capacity shortages.

Follow-up they may ask: What if the provider returns a generic 500 instead of a capacity code?

How do you test code that only runs during a shortage?

Inject faults with a fake provider and assert call order. Cover every availability combination, exhaustion, and permanent errors. Add a periodic game day in staging that forces a refusal. Without such tests, a defect can sit unnoticed until production runs out of capacity.

Follow-up they may ask: How would you detect a fallback that never fires?

The NexGismo verdict: My recommendation: treat the fallback list as production code with its own tests, not as configuration. Use managed allocation strategies where your platform offers them. Write a custom walker only for what they cannot cover, and always alert on non-preferred launches. The cheapest incident is the one a fault-injection test already rehearsed.

AWS documents InsufficientInstanceCapacity as a supply shortage, with four listed responses: wait, request fewer instances, drop the zone, or change type.
CircleCI’s October 8, 2026 incident ran from 12:48 UTC to a 14:49 fix, made worse by a defect in instance-type fallback logic.
Fallback should trigger on capacity errors only; every other error must propagate.
A replacement server must exist before the old one is destroyed, or a shortage becomes an outage.
Fallback does not guarantee capacity, and larger alternative types can raise cost.
An exhaustive fault-injection test over all pool combinations catches fallbacks that never fire.

Frequently Asked Questions

What is instance capacity fallback?

Instance capacity fallback is a provisioning pattern. When a cloud provider cannot supply the instance type you asked for, the system tries the next acceptable type or zone from an ordered list. It turns a hard launch failure into a slower or costlier success, within limits you set in advance.

What does InsufficientInstanceCapacity mean in EC2?

It means AWS does not currently have enough On-Demand capacity to meet your launch or start request. AWS documents several responses: wait and retry, request fewer instances per call, drop the Availability Zone, or use a different instance type. The error is about supply, not your account or permissions.

Should you retry the same instance type or switch types?

Do both, in order. Retry briefly in case capacity returns, then move to the next pool. AWS says capacity can shift frequently, but a pool that just failed is a poor bet for the next few seconds. A short cooldown per pool avoids hammering it.

Is it safe to replace a server before checking capacity?

No. Acquire the replacement first, then terminate the old server. A rebuild that destroys first can leave you with neither. In the October 2026 event-junkie postmortem, destroy-before-create left production down for about 40 minutes when the replacement type could not be placed.

How do you test capacity fallback logic?

Inject faults. Use a fake provider that raises a capacity error for chosen pools, then assert which pools were tried and in what order. Cover every availability combination, the all-pools-failed case, and permanent errors that must not trigger fallback. The tests should run in milliseconds in CI.

Capacity shortages are rare enough to forget and common enough to matter. The fix is modest: an ordered list, a clear rule for moving on, and a test that proves the list is actually walked. Start by adding one fault-injection test to your own provisioner this week. If it passes, add the alert. If it fails, you found the bug before the provider did.

For related reading, see NexGismo’s guides on retry storm prevention and idempotency keys.