Bitcoin and Litecoin payouts on BlockCypher's free plan: the rate limit, a leaked token, and no double payouts

A few of my Django projects pay withdrawals out in crypto, and Bitcoin and Litecoin (and Dogecoin) go out through BlockCypher, on its free plan. This is how a payout queue used up BlockCypher's hourly quota, how the API token got into the payment log, and what I changed so that no payout is ever built twice.

The setup

Withdrawals wait in a payout queue. Celery beat ticks every five minutes and fans the queue out into one task per withdrawal; it also refreshes the hot wallets' balances for the admin dashboard.

A payout through BlockCypher is three steps. POST txs/new returns a transaction skeleton with the hashes to sign (tosign); my server signs them with the wallet's WIF key, which never leaves it; POST txs/send broadcasts the result. The API token rides in the query string, as ?token=.

The free plan allows 3 requests a second, 100 an hour and 1,000 a day, the hour counted from the top of the hour in UTC. That is enough for a payout queue, as long as none of it is wasted.

What went wrong

• Two balance reads per payout. The payer checked the hot wallet before sending, then the sender read it again, only to print it.
• A refusal, retried every tick. A failed payout stayed queued and was tried again five minutes later, with the same requests.
• One tick's payouts, built on the same coins. Several payouts from one wallet started together. BlockCypher sent the first and refused the rest with unable to lock an input ..., already locked by ..., and each refusal still counted.
• Limits reached. until the hour turned. After the first refusal for the limit, BlockCypher refuses every request until the window resets, and counts those too. Retrying every five minutes just kept the quota spent.
• The token in the payment log. An HTTPError from requests names the URL it failed on, ?token= and all, and that message went straight into the payment log every admin reads, and to the console.

The fixes

The token never leaves in a text

Every error that is kept (the reason for the last failed balance read, which last_balance_error hands to the dashboard) or printed goes through redact() first. It masks the token=… pattern and the raw token value wherever else it shows up:

_TOKEN_IN_TEXT = re.compile(r"(token=)[^&\s'\"]+")


def redact(text) -> str:
    """`text` with the API token taken out."""
    text = _TOKEN_IN_TEXT.sub(r"\1***", str(text))
    if BLOCK_CYPHER_API_TOKEN:
        text = text.replace(str(BLOCK_CYPHER_API_TOKEN), "***")
    return text

Over the limit, nothing until the next hour

A 429, or Limits reached. anywhere in the response body, pauses BlockCypher until a minute past the next hour. The pause lives in Django's cache (Redis here), so every Celery worker sees it:

def _is_over_the_limit(status_code, body) -> bool:
    if status_code == 429:
        return True
    text = json.dumps(body) if isinstance(body, (dict, list)) else str(body or "")
    return "Limits reached" in text


def pause_blockcypher() -> datetime:
    """Ask BlockCypher nothing until a minute past the next hour."""
    current = datetime.now(timezone.utc)
    until = current.replace(minute=0, second=0, microsecond=0) + timedelta(
        hours=1, minutes=1
    )
    seconds = int((until - current).total_seconds()) + 60
    cache.set(PAUSE_CACHE_KEY, until.isoformat(), timeout=seconds)
    return until

The balance read, the fee quote and the send all check blockcypher_paused_until() first and ask nothing while it is set. So does the payer: the withdrawal keeps its place in the queue and its locked funds, and the log says when it will be tried again.

Fewer requests

• A wallet's balance is kept for BALANCE_FRESH_SECONDS, nine minutes: a little under two ticks. With a five-minute beat that is a read every other tick, six requests an hour per wallet instead of twelve.
• After a successful send the kept balance is dropped (forget_blockcypher_balance), because the number from before the send is no longer true.
• The send itself reads no balance any more. A payout is exactly two requests: txs/new and txs/send.
• An error is one line. _errors_of keeps what BlockCypher said, not the whole skeleton with its inputs and outputs.

A send that may have gone out is held, never rebuilt

This was the part that mattered most, because "not sent" is not one state. If txs/send reached BlockCypher and the answer was lost (a timeout, a dropped connection), or came back as a 5xx, I can't tell whether the transaction was broadcast. Leave that payout in the queue and the next tick builds a second transaction; once the first one mines, the user has been paid twice.

# For a segwit spend the hash does not change with the signatures.
skeleton_hash = (tx_skel.get("tx") or {}).get("hash")
# ... sign each of tx_skel["tosign"] ...
try:
    r2 = requests.post(f"{base_url}/txs/send", json=tx_skel, ...)
except requests.RequestException as e:
    # It may have reached BlockCypher and only its answer been lost.
    return (
        False,
        skeleton_hash,
        {"stage": "send", "error": redact(e), "maybe_sent": True},
    )
res = _body(r2)
if r2.status_code >= 500:
    return (
        False,
        skeleton_hash,
        {"stage": "send", "error": _errors_of(res), "maybe_sent": True},
    )
if r2.status_code >= 400 or "errors" in res or "error" in res:
    # Refused, not sent: an input locked by another transaction, the limit.
    if _is_over_the_limit(r2.status_code, res):
        pause_blockcypher()
    return False, None, {"stage": "send", "error": _errors_of(res)}

The skeleton's hash comes from the txs/new answer. The sender is written for BIP84 (bech32) wallets, and in a segwit spend the signatures live in the witness, outside the data the txid is computed from, so the skeleton's hash is the hash the transaction will have if it mines. For a legacy input that would not hold.

send_btc and send_ltc pass the hash up with the failure, and the payer parks the withdrawal on it. In the project that runs this queue, receipts are reconciled automatically only on the EVM chains, so a parked Bitcoin payout waits for an admin to look the hash up in an explorer and settle it by hand. A few minutes of an admin's time is cheaper than paying twice.

One payout at a time per chain

BLOCKCYPHER_PAYOUT_CHAINS = ("btc", "ltc", "doge") names the chains whose payouts BlockCypher builds from one wallet's coins. On those chains the payer takes a slot in the cache before it sends:

SERIAL_PAYOUT_GAP_SECONDS = 4 * 60


def reserve_serial_payout(chain) -> bool:
    if chain not in cs.BLOCKCYPHER_PAYOUT_CHAINS:
        return True
    return cache.add(SERIAL_PAYOUT_KEY.format(chain), True, SERIAL_PAYOUT_GAP_SECONDS)


# ... in pay_withdrawal, after the send:
if result:
    complete_withdrawal(withdrawal, txid)
    return True
if txid:
    # Broadcast but unconfirmed. Held, never re-sent.
    park_withdrawal_in_flight(withdrawal, chain, txid, msg)
    return False
# Nothing went out: the request keeps its place in the queue.
log_withdrawal_error(withdrawal, f"Payment service failed: {msg}")
return False

cache.add only writes when the key is absent, so of several tasks running at once exactly one wins. The slot is never given back; it lapses after four minutes. With a five-minute beat that makes one payout per chain per tick, and the next one finds the previous payout's change in the wallet instead of the same locked input.

How the tests pin it down

tests/test_blockcypher_limits.py makes no real requests: requests.get and requests.post are replaced with MagicMock, and the token is a dummy value. Each fix above has a test that counts calls or reads the returned tuple: the token is absent from kept errors and from capsys output; after a 429 nothing more is asked (get.call_count == 1); a payout is two POSTs and no GET; a refused send holds no hash, while a timeout or a 502 hands back the skeleton's. One of them:

def test_a_send_whose_answer_was_lost_hands_its_hash_back():
    """Perhaps sent: a payout built again would pay twice once this one mines."""
    (ok, tx_hash, detail), _post, _get = _send(
        [_answer(201, SKELETON), requests.Timeout(f"read timed out ?token={TOKEN}")]
    )

    assert (ok, tx_hash) == (False, SKELETON_HASH)
    assert detail["maybe_sent"] is True
    assert TOKEN not in detail["error"]

What I took from it

• Around an API that moves money, failure has three states: sent, refused, unknown. Retry the refused; hold the unknown, on an identifier you got before the irreversible step, here the skeleton's hash.
• A secret in a URL ends up in a log. HTTP libraries put the URL in their exceptions, and those climb until someone reads them. Strip it at the client boundary, and keep a test on it.
• A quota is shared state. It belongs in the shared cache, not in one worker's memory. When the provider says stop, stop until its window turns; retrying only burns the window.
• Count each operation's requests and pin the number in a test. A print of the balance looks harmless until it turns out to be one of the hour's hundred.