Bitfinex Lending Bot Dev Notes, Part 2: Nonce, a Nasty Final Boss

From late January 2026, the log kept showing nonce: small coming back from Bitfinex. It took two rounds of fixes before it was really fixed, and two API keys had to be replaced because of it. The most absurd version along the way pushed the nonce 180 days into the future.

What a nonce is

Every call to a Bitfinex API that needs your API key (checking the wallet, placing offers, cancelling them) has to carry a nonce, a number that can only keep getting bigger. Bitfinex remembers the largest nonce it has received for each key, and if the next one isn't bigger than that, it replies:

nonce: small

The usual approach is to use the current time (in microseconds) as the nonce; time only moves forward, so the number naturally keeps growing.

Symptoms

I had planned a log system for easier debugging from the start, and the log kept reporting nonce errors.

What the git history shows:

• On January 5, a "strictly increasing" guarantee was added: within one client, each nonce generated is always bigger than the previous one
• The January 24 commit has a line about not letting nonce: small count as a disconnect, which means that at the time this error was treated as a lost connection and made the bot switch itself to offline

What I tried

The first round of fixes ran from January 24 to February 6, about two weeks, and the approach was: whenever nonce: small came back, push every later nonce forward by some amount of time (an offset). AI kept raising these numbers on its own:

1. January 24: add 5 seconds per error (didn't work)
2. February 2, 10:11: changed to 60 seconds each time; the commit says "5 retries can jump forward up to 5 minutes" (didn't work)
3. February 2, 10:26: start 5 minutes ahead as soon as the bot launches, on the reasoning that "after a container restart, the server may still remember a higher nonce from the previous session" (didn't work)
4. February 2, 10:32: start 10 minutes ahead and add 2 minutes each time; later that same day, changed to add 10 minutes each time (didn't work)
5. February 3: start 7 days ahead and add 1 hour each time (didn't work)
6. February 6: start 180 days ahead and add 1 day each time, and save the offset to a file so it carries over after a restart (didn't work)

By this point, the nonce being sent to Bitfinex was a time half a year in the future.

Why it only got worse

A nonce can only go up, never down. Once the largest value Bitfinex remembers has been pushed 180 days ahead, every nonce that key sends from then on has to beat it. Switch the code back to using the current time as the nonce and it gets rejected every time, so the only option is to replace that key.

The first time it happened, I was the only one using the bot, so changing the key was no big deal; the second time, I had already asked a friend to join the testing, so I had to ask them to replace their key too.

And if two counters are using the same key at the same time, adding offsets can't save you: A jumps ahead, so B becomes the smaller one and gets rejected; B jumps ahead, and now A gets rejected. Every rejection adds more, and the number just keeps piling up.

The causes guessed in the commits at the time were "multiple components competing for the same client" and "the server remembering an old nonce after a restart", and the response was to keep adding offsets, without checking who was actually competing.

The real cause

The second round of fixes was April 7 to 10, 4 days:

1. April 7: removed the offset and switched to the official Go SDK's approach. At startup, use "the current time in seconds × 1,000,000" as the starting point, then +1 each time; only save to the file after a successful request, with a separate file for each key
2. April 9–10: found the root cause. Two places in the code each kept their own cache of clients (the scheduler manager had one, and the multi-user client factory had another), so the same key would end up with two clients, each with its own nonce counter
3. At the same time, the 4 paths that generate nonces (regular API calls, WebSocket login, fetching user info, fetching key permissions) were all switched to the same counter
4. nonce: small was taken off the automatic retry list; it isn't a network problem, and retrying just runs into it again and again
5. Requests from the same client were made to queue up and go out one at a time. When requests are sent at the same time, the one with the bigger nonce can arrive first, and the smaller one that arrives later gets rejected. A round is about 11 requests at 200–400 milliseconds each, so with queuing a round takes 3–4 seconds, which is actually faster than all the retrying before

There's no way to tell from the git history anymore whether the errors back in January and February had this same cause. The code back then also had a separate executable for a high-frequency mode, which created its own client too.

It happened twice more

After the fix, the problem of a second client being built for the same key came up twice more:

• May 19: I wrote a one-off reconciliation tool that called the client factory directly to build a client, bypassing the scheduler's cache. Running it while the bot was up made Bitfinex start rejecting every request, and the web UI was down for about 5 minutes until a restart. Afterwards the tool was deleted entirely, and the client factory's documentation now names the only two places allowed to call it
• August 21: while reworking the flow for saving API keys, AI wrote "refresh the cache when saving the key fails", which would again build a second client for the same key. This time it was caught in the codex review and never went live

Honestly, I'm completely unfamiliar with backend nonce mechanisms like this, so there was no experience I could draw on for a bug like this (truth is, I couldn't be bothered to read the code anymore either); all I could do was rely on AI to try things and solve it. But maybe because I had designed a log system for troubleshooting from the start, this bug, serious as it was, didn't take too long to fix.