What a WebSocket needs to survive the night

Almost everything written about WebSockets stops at onmessage. You open a connection, you log the frames, the tutorial ends, and everyone goes home.

Then you run it against a real feed for a week and discover that the interesting code is all in the part nobody wrote about. I've had a market data client in production for a while now, talking raw protocol over a socket with no client library in the way, and this is the set of problems that turned up. None of them are exotic. All of them will happen to you.

Open does not mean ready

The open event fires when the TCP and TLS handshake completes. Almost no real protocol is ready to use at that point, because there is an application-level handshake still to come.

The feed I work with wants five steps before it will send a single quote:

SETUP           (channel 0)  -> AUTH_STATE: UNAUTHORIZED
AUTH            (channel 0)  -> AUTH_STATE: AUTHORIZED
CHANNEL_REQUEST (channel 3)  -> CHANNEL_OPENED
FEED_SETUP      (channel 3)  -> FEED_CONFIG
FEED_SUBSCRIPTION (channel 3)

If your connect() resolves on the open event, every caller downstream believes it can subscribe, and their subscriptions land on a socket that has not authenticated. So resolve on the protocol being ready instead, which means holding a promise open across several round trips:

async connect(options: ConnectOptions): Promise<void> {
  this.authReady = new Promise<void>((resolve, reject) => {
    this.authResolve = resolve;
    this.authReject = reject;
  });

  this.ws = new WebSocket(options.dxlinkUrl);
  this.ws.addEventListener('open', () => this.send({ type: 'SETUP', channel: 0, /* ... */ }));
  this.ws.addEventListener('message', (ev) => this.handleMessage(String(ev.data), options.token));
  this.ws.addEventListener('close', (ev) => {
    this.stopKeepalive();
    if (!this.closedIntentionally) {
      this.authReject?.(new Error(`closed unexpectedly (code ${ev.code})`));
    }
  });

  await this.authReady;   // resolves several messages later
}

The resolve happens in the message handler, at the point the server confirms the feed is configured:

case 'FEED_CONFIG':
  this.startKeepalive();
  this.authResolve?.();
  this.authResolve = null;
  this.authReject = null;
  break;

Clear the handles after use. If the socket later closes, the close listener calls authReject, and rejecting an already-settled promise is a silent no-op that swallows the disconnect before anything can handle it.

The reconnect loop is the program

The structural mistake is treating disconnection as an error case bolted onto a happy path. Sockets close. Servers restart, tokens expire, laptops sleep, and somebody's load balancer has a thirty minute idle timeout nobody documented.

The shape that worked is an outer loop that owns the lifetime, where a single connected session is just one iteration:

while (!this.stopRequested) {
  try {
    await this.runSession();     // returns when the socket dies
    attempt = 0;
  } catch (err) {
    attempt += 1;
    const delayMs = Math.min(1000 * 2 ** Math.min(attempt, 5), 30_000);
    log.warn(`reconnect in ${delayMs}ms (attempt ${attempt})`);
    await sleep(delayMs);
  }
}

Exponential backoff capped at thirty seconds, with the exponent clamped separately so a long outage does not overflow into an absurd delay. Reset the counter on a clean session so a connection that survived four hours is not punished for a previous bad night.

runSession() does the whole job: fetch a fresh token, connect, subscribe, then block until something goes wrong. Everything above it stays running across reconnects.

A dead socket does not always say so

Half-open connections are the failure that cost me the most time. The remote end goes away, no close event ever fires, and your client sits there in readyState === OPEN receiving nothing, perfectly happy, for hours.

Two defenses, and you want both.

Application-level keepalive, because TCP will not tell you in a useful timeframe:

private startKeepalive(): void {
  this.stopKeepalive();
  this.keepaliveTimer = setInterval(() => {
    this.send({ type: 'KEEPALIVE', channel: 0 });
  }, 30_000);
}

Most protocols specify this and most people skip it because things work fine in testing. The server side is usually enforcing it too, so a client that stays quiet gets hung up on.

Then something that actually watches. The session promise polls rather than trusting events:

await new Promise<void>((resolve, reject) => {
  const check = setInterval(() => {
    if (this.stopRequested || !client.isOpen()) {
      clearInterval(check);
      this.stopRequested ? resolve() : reject(new Error('disconnected'));
    }
  }, 1000);
});

Polling readyState on a timer is unfashionable and it catches cases the event handlers miss. If you want to go further, track the timestamp of the last received frame and treat a long silence as a disconnect even when the socket claims otherwise, which is the only way to catch a truly wedged connection.

Resubscribing is your job

Nothing about your subscription state survives a reconnect. The server has no memory of you. Every symbol you asked for is gone the moment the socket drops, and if you do not replay them, the reconnect looks successful and delivers nothing.

That means holding the desired state separately from the connection:

subscribeEntries(entries: Array<{ type: string; symbol: string }>, batchSize = 100): void {
  this.wantedSubscriptions = entries;      // remember, so a reconnect can replay
  for (let i = 0; i < entries.length; i += batchSize) {
    this.send({
      type: 'FEED_SUBSCRIPTION',
      channel: this.channel,
      reset: i === 0,                      // only the first batch clears server state
      add: entries.slice(i, i + batchSize),
    });
  }
}

Two details in there earned their place. The batching exists because the server rejects oversized subscription frames, and a few hundred instruments across several event types adds up faster than you would think. And reset is true only on the first batch, because setting it on all of them means each batch wipes the previous one and you end up subscribed to the last hundred entries with no error to tell you.

That failure is silent, which is the theme of this entire post.

Disconnect on purpose

The counterintuitive one. Some problems are best solved by killing a healthy connection.

The credential authorizing my feed expires on a schedule. Rather than trying to refresh it in place on a live socket, a timer closes the connection deliberately, the reconnect loop notices, and runSession() fetches a fresh token on the way back up:

this.quoteTokenTimer = setInterval(() => {
  log.info('scheduled token refresh: closing socket to reacquire');
  this.client?.close();
}, this.config.quoteTokenRefreshHours * 3_600_000);

Reusing the recovery path you already have beats writing a second path that only runs every twenty hours and is therefore never tested. The reconnect logic runs constantly and you know it works.

This is what the closedIntentionally flag is for. Without it, a deliberate close looks identical to a failure, gets counted as one, and drags your backoff up for no reason:

close(): void {
  this.closedIntentionally = true;
  this.stopKeepalive();
  try { this.ws?.close(); } catch { /* already gone */ }
  this.ws = null;
}

Decoding against a schema you sent

This one is specific to compact binary-ish protocols, and it generalizes further than it looks.

To cut bandwidth, the feed sends events as positional arrays with no field names, batched by type:

["Quote", [ "Quote", ".SPY260918P490", 4.15, 4.25, 12, 30,
            "Quote", ".SPY260918P485", 3.40, 3.55, 8, 41 ]]

The only thing that turns that back into objects is the field map you declared during setup:

{
  type: 'FEED_SETUP',
  channel: 3,
  acceptDataFormat: 'COMPACT',
  acceptEventFields: {
    Quote: ['eventType', 'eventSymbol', 'bidPrice', 'askPrice', 'bidSize', 'askSize'],
  },
}

Decoding is then a stride over the array:

function expandCompactTypeBlock(
  type: string,
  values: unknown[],
  fieldMap: Record<string, string[]>,
) {
  const fields = fieldMap[type];
  if (!fields?.length) return [];
  const events = [];
  for (let i = 0; i + fields.length <= values.length; i += fields.length) {
    const record: Record<string, unknown> = {};
    fields.forEach((name, f) => { record[name] = values[i + f]; });
    if (record.eventSymbol) {
      events.push({ type, symbol: String(record.eventSymbol), fields: record });
    }
  }
  return events;
}

Let the declared map drift from what the parser expects and the decode shifts by one position, so every field quietly takes its neighbor's value. The output still looks entirely reasonable, which is the problem. I hit a version of this while building a screener on top of this feed.

The field list has to be a single constant used both to configure the connection and to decode it, because two copies will diverge and nothing will tell you when they do.

The through line

Every failure above is quiet. A socket that is open and dead, a reconnect that succeeds and subscribes to nothing, a batch that silently replaced the previous batch, a decoder reading the wrong columns. Not one of them throws.

That is what makes a long-running WebSocket client different from most code you write. The compiler cannot help, the tests pass, and the only defense is deciding in advance what "working" means and measuring it continuously. Mine emits a heartbeat every sixty seconds with the connection state, the number of instruments tracked, and how many of them have received an update recently. When that last number goes to zero while the socket claims to be open, I know within a minute.

Write that check before you need it. You will need it around 3am.