DocumentationarchitectureCircuit Breaker

Circuit Breaker Pattern

The circuit breaker prevents cascade failures when external services are unavailable.

Overview

         ┌─────────────────────────────────────┐
         │           Circuit Breaker           │
         │                                     │
Closed ──►  [Request] ──► [Success] ──► Closed │
   │       │     │                           │ │
   │       │     ▼ (Failures > Threshold)    │ │
   │       │   Open ──► Half-Open ──► Closed │ │
   │       │     │           │               │ │
   │       │     │   (Test)  │               │ │
   │       │     └───────────┘               │ │
   │       └─────────────────────────────────┘ │
   └─────────────────────────────────────────┘

States

StateBehavior
ClosedRequests pass through normally
OpenRequests fail immediately without calling service
Half-OpenLimited test requests to check if service recovered

Implementation

import { CircuitBreaker, withCircuitBreaker } from '@/utils/circuitBreaker'
 
// Create breaker with custom config
const breaker = new CircuitBreaker({
  failureThreshold: 5,    // Open after 5 failures
  resetTimeout: 30000,    // Try half-open after 30s
  halfOpenMaxCalls: 1     // Allow 1 test request
})
 
// Wrap external calls
const result = await breaker.execute(() => 
  fetchExternalAPI()
)

Pre-configured Breakers

FairScale Breaker

import { fairScaleBreaker } from '@/utils/circuitBreaker'
 
// Configured for FairScale API
// - 3 failure threshold
// - 15 second reset timeout
 
try {
  const score = await fairScaleBreaker.execute(() => 
    fairScaleClient.getScore(wallet)
  )
} catch (error) {
  if (error.message.includes('Circuit breaker is open')) {
    // Fallback to cached score or default
    return getCachedScore(wallet)
  }
}

RPC Breaker

import { rpcBreaker } from '@/utils/circuitBreaker'
 
// Configured for Solana RPC
// - 5 failure threshold
// - 30 second reset timeout
 
const account = await rpcBreaker.execute(() =>
  connection.getAccountInfo(publicKey)
)

API Breaker

import { apiBreaker } from '@/utils/circuitBreaker'
 
// Configured for internal API
// - 4 failure threshold
// - 20 second reset timeout
 
const proposals = await apiBreaker.execute(() =>
  fetch('/api/proposals').then(r => r.json())
)

Helper Function

import { withCircuitBreaker, apiBreaker } from '@/utils/circuitBreaker'
 
// Alternative syntax
const result = await withCircuitBreaker(apiBreaker, () => 
  someAsyncOperation()
)

Monitoring

Get Current State

const state = breaker.getState()
 
// Returns:
interface CircuitState {
  status: 'closed' | 'open' | 'half-open'
  failures: number
  lastFailure: number | null
  nextRetry: number | null
}

Reset Breaker

// Force reset (useful after fixing underlying issue)
breaker.reset()

Fallback Strategies

1. Cached Data

try {
  return await fairScaleBreaker.execute(() => 
    fairScaleClient.getScore(wallet)
  )
} catch (error) {
  if (error.message.includes('Circuit breaker')) {
    // Return cached value
    const cached = cache.get(`score:${wallet}`)
    if (cached) return cached
    
    // Or default
    return { score: 0, tier: 'unscored' }
  }
  throw error
}

2. Graceful Degradation

async function getVotingPower(wallet: string) {
  try {
    const score = await fairScaleBreaker.execute(() => 
      fairScaleClient.getScore(wallet)
    )
    return calculatePower(score)
  } catch {
    // Use base voting power without reputation bonus
    return calculateBasePower(wallet)
  }
}

3. Retry with Alternative

try {
  return await primaryBreaker.execute(() => primaryAPI.get())
} catch {
  return await fallbackBreaker.execute(() => fallbackAPI.get())
}

Best Practices

1. Right-size Thresholds

// For unreliable services
new CircuitBreaker({ failureThreshold: 3, resetTimeout: 15000 })
 
// For stable services
new CircuitBreaker({ failureThreshold: 10, resetTimeout: 60000 })

2. Don’t Retry on Open Circuit

// Bad - retries waste time
const result = await withRetry(() => breaker.execute(fn))
 
// Good - let circuit handle it
const result = await breaker.execute(fn)

3. Log State Changes

class LoggingCircuitBreaker extends CircuitBreaker {
  onFailure() {
    super.onFailure()
    const state = this.getState()
    if (state.status === 'open') {
      logger.warn('Circuit opened', { service: this.name })
    }
  }
}

4. Expose Health Checks

// In API health endpoint
app.get('/health', (req, res) => {
  res.json({
    status: 'ok',
    services: {
      fairscale: fairScaleBreaker.getState().status,
      rpc: rpcBreaker.getState().status,
      api: apiBreaker.getState().status
    }
  })
})

Integration Points

ServiceBreakerFallback
FairScale APIfairScaleBreakerCached score
Solana RPCrpcBreakerRetry with different endpoint
Internal APIapiBreakerLocal cache
IPFS/PinataCustomRetry without circuit