Circuit Breaker Pattern
The circuit breaker prevents cascade failures when external services are unavailable.
Overview
┌─────────────────────────────────────┐
│ Circuit Breaker │
│ │
Closed ──► [Request] ──► [Success] ──► Closed │
│ │ │ │ │
│ │ ▼ (Failures > Threshold) │ │
│ │ Open ──► Half-Open ──► Closed │ │
│ │ │ │ │ │
│ │ │ (Test) │ │ │
│ │ └───────────┘ │ │
│ └─────────────────────────────────┘ │
└─────────────────────────────────────────┘States
| State | Behavior |
|---|---|
| Closed | Requests pass through normally |
| Open | Requests fail immediately without calling service |
| Half-Open | Limited test requests to check if service recovered |
Implementation
import { CircuitBreaker, withCircuitBreaker } from '@/utils/circuitBreaker'
// Create breaker with custom config
const breaker = new CircuitBreaker({
failureThreshold: 5, // Open after 5 failures
resetTimeout: 30000, // Try half-open after 30s
halfOpenMaxCalls: 1 // Allow 1 test request
})
// Wrap external calls
const result = await breaker.execute(() =>
fetchExternalAPI()
)Pre-configured Breakers
FairScale Breaker
import { fairScaleBreaker } from '@/utils/circuitBreaker'
// Configured for FairScale API
// - 3 failure threshold
// - 15 second reset timeout
try {
const score = await fairScaleBreaker.execute(() =>
fairScaleClient.getScore(wallet)
)
} catch (error) {
if (error.message.includes('Circuit breaker is open')) {
// Fallback to cached score or default
return getCachedScore(wallet)
}
}RPC Breaker
import { rpcBreaker } from '@/utils/circuitBreaker'
// Configured for Solana RPC
// - 5 failure threshold
// - 30 second reset timeout
const account = await rpcBreaker.execute(() =>
connection.getAccountInfo(publicKey)
)API Breaker
import { apiBreaker } from '@/utils/circuitBreaker'
// Configured for internal API
// - 4 failure threshold
// - 20 second reset timeout
const proposals = await apiBreaker.execute(() =>
fetch('/api/proposals').then(r => r.json())
)Helper Function
import { withCircuitBreaker, apiBreaker } from '@/utils/circuitBreaker'
// Alternative syntax
const result = await withCircuitBreaker(apiBreaker, () =>
someAsyncOperation()
)Monitoring
Get Current State
const state = breaker.getState()
// Returns:
interface CircuitState {
status: 'closed' | 'open' | 'half-open'
failures: number
lastFailure: number | null
nextRetry: number | null
}Reset Breaker
// Force reset (useful after fixing underlying issue)
breaker.reset()Fallback Strategies
1. Cached Data
try {
return await fairScaleBreaker.execute(() =>
fairScaleClient.getScore(wallet)
)
} catch (error) {
if (error.message.includes('Circuit breaker')) {
// Return cached value
const cached = cache.get(`score:${wallet}`)
if (cached) return cached
// Or default
return { score: 0, tier: 'unscored' }
}
throw error
}2. Graceful Degradation
async function getVotingPower(wallet: string) {
try {
const score = await fairScaleBreaker.execute(() =>
fairScaleClient.getScore(wallet)
)
return calculatePower(score)
} catch {
// Use base voting power without reputation bonus
return calculateBasePower(wallet)
}
}3. Retry with Alternative
try {
return await primaryBreaker.execute(() => primaryAPI.get())
} catch {
return await fallbackBreaker.execute(() => fallbackAPI.get())
}Best Practices
1. Right-size Thresholds
// For unreliable services
new CircuitBreaker({ failureThreshold: 3, resetTimeout: 15000 })
// For stable services
new CircuitBreaker({ failureThreshold: 10, resetTimeout: 60000 })2. Don’t Retry on Open Circuit
// Bad - retries waste time
const result = await withRetry(() => breaker.execute(fn))
// Good - let circuit handle it
const result = await breaker.execute(fn)3. Log State Changes
class LoggingCircuitBreaker extends CircuitBreaker {
onFailure() {
super.onFailure()
const state = this.getState()
if (state.status === 'open') {
logger.warn('Circuit opened', { service: this.name })
}
}
}4. Expose Health Checks
// In API health endpoint
app.get('/health', (req, res) => {
res.json({
status: 'ok',
services: {
fairscale: fairScaleBreaker.getState().status,
rpc: rpcBreaker.getState().status,
api: apiBreaker.getState().status
}
})
})Integration Points
| Service | Breaker | Fallback |
|---|---|---|
| FairScale API | fairScaleBreaker | Cached score |
| Solana RPC | rpcBreaker | Retry with different endpoint |
| Internal API | apiBreaker | Local cache |
| IPFS/Pinata | Custom | Retry without circuit |