⚠️ Production Incident Report
Date: [YOUR DATE PLACEHOLDER]
Impact: Production line stopped for [TIME]
Root Cause: Infinite loop in temperature monitoring system
Cost: [YOUR COST PLACEHOLDER]
Date: [YOUR DATE PLACEHOLDER]
Impact: Production line stopped for [TIME]
Root Cause: Infinite loop in temperature monitoring system
Cost: [YOUR COST PLACEHOLDER]
[YOUR INFINITE LOOP STORY PLACEHOLDER]
Describe the actual incident. What system were you working on? What was the loop supposed to do? How did it bring down production? What was the conversation with your manager like?
I thought I understood loops. I'd written hundreds of them:
for i in range(10):
print(i)
# I could write this in my sleep
# But I had no idea what was actually happening
Then I deployed this to production:
# The killer code (simplified)
def monitor_furnace(sensor):
temperatures = []
while sensor.is_active: # I didn't realize this never became False
temp = sensor.read_temperature()
temperatures.append(temp) # Memory leak!
if temp > 1600:
alert_operator()
return temperatures
# This ran forever, consuming all memory, crashing the monitoring system
What I Didn't Understand About Loops
1. The While Loop Trap
I thought while loops were just "keep doing until false". I didn't understand:
# What I wrote (WRONG):
i = 0
while i < 10:
process_item(i)
# Forgot to increment i - infinite loop!
# The sensor nightmare (WRONG):
while sensor.temperature < 1500:
wait() # What if sensor breaks and always reads 0?
# What I should have written (RIGHT):
max_iterations = 1000
counter = 0
while sensor.temperature < 1500 and counter < max_iterations:
wait()
counter += 1
if counter >= max_iterations:
raise TimeoutError("Sensor may be malfunctioning")
2. The For Loop Misconception
I thought for loops were safe. They're not:
# The memory disaster:
readings = [1500, 1502, 1498]
for temp in readings:
if temp < 1500:
readings.append(temp + 10) # Modifying list while iterating!
# This can cause infinite loops or skip elements
# The index confusion:
for i in range(len(readings)):
if readings[i] > 1500:
del readings[i] # Index shifts, skips next element!
3. The Range() Mystery
# What I thought range() was:
range(5) # I thought: [0, 1, 2, 3, 4]
# What it actually is:
range(5) # A range object, not a list! Generates values on demand
# Why this matters in industrial systems:
# WRONG - Creates entire list in memory:
for i in list(range(100000000)): # Boom! Memory explosion
if i > 10:
break
# RIGHT - Generates values as needed:
for i in range(100000000): # Memory efficient
if i > 10:
break
Loop Patterns That Actually Work
Pattern 1: The Timeout Guardian
import time
def safe_sensor_read(sensor, timeout=60):
"""Read sensor with timeout protection."""
start_time = time.time()
readings = []
while True:
# Check timeout FIRST
if time.time() - start_time > timeout:
print(f"Timeout after {timeout} seconds")
break
# Check sensor status
if not sensor.is_active:
print("Sensor deactivated")
break
# Read and process
reading = sensor.read()
readings.append(reading)
# Check completion condition
if len(readings) >= 100:
print("Collected enough readings")
break
time.sleep(0.1) # Prevent CPU spinning
return readings
Pattern 2: The Iterator Protocol
# Understanding what for loops actually do:
readings = [1500, 1502, 1505]
# For loop is syntactic sugar for:
iterator = iter(readings)
while True:
try:
value = next(iterator)
print(value)
except StopIteration:
break
# This is why modifying during iteration is dangerous!
Pattern 3: The Enumerate Enlightenment
# What I used to write:
temperatures = [1500, 1502, 1498, 1505]
for i in range(len(temperatures)):
print(f"Reading {i}: {temperatures[i]}°C")
# What I should write:
for i, temp in enumerate(temperatures):
print(f"Reading {i}: {temp}°C")
# Even better for industrial systems:
for i, temp in enumerate(temperatures, start=1):
if temp > 1550:
print(f"Alert: Reading #{i} is {temp}°C")
The Real Cost of Bad Loops
Industrial Reality Check:
• An infinite loop in a control system can damage equipment
• A memory leak from unbounded collection can crash safety systems
• An off-by-one error can skip critical sensor readings
• A CPU-spinning loop can make real-time systems miss deadlines
• An infinite loop in a control system can damage equipment
• A memory leak from unbounded collection can crash safety systems
• An off-by-one error can skip critical sensor readings
• A CPU-spinning loop can make real-time systems miss deadlines
Exercise: Loop Mastery
Exercise 1: Fix the Dangerous Loops
Each of these loops has a serious bug. Find and fix them:
# Bug 1: The Infinite Collector
def collect_stable_readings(sensor):
"""Collect readings until temperature stabilizes."""
readings = []
previous = sensor.read()
while True:
current = sensor.read()
readings.append(current)
if abs(current - previous) < 0.5: # Stable?
break
# BUG: What's missing here?
return readings
# Bug 2: The List Modifier
def remove_outliers(temperatures):
"""Remove temperatures outside normal range."""
for i in range(len(temperatures)):
if temperatures[i] < 1000 or temperatures[i] > 2000:
del temperatures[i] # BUG: This will crash or skip!
return temperatures
# Bug 3: The Nested Nightmare
def find_correlated_sensors(all_readings):
"""Find sensors with similar patterns."""
correlated = []
for i in range(len(all_readings)):
for j in range(len(all_readings)):
if i != j: # Don't compare with self
if correlate(all_readings[i], all_readings[j]) > 0.9:
correlated.append((i, j))
# BUG: What's wrong with this approach?
return correlated
Industrial Loop Patterns
The Sliding Window
def rolling_average(readings, window_size=10):
"""Calculate rolling average for smoothing sensor data."""
averages = []
for i in range(len(readings) - window_size + 1):
window = readings[i:i + window_size]
avg = sum(window) / window_size
averages.append(avg)
return averages
# More efficient version using deque:
from collections import deque
def rolling_average_efficient(readings, window_size=10):
"""Memory-efficient rolling average."""
window = deque(maxlen=window_size)
averages = []
for reading in readings:
window.append(reading)
if len(window) == window_size:
averages.append(sum(window) / window_size)
return averages
The State Machine
def monitor_furnace_states(sensor):
"""Monitor furnace through different states."""
states = ['HEATING', 'STABLE', 'COOLING', 'IDLE']
current_state = 'IDLE'
for reading in sensor.get_readings():
previous_state = current_state
# State transitions
if current_state == 'IDLE' and reading > 500:
current_state = 'HEATING'
elif current_state == 'HEATING' and reading > 1500:
current_state = 'STABLE'
elif current_state == 'STABLE' and reading < 1450:
current_state = 'COOLING'
elif current_state == 'COOLING' and reading < 500:
current_state = 'IDLE'
if current_state != previous_state:
log_state_change(previous_state, current_state, reading)
The Challenge: Production Line Monitor
Challenge: Build a Safe Production Monitor
Create a monitoring system that processes sensor data safely:
class ProductionMonitor:
def __init__(self, max_readings=10000):
self.max_readings = max_readings
self.readings = []
self.alerts = []
def monitor_production(self, sensor, duration=3600):
"""
Monitor production for specified duration (seconds).
Requirements:
1. Never exceed max_readings in memory
2. Include timeout protection
3. Handle sensor failures gracefully
4. Calculate running statistics without storing all data
5. Detect anomalies in real-time
"""
# YOUR CODE HERE
pass
def detect_anomaly(self, current_reading, recent_readings):
"""
Detect if current reading is anomalous.
Anomaly = more than 3 standard deviations from recent mean
"""
# YOUR CODE HERE
pass
def safe_cleanup(self):
"""
Safely clean up old readings to prevent memory issues.
Keep only last 1000 readings.
"""
# YOUR CODE HERE
pass
# Your monitor should handle:
# - Sensor that never stops sending data
# - Sensor that fails mid-operation
# - Memory constraints
# - Real-time processing requirements
Lessons Learned the Hard Way
- Always have an exit strategy - Every loop needs a guaranteed way out
- Never trust external conditions - Sensors fail, networks drop, files corrupt
- Bound your collections - Unbounded lists are memory leaks waiting to happen
- Don't modify while iterating - Create new lists or iterate over copies
- Understand your iterator - Know what you're actually looping over
- Test edge cases - Empty lists, single elements, maximum sizes
[YOUR RECOVERY PLACEHOLDER]
How did you fix the production issue? What did you learn? What do you do differently now?
The Truth: Most production failures aren't from complex algorithms – they're from simple loops that weren't thought through. A junior dev who writes safe loops is more valuable than a senior who writes clever but dangerous ones.