Complete rewrite: SnapRAID SMART logger with BackBlaze algorithm

Fetches SMART data from Scrutiny's InfluxDB and calculates failure probabilities using the exact BackBlaze tables from SnapRAID source. Key features: - Calculates both v12 (with attr 193) and v13 (without attr 193) algorithms - v12 matches SnapRAID pre-v13.0 (includes Load Cycle Count) - v13 matches modern SnapRAID v13.0+ (excludes Load Cycle Count) - Stores both values for trending and comparison - Correctly applies bit masks (16-bit for 187/188, 32-bit for others) - Annualizes monthly rates and applies Poisson distribution - Logs to PostgreSQL with device metadata Solved the mystery: SnapRAID v12 uses attribute 193 which was removed in v13.0. Load cycle count has massive impact on failure predictions for some drives (80% vs 4% difference). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
2025-12-07 06:55:39 -05:00
commit 27afedb32e
25 changed files with 3207 additions and 0 deletions
@@ -0,0 +1,321 @@
+# SnapRAID SMART Logger - Investigation Notes
+
+## Project Goal
+
+Create a Python utility that:
+1. Fetches SMART data from Scrutiny's InfluxDB
+2. Calculates failure probability **exactly like SnapRAID does** (using BackBlaze algorithm)
+3. Logs enhanced data to PostgreSQL
+4. Eventually add alerts and automation
+
+## Current Status: BLOCKED - Algorithm Mismatch
+
+We successfully built the infrastructure but **cannot match SnapRAID's failure probability calculations**.
+
+---
+
+## The Mystery: 81% vs 4.63%
+
+### What SnapRAID Reports for /dev/sdc (ZLW0A3QJ)
+```
+   Temp  Power   Error   FP Size
+      C OnDays   Count        TB  Serial        Device        Disk
+ -----------------------------------------------------------------------
+     24   1968       0  81% 12.0  ZLW0A3QJ      /dev/sdc      d1
+```
+
+**Failure Probability: 81%**
+
+### What smartctl -A Shows
+```
+ID# ATTRIBUTE_NAME          VALUE WORST THRESH RAW_VALUE
+  5 Reallocated_Sector_Ct   100   100   010    0
+187 Reported_Uncorrect      100   100   000    0
+188 Command_Timeout         100   100   000    0
+197 Current_Pending_Sector  100   100   000    0
+198 Offline_Uncorrectable   100   100   000    0
+```
+
+**All critical attributes are PERFECT** (VALUE=100, RAW_VALUE=0)
+
+### What Our BackBlaze Calculation Gives
+Using the exact same BackBlaze tables extracted from SnapRAID source:
+- With RAW_VALUE = 0: **Probability = 4.63%**
+- With VALUE = 100: **Probability = 100%** (saturated, wrong)
+
+**Neither matches SnapRAID's 81%!**
+
+---
+
+## What We Know About SnapRAID's Algorithm
+
+### Source Code Analysis
+
+From `cmdline/device.c`:
+
+```c
+static double smart_afr(uint64_t* smart, const char* model)
+{
+    double afr = 0;
+    uint64_t mask32 = 0xffffffffU;
+    uint64_t mask16 = 0xffffU;
+
+    // Attribute 5: Reallocated Sectors (32-bit mask)
+    if (smart[5] != SMART_UNASSIGNED) {
+        double r = smart_afr_value(SMART_5_R, SMART_5_STEP, smart[5] & mask32);
+        if (afr < r) afr = r;
+    }
+
+    // Attribute 187: Reported Uncorrectable (16-bit mask)
+    if (smart[187] != SMART_UNASSIGNED) {
+        double r = smart_afr_value(SMART_187_R, SMART_187_STEP, smart[187] & mask16);
+        if (afr < r) afr = r;
+    }
+
+    // Attribute 188: Command Timeout (16-bit, skipped for Seagate)
+    if (strncmp(model, "ST", 2) != 0 && smart[188] != SMART_UNASSIGNED) {
+        double r = smart_afr_value(SMART_188_R, SMART_188_STEP, smart[188] & mask16);
+        if (afr < r) afr = r;
+    }
+
+    // Attribute 197: Current Pending Sectors (32-bit)
+    if (smart[197] != SMART_UNASSIGNED) {
+        double r = smart_afr_value(SMART_197_R, SMART_197_STEP, smart[197] & mask32);
+        if (afr < r) afr = r;
+    }
+
+    // Attribute 198: Offline Uncorrectable (32-bit)
+    if (smart[198] != SMART_UNASSIGNED) {
+        double r = smart_afr_value(SMART_198_R, SMART_198_STEP, smart[198] & mask32);
+        if (afr < r) afr = r;
+    }
+
+    return afr;
+}
+```
+
+### Table Lookup Function
+
+```c
+static double smart_afr_value(double* tab, unsigned step, uint64_t value)
+{
+    value /= step;
+    if (value >= SMART_MEASURES) value = SMART_MEASURES - 1;
+    return 365.0 / 30.0 * tab[value];  // Annualize monthly rate
+}
+```
+
+### Final Probability Calculation
+
+```c
+// Poisson distribution: P(at least 1 failure) = 1 - e^(-AFR)
+poisson_prob_n_or_more_failures(afr, 1) * 100
+```
+
+**Formula:** `P = (1 - e^(-AFR)) × 100`
+
+---
+
+## Key Findings
+
+### 1. BackBlaze Tables Are Correct
+We extracted all 5 tables (256 values each) from SnapRAID source:
+- `SMART_5_R` (Reallocated Sectors)
+- `SMART_187_R` (Reported Uncorrectable)
+- `SMART_188_R` (Command Timeout)
+- `SMART_197_R` (Current Pending Sectors)
+- `SMART_198_R` (Offline Uncorrectable)
+
+Tables match exactly - verified byte-for-byte.
+
+### 2. Algorithm Is Correct
+Our implementation:
+1. ✅ Uses correct bit masks (16-bit for 187/188, 32-bit for 5/197/198)
+2. ✅ Applies correct step size (all = 1)
+3. ✅ Takes maximum AFR across attributes
+4. ✅ Scales monthly to annual (365/30)
+5. ✅ Applies Poisson: `1 - e^(-AFR)`
+
+### 3. The Math Works Backwards
+
+**If SnapRAID reports 81%, then:**
+```
+0.81 = 1 - e^(-AFR)
+e^(-AFR) = 0.19
+AFR = -ln(0.19) = 1.66
+Monthly rate = 1.66 × (30/365) = 0.136
+```
+
+**Looking at SMART_187_R table:**
+- `table[1] = 0.1287` → AFR = 1.565 → **P = 79%** ✓
+- `table[2] = 0.1579` → AFR = 1.920 → **P = 85%** ✓
+
+**Conclusion: SnapRAID is using an attribute value of 1 or 2 for one of the attributes!**
+
+### 4. But We Can't Find That Value!
+
+From both smartctl AND InfluxDB:
+- RAW_VALUE = 0 for all 5 attributes
+- VALUE (normalized) = 100 for all 5 attributes
+- WORST = 100 for all 5 attributes
+- No errors, no degradation visible
+
+---
+
+## Current Hypothesis
+
+SnapRAID must be:
+
+1. **Extracting a specific byte** from RAW_VALUE that contains 1-2?
+2. **Using a different field** we haven't checked (FLAGS? WHEN_FAILED?)?
+3. **Calculating a derived value** from multiple fields?
+4. **Using a different version** of the algorithm than GitHub master?
+
+---
+
+## What We Need
+
+### Immediate Next Step
+Check if RAW_VALUE has byte-level data we're missing:
+
+```bash
+smartctl -A /dev/sdc | grep -E "^(  5|187|188|197|198)"
+```
+
+Need to see if there are parenthetical byte breakdowns like:
+```
+194 Temperature_Celsius  24 (0 15 0 0 0)
+                            ^^^^^^^^^^^^^ Individual bytes
+```
+
+### Alternative Approaches
+
+1. **Run SnapRAID with debug/verbose mode** to see what values it reads
+2. **Find the smartctl parsing code** in SnapRAID source
+3. **Check SnapRAID version** - might be using different algorithm
+4. **Contact SnapRAID author** if needed
+
+---
+
+## Project Files
+
+### Core Implementation
+- `smart_logger_v3.py` - Main logger (currently uses Scrutiny's failure rates)
+- `backblaze_tables.py` - BackBlaze failure rate tables (256 values × 5 attributes)
+- `schema_v2.sql` - PostgreSQL schema with devices + smart_metrics tables
+- `populate_devices_final.sql` - Your 9 drives' metadata
+
+### Debug Scripts
+- `debug_smart_values.py` - Check InfluxDB raw SMART values
+- `debug_scrutiny_failure_rate.py` - Check Scrutiny's calculated rates
+- `debug_value_field.py` - Compare VALUE vs RAW_VALUE fields
+- `test_backblaze_calc.py` - Test BackBlaze calculation with different inputs
+
+### Setup
+- `setup_database.sh` - PostgreSQL database creation
+- `get_device_info.py` - Extract device WWNs from InfluxDB
+- `requirements.txt` - Python dependencies
+
+---
+
+## Database Schema
+
+### devices (manually populated)
+```sql
+device_wwn, device_path, serial_number, model, manufacturer,
+capacity_bytes, size_tb, disk_role, notes
+```
+
+### smart_metrics (auto-populated by logger)
+```sql
+device_wwn, timestamp, temperature_celsius, power_on_days,
+error_count, failure_probability_pct, smart_attributes (JSONB)
+```
+
+### Views
+- `smart_latest` - Latest metrics per device with metadata
+- `smart_high_risk` - Devices >50% failure probability
+- `smart_summary` - SnapRAID-style summary table
+
+---
+
+## Database Connection Info
+
+- **InfluxDB**: cyrion.c0smere.net:8086
+  - Org: scrutiny
+  - Bucket: metrics
+  - Token: (in .env)
+
+- **PostgreSQL**: phlegethon.c0smere.net:5432
+  - Database: smart_monitoring
+  - User: smart_logger
+  - Password: saddog095
+
+---
+
+## Next Steps (After Solving Algorithm)
+
+### Option 2 Features (After Option 1 Works)
+
+1. **Alerts**
+   - Email/webhook when failure probability crosses thresholds
+   - Trend detection (rapidly increasing failure rate)
+   - Integration with monitoring systems
+
+2. **SnapRAID Integration**
+   - Trigger scrub when high-risk drives detected
+   - Pre-scrub notifications
+   - Post-scrub verification
+
+3. **Historical Analysis**
+   - Failure probability trends over time
+   - Predict when drives will cross risk thresholds
+   - Identify patterns across drive models/manufacturers
+
+4. **Custom Queries**
+   - Drive health reports
+   - Lifetime statistics
+   - Comparison across drive families
+
+---
+
+## Comparison: SnapRAID vs Scrutiny vs Our Calculator
+
+| Drive | SnapRAID | Scrutiny | Our Calc (raw=0) | Our Calc (val=100) |
+|-------|----------|----------|------------------|---------------------|
+| /dev/sdc (d1) | **81%** | 11% | 4.63% | 100% |
+| /dev/sdb (d2) | 22% | 19% | 4.63% | 100% |
+| /dev/sdd (d5) | 51% | 11% | 4.63% | 100% |
+| /dev/sda (parity) | 49% | 16% | 4.63% | 100% |
+| /dev/sdg (2-parity) | 30% | 12% | 4.63% | 100% |
+| /dev/sdh (2-parity) | 4% | 5% | 4.63% | 100% |
+| /dev/sde | 4% | 5% | 4.63% | 100% |
+| /dev/sdf | 84% | 11% | 4.63% | 100% |
+| /dev/sdi | 84% | 11% | 4.63% | 100% |
+
+**Pattern:** SnapRAID shows high variance (4-84%), we show either 4.63% or 100%.
+
+---
+
+## Questions to Answer
+
+1. **What exact value does SnapRAID read for attribute 5/187/188/197/198?**
+2. **Is SnapRAID using RAW_VALUE, VALUE, WORST, or something else?**
+3. **Could there be byte-level parsing we're missing?**
+4. **What version of SnapRAID is running, and does it match GitHub master?**
+5. **Is there a verbose mode to see what SnapRAID actually reads?**
+
+---
+
+## References
+
+- [SnapRAID GitHub](https://github.com/amadvance/snapraid)
+- [SnapRAID device.c](https://github.com/amadvance/snapraid/blob/master/cmdline/device.c)
+- [Scrutiny GitHub](https://github.com/AnalogJ/scrutiny)
+- [BackBlaze Hard Drive Stats](https://www.backblaze.com/cloud-storage/resources/hard-drive-test-data)
+
+---
+
+**Last Updated:** 2025-12-05
+**Status:** 🔴 BLOCKED - Cannot match SnapRAID algorithm
+**Blocker:** Unknown how SnapRAID calculates 81% with all attributes = 0/100