snapraid_smart_logging/.claude.md

SnapRAID SMART Logger - Investigation Notes

Project Goal

Create a Python utility that:

1. Fetches SMART data from Scrutiny's InfluxDB
2. Calculates failure probability exactly like SnapRAID does (using BackBlaze algorithm)
3. Logs enhanced data to PostgreSQL
4. Eventually add alerts and automation

Current Status: BLOCKED - Algorithm Mismatch

We successfully built the infrastructure but cannot match SnapRAID's failure probability calculations.

---

The Mystery: 81% vs 4.63%

What SnapRAID Reports for /dev/sdc (ZLW0A3QJ)

   Temp  Power   Error   FP Size
      C OnDays   Count        TB  Serial        Device        Disk
 -----------------------------------------------------------------------
     24   1968       0  81% 12.0  ZLW0A3QJ      /dev/sdc      d1

Failure Probability: 81%

What smartctl -A Shows

ID# ATTRIBUTE_NAME          VALUE WORST THRESH RAW_VALUE
  5 Reallocated_Sector_Ct   100   100   010    0
187 Reported_Uncorrect      100   100   000    0
188 Command_Timeout         100   100   000    0
197 Current_Pending_Sector  100   100   000    0
198 Offline_Uncorrectable   100   100   000    0

All critical attributes are PERFECT (VALUE=100, RAW_VALUE=0)

What Our BackBlaze Calculation Gives

Using the exact same BackBlaze tables extracted from SnapRAID source:

• With RAW_VALUE = 0: Probability = 4.63%
• With VALUE = 100: Probability = 100% (saturated, wrong)

Neither matches SnapRAID's 81%!

---

What We Know About SnapRAID's Algorithm

Source Code Analysis

From cmdline/device.c:

static double smart_afr(uint64_t* smart, const char* model)
{
    double afr = 0;
    uint64_t mask32 = 0xffffffffU;
    uint64_t mask16 = 0xffffU;

    // Attribute 5: Reallocated Sectors (32-bit mask)
    if (smart[5] != SMART_UNASSIGNED) {
        double r = smart_afr_value(SMART_5_R, SMART_5_STEP, smart[5] & mask32);
        if (afr < r) afr = r;
    }

    // Attribute 187: Reported Uncorrectable (16-bit mask)
    if (smart[187] != SMART_UNASSIGNED) {
        double r = smart_afr_value(SMART_187_R, SMART_187_STEP, smart[187] & mask16);
        if (afr < r) afr = r;
    }

    // Attribute 188: Command Timeout (16-bit, skipped for Seagate)
    if (strncmp(model, "ST", 2) != 0 && smart[188] != SMART_UNASSIGNED) {
        double r = smart_afr_value(SMART_188_R, SMART_188_STEP, smart[188] & mask16);
        if (afr < r) afr = r;
    }

    // Attribute 197: Current Pending Sectors (32-bit)
    if (smart[197] != SMART_UNASSIGNED) {
        double r = smart_afr_value(SMART_197_R, SMART_197_STEP, smart[197] & mask32);
        if (afr < r) afr = r;
    }

    // Attribute 198: Offline Uncorrectable (32-bit)
    if (smart[198] != SMART_UNASSIGNED) {
        double r = smart_afr_value(SMART_198_R, SMART_198_STEP, smart[198] & mask32);
        if (afr < r) afr = r;
    }

    return afr;
}

Table Lookup Function

static double smart_afr_value(double* tab, unsigned step, uint64_t value)
{
    value /= step;
    if (value >= SMART_MEASURES) value = SMART_MEASURES - 1;
    return 365.0 / 30.0 * tab[value];  // Annualize monthly rate
}

Final Probability Calculation

// Poisson distribution: P(at least 1 failure) = 1 - e^(-AFR)
poisson_prob_n_or_more_failures(afr, 1) * 100

Formula: P = (1 - e^(-AFR)) × 100

---

Key Findings

1. BackBlaze Tables Are Correct

We extracted all 5 tables (256 values each) from SnapRAID source:

• SMART_5_R (Reallocated Sectors)
• SMART_187_R (Reported Uncorrectable)
• SMART_188_R (Command Timeout)
• SMART_197_R (Current Pending Sectors)
• SMART_198_R (Offline Uncorrectable)

Tables match exactly - verified byte-for-byte.

2. Algorithm Is Correct

Our implementation:

1. ✅ Uses correct bit masks (16-bit for 187/188, 32-bit for 5/197/198)
2. ✅ Applies correct step size (all = 1)
3. ✅ Takes maximum AFR across attributes
4. ✅ Scales monthly to annual (365/30)
5. ✅ Applies Poisson: 1 - e^(-AFR)

3. The Math Works Backwards

If SnapRAID reports 81%, then:

0.81 = 1 - e^(-AFR)
e^(-AFR) = 0.19
AFR = -ln(0.19) = 1.66
Monthly rate = 1.66 × (30/365) = 0.136

Looking at SMART_187_R table:

• table[1] = 0.1287 → AFR = 1.565 → P = 79% ✓
• table[2] = 0.1579 → AFR = 1.920 → P = 85% ✓

Conclusion: SnapRAID is using an attribute value of 1 or 2 for one of the attributes!

4. But We Can't Find That Value!

From both smartctl AND InfluxDB:

• RAW_VALUE = 0 for all 5 attributes
• VALUE (normalized) = 100 for all 5 attributes
• WORST = 100 for all 5 attributes
• No errors, no degradation visible

---

Current Hypothesis

SnapRAID must be:

1. Extracting a specific byte from RAW_VALUE that contains 1-2?
2. Using a different field we haven't checked (FLAGS? WHEN_FAILED?)?
3. Calculating a derived value from multiple fields?
4. Using a different version of the algorithm than GitHub master?

---

What We Need

Immediate Next Step

Check if RAW_VALUE has byte-level data we're missing:

smartctl -A /dev/sdc | grep -E "^(  5|187|188|197|198)"

Need to see if there are parenthetical byte breakdowns like:

194 Temperature_Celsius  24 (0 15 0 0 0)
                            ^^^^^^^^^^^^^ Individual bytes

Alternative Approaches

1. Run SnapRAID with debug/verbose mode to see what values it reads
2. Find the smartctl parsing code in SnapRAID source
3. Check SnapRAID version - might be using different algorithm
4. Contact SnapRAID author if needed

---

Project Files

Core Implementation

• smart_logger_v3.py - Main logger (currently uses Scrutiny's failure rates)
• backblaze_tables.py - BackBlaze failure rate tables (256 values × 5 attributes)
• schema_v2.sql - PostgreSQL schema with devices + smart_metrics tables
• populate_devices_final.sql - Your 9 drives' metadata

Debug Scripts

• debug_smart_values.py - Check InfluxDB raw SMART values
• debug_scrutiny_failure_rate.py - Check Scrutiny's calculated rates
• debug_value_field.py - Compare VALUE vs RAW_VALUE fields
• test_backblaze_calc.py - Test BackBlaze calculation with different inputs

Setup

• setup_database.sh - PostgreSQL database creation
• get_device_info.py - Extract device WWNs from InfluxDB
• requirements.txt - Python dependencies

---

Database Schema

devices (manually populated)

device_wwn, device_path, serial_number, model, manufacturer,
capacity_bytes, size_tb, disk_role, notes

smart_metrics (auto-populated by logger)

device_wwn, timestamp, temperature_celsius, power_on_days,
error_count, failure_probability_pct, smart_attributes (JSONB)

Views

• smart_latest - Latest metrics per device with metadata
• smart_high_risk - Devices >50% failure probability
• smart_summary - SnapRAID-style summary table

---

Database Connection Info

• InfluxDB: cyrion.c0smere.net:8086

  • Org: scrutiny
  • Bucket: metrics
  • Token: (in .env)

• PostgreSQL: phlegethon.c0smere.net:5432

  • Database: smart_monitoring
  • User: smart_logger
  • Password: saddog095

---

Next Steps (After Solving Algorithm)

Option 2 Features (After Option 1 Works)

1. Alerts

   • Email/webhook when failure probability crosses thresholds
   • Trend detection (rapidly increasing failure rate)
   • Integration with monitoring systems

2. SnapRAID Integration

   • Trigger scrub when high-risk drives detected
   • Pre-scrub notifications
   • Post-scrub verification

3. Historical Analysis

   • Failure probability trends over time
   • Predict when drives will cross risk thresholds
   • Identify patterns across drive models/manufacturers

4. Custom Queries

   • Drive health reports
   • Lifetime statistics
   • Comparison across drive families

---

Comparison: SnapRAID vs Scrutiny vs Our Calculator

Drive	SnapRAID	Scrutiny	Our Calc (raw=0)	Our Calc (val=100)
/dev/sdc (d1)	81%	11%	4.63%	100%
/dev/sdb (d2)	22%	19%	4.63%	100%
/dev/sdd (d5)	51%	11%	4.63%	100%
/dev/sda (parity)	49%	16%	4.63%	100%
/dev/sdg (2-parity)	30%	12%	4.63%	100%
/dev/sdh (2-parity)	4%	5%	4.63%	100%
/dev/sde	4%	5%	4.63%	100%
/dev/sdf	84%	11%	4.63%	100%
/dev/sdi	84%	11%	4.63%	100%

Pattern: SnapRAID shows high variance (4-84%), we show either 4.63% or 100%.

---

Questions to Answer

1. What exact value does SnapRAID read for attribute 5/187/188/197/198?
2. Is SnapRAID using RAW_VALUE, VALUE, WORST, or something else?
3. Could there be byte-level parsing we're missing?
4. What version of SnapRAID is running, and does it match GitHub master?
5. Is there a verbose mode to see what SnapRAID actually reads?

---

References

• SnapRAID GitHub
• SnapRAID device.c
• Scrutiny GitHub
• BackBlaze Hard Drive Stats

---

Last Updated: 2025-12-05 Status: 🔴 BLOCKED - Cannot match SnapRAID algorithm Blocker: Unknown how SnapRAID calculates 81% with all attributes = 0/100