snapraid_smart_logging/.claude.md

# SnapRAID SMART Logger - Investigation Notes

## Project Goal

Create a Python utility that:
1. Fetches SMART data from Scrutiny's InfluxDB
2. Calculates failure probability **exactly like SnapRAID does** (using BackBlaze algorithm)
3. Logs enhanced data to PostgreSQL
4. Eventually add alerts and automation

## Current Status: BLOCKED - Algorithm Mismatch

We successfully built the infrastructure but **cannot match SnapRAID's failure probability calculations**.

---

## The Mystery: 81% vs 4.63%

### What SnapRAID Reports for /dev/sdc (ZLW0A3QJ)
```
   Temp  Power   Error   FP Size
      C OnDays   Count        TB  Serial        Device        Disk
 -----------------------------------------------------------------------
     24   1968       0  81% 12.0  ZLW0A3QJ      /dev/sdc      d1
```

**Failure Probability: 81%**

### What smartctl -A Shows
```
ID# ATTRIBUTE_NAME          VALUE WORST THRESH RAW_VALUE
  5 Reallocated_Sector_Ct   100   100   010    0
187 Reported_Uncorrect      100   100   000    0
188 Command_Timeout         100   100   000    0
197 Current_Pending_Sector  100   100   000    0
198 Offline_Uncorrectable   100   100   000    0
```

**All critical attributes are PERFECT** (VALUE=100, RAW_VALUE=0)

### What Our BackBlaze Calculation Gives
Using the exact same BackBlaze tables extracted from SnapRAID source:
- With RAW_VALUE = 0: **Probability = 4.63%**
- With VALUE = 100: **Probability = 100%** (saturated, wrong)

**Neither matches SnapRAID's 81%!**

---

## What We Know About SnapRAID's Algorithm

### Source Code Analysis

From `cmdline/device.c`:

```c
static double smart_afr(uint64_t* smart, const char* model)
{
    double afr = 0;
    uint64_t mask32 = 0xffffffffU;
    uint64_t mask16 = 0xffffU;

    // Attribute 5: Reallocated Sectors (32-bit mask)
    if (smart[5] != SMART_UNASSIGNED) {
        double r = smart_afr_value(SMART_5_R, SMART_5_STEP, smart[5] & mask32);
        if (afr < r) afr = r;
    }

    // Attribute 187: Reported Uncorrectable (16-bit mask)
    if (smart[187] != SMART_UNASSIGNED) {
        double r = smart_afr_value(SMART_187_R, SMART_187_STEP, smart[187] & mask16);
        if (afr < r) afr = r;
    }

    // Attribute 188: Command Timeout (16-bit, skipped for Seagate)
    if (strncmp(model, "ST", 2) != 0 && smart[188] != SMART_UNASSIGNED) {
        double r = smart_afr_value(SMART_188_R, SMART_188_STEP, smart[188] & mask16);
        if (afr < r) afr = r;
    }

    // Attribute 197: Current Pending Sectors (32-bit)
    if (smart[197] != SMART_UNASSIGNED) {
        double r = smart_afr_value(SMART_197_R, SMART_197_STEP, smart[197] & mask32);
        if (afr < r) afr = r;
    }

    // Attribute 198: Offline Uncorrectable (32-bit)
    if (smart[198] != SMART_UNASSIGNED) {
        double r = smart_afr_value(SMART_198_R, SMART_198_STEP, smart[198] & mask32);
        if (afr < r) afr = r;
    }

    return afr;
}
```

### Table Lookup Function

```c
static double smart_afr_value(double* tab, unsigned step, uint64_t value)
{
    value /= step;
    if (value >= SMART_MEASURES) value = SMART_MEASURES - 1;
    return 365.0 / 30.0 * tab[value];  // Annualize monthly rate
}
```

### Final Probability Calculation

```c
// Poisson distribution: P(at least 1 failure) = 1 - e^(-AFR)
poisson_prob_n_or_more_failures(afr, 1) * 100
```

**Formula:** `P = (1 - e^(-AFR)) × 100`

---

## Key Findings

### 1. BackBlaze Tables Are Correct
We extracted all 5 tables (256 values each) from SnapRAID source:
- `SMART_5_R` (Reallocated Sectors)
- `SMART_187_R` (Reported Uncorrectable)
- `SMART_188_R` (Command Timeout)
- `SMART_197_R` (Current Pending Sectors)
- `SMART_198_R` (Offline Uncorrectable)

Tables match exactly - verified byte-for-byte.

### 2. Algorithm Is Correct
Our implementation:
1. ✅ Uses correct bit masks (16-bit for 187/188, 32-bit for 5/197/198)
2. ✅ Applies correct step size (all = 1)
3. ✅ Takes maximum AFR across attributes
4. ✅ Scales monthly to annual (365/30)
5. ✅ Applies Poisson: `1 - e^(-AFR)`

### 3. The Math Works Backwards

**If SnapRAID reports 81%, then:**
```
0.81 = 1 - e^(-AFR)
e^(-AFR) = 0.19
AFR = -ln(0.19) = 1.66
Monthly rate = 1.66 × (30/365) = 0.136
```

**Looking at SMART_187_R table:**
- `table[1] = 0.1287` → AFR = 1.565 → **P = 79%** ✓
- `table[2] = 0.1579` → AFR = 1.920 → **P = 85%** ✓

**Conclusion: SnapRAID is using an attribute value of 1 or 2 for one of the attributes!**

### 4. But We Can't Find That Value!

From both smartctl AND InfluxDB:
- RAW_VALUE = 0 for all 5 attributes
- VALUE (normalized) = 100 for all 5 attributes
- WORST = 100 for all 5 attributes
- No errors, no degradation visible

---

## Current Hypothesis

SnapRAID must be:

1. **Extracting a specific byte** from RAW_VALUE that contains 1-2?
2. **Using a different field** we haven't checked (FLAGS? WHEN_FAILED?)?
3. **Calculating a derived value** from multiple fields?
4. **Using a different version** of the algorithm than GitHub master?

---

## What We Need

### Immediate Next Step
Check if RAW_VALUE has byte-level data we're missing:

```bash
smartctl -A /dev/sdc | grep -E "^(  5|187|188|197|198)"
```

Need to see if there are parenthetical byte breakdowns like:
```
194 Temperature_Celsius  24 (0 15 0 0 0)
                            ^^^^^^^^^^^^^ Individual bytes
```

### Alternative Approaches

1. **Run SnapRAID with debug/verbose mode** to see what values it reads
2. **Find the smartctl parsing code** in SnapRAID source
3. **Check SnapRAID version** - might be using different algorithm
4. **Contact SnapRAID author** if needed

---

## Project Files

### Core Implementation
- `smart_logger_v3.py` - Main logger (currently uses Scrutiny's failure rates)
- `backblaze_tables.py` - BackBlaze failure rate tables (256 values × 5 attributes)
- `schema_v2.sql` - PostgreSQL schema with devices + smart_metrics tables
- `populate_devices_final.sql` - Your 9 drives' metadata

### Debug Scripts
- `debug_smart_values.py` - Check InfluxDB raw SMART values
- `debug_scrutiny_failure_rate.py` - Check Scrutiny's calculated rates
- `debug_value_field.py` - Compare VALUE vs RAW_VALUE fields
- `test_backblaze_calc.py` - Test BackBlaze calculation with different inputs

### Setup
- `setup_database.sh` - PostgreSQL database creation
- `get_device_info.py` - Extract device WWNs from InfluxDB
- `requirements.txt` - Python dependencies

---

## Database Schema

### devices (manually populated)
```sql
device_wwn, device_path, serial_number, model, manufacturer,
capacity_bytes, size_tb, disk_role, notes
```

### smart_metrics (auto-populated by logger)
```sql
device_wwn, timestamp, temperature_celsius, power_on_days,
error_count, failure_probability_pct, smart_attributes (JSONB)
```

### Views
- `smart_latest` - Latest metrics per device with metadata
- `smart_high_risk` - Devices >50% failure probability
- `smart_summary` - SnapRAID-style summary table

---

## Database Connection Info

- **InfluxDB**: cyrion.c0smere.net:8086
  - Org: scrutiny
  - Bucket: metrics
  - Token: (in .env)

- **PostgreSQL**: phlegethon.c0smere.net:5432
  - Database: smart_monitoring
  - User: smart_logger
  - Password: saddog095

---

## Next Steps (After Solving Algorithm)

### Option 2 Features (After Option 1 Works)

1. **Alerts**
   - Email/webhook when failure probability crosses thresholds
   - Trend detection (rapidly increasing failure rate)
   - Integration with monitoring systems

2. **SnapRAID Integration**
   - Trigger scrub when high-risk drives detected
   - Pre-scrub notifications
   - Post-scrub verification

3. **Historical Analysis**
   - Failure probability trends over time
   - Predict when drives will cross risk thresholds
   - Identify patterns across drive models/manufacturers

4. **Custom Queries**
   - Drive health reports
   - Lifetime statistics
   - Comparison across drive families

---

## Comparison: SnapRAID vs Scrutiny vs Our Calculator

| Drive | SnapRAID | Scrutiny | Our Calc (raw=0) | Our Calc (val=100) |
|-------|----------|----------|------------------|---------------------|
| /dev/sdc (d1) | **81%** | 11% | 4.63% | 100% |
| /dev/sdb (d2) | 22% | 19% | 4.63% | 100% |
| /dev/sdd (d5) | 51% | 11% | 4.63% | 100% |
| /dev/sda (parity) | 49% | 16% | 4.63% | 100% |
| /dev/sdg (2-parity) | 30% | 12% | 4.63% | 100% |
| /dev/sdh (2-parity) | 4% | 5% | 4.63% | 100% |
| /dev/sde | 4% | 5% | 4.63% | 100% |
| /dev/sdf | 84% | 11% | 4.63% | 100% |
| /dev/sdi | 84% | 11% | 4.63% | 100% |

**Pattern:** SnapRAID shows high variance (4-84%), we show either 4.63% or 100%.

---

## Questions to Answer

1. **What exact value does SnapRAID read for attribute 5/187/188/197/198?**
2. **Is SnapRAID using RAW_VALUE, VALUE, WORST, or something else?**
3. **Could there be byte-level parsing we're missing?**
4. **What version of SnapRAID is running, and does it match GitHub master?**
5. **Is there a verbose mode to see what SnapRAID actually reads?**

---

## References

- [SnapRAID GitHub](https://github.com/amadvance/snapraid)
- [SnapRAID device.c](https://github.com/amadvance/snapraid/blob/master/cmdline/device.c)
- [Scrutiny GitHub](https://github.com/AnalogJ/scrutiny)
- [BackBlaze Hard Drive Stats](https://www.backblaze.com/cloud-storage/resources/hard-drive-test-data)

---

**Last Updated:** 2025-12-05
**Status:** 🔴 BLOCKED - Cannot match SnapRAID algorithm
**Blocker:** Unknown how SnapRAID calculates 81% with all attributes = 0/100