snapraid_smart_logging/README.md

# SnapRAID SMART Logger

A Python utility that collects SMART data from Scrutiny's InfluxDB, calculates BackBlaze-based failure probabilities (like SnapRAID does), and logs the enhanced data to PostgreSQL.

## Features

- ✅ Connects to Scrutiny's InfluxDB to fetch SMART attributes
- ✅ Implements SnapRAID's BackBlaze failure probability algorithm
- ✅ Calculates annual failure rate (AFR) for each disk
- ✅ Stores historical data in PostgreSQL with time-series support
- ✅ Provides views for latest metrics and high-risk devices

## Quick Start

### 1. Install Dependencies

```bash
python3 -m venv venv
source venv/bin/activate
pip install -r requirements.txt
```

### 2. Configure Environment

```bash
cp .env.example .env
# Edit .env with your actual values
```

Required configuration:
- `INFLUXDB_URL`: Your Scrutiny InfluxDB URL (e.g., `http://192.168.1.100:8086`)
- `INFLUXDB_TOKEN`: Your Scrutiny InfluxDB token
- `POSTGRES_*`: Your PostgreSQL connection details

### 3. Set Up PostgreSQL Schema

```bash
psql -h <host> -U <user> -d <database> -f schema.sql
```

### 4. Explore Scrutiny's InfluxDB Schema

**IMPORTANT**: Run this first to understand how Scrutiny stores SMART data:

```bash
python explore_influx.py
```

This will show you:
- Available measurements in the `metrics` bucket
- Field names for each measurement
- Sample data structure
- Tags used for device identification

### 5. Customize the Logger

Based on the output from `explore_influx.py`, update `smart_logger.py`:

1. **Update the InfluxDB query** (`fetch_smart_data` method):
   - Set correct `_measurement` name
   - Map field names to device properties (temp, serial, model, etc.)
   - Map SMART attribute fields to IDs (5, 187, 188, 197, 198)

2. **Adjust device detection**:
   - Update how device path, serial, and model are extracted
   - Add disk role mapping if you want to match SnapRAID roles

### 6. Run the Logger

```bash
python smart_logger.py
```

## Output Format

The logger will display a summary like SnapRAID:

```
Device: /dev/sda
  Serial: ZX20AAMV
  Model: WDC WD200EFAX-68FB5N0
  Temp: 25°C
  Power-On: 636 days
  Failure Probability: 49.00%

================================================================================
Summary:
================================================================================
🔴 HIGH RISK /dev/sdc: 81.00% failure probability
🟢 OK /dev/sdb: 22.00% failure probability
```

## PostgreSQL Schema

### Tables

- **`smart_metrics`**: Main table storing all SMART data snapshots
  - Includes temperature, power-on time, error counts
  - Stores calculated failure probability
  - Raw SMART attributes as JSONB for flexibility

### Views

- **`smart_latest`**: Latest metrics for each device
- **`smart_high_risk`**: Devices with >50% annual failure probability

### Example Queries

```sql
-- Get latest status for all devices
SELECT device_path, serial_number, temperature_celsius,
       failure_probability_pct, timestamp
FROM smart_latest
ORDER BY failure_probability_pct DESC;

-- Historical trend for a specific disk
SELECT timestamp, temperature_celsius, failure_probability_pct
FROM smart_metrics
WHERE serial_number = 'ZX20AAMV'
ORDER BY timestamp DESC
LIMIT 100;

-- Check specific SMART attribute over time
SELECT timestamp,
       (smart_attributes->>'5')::int as reallocated_sectors,
       (smart_attributes->>'197')::int as pending_sectors
FROM smart_metrics
WHERE device_path = '/dev/sda'
ORDER BY timestamp DESC;
```

## BackBlaze Algorithm

The failure probability calculation is based on SnapRAID's implementation, which uses BackBlaze's 2014 dataset (47,322 disk observations) to create lookup tables for these SMART attributes:

- **Attribute 5**: Reallocated Sectors Count
- **Attribute 187**: Reported Uncorrectable Errors
- **Attribute 188**: Command Timeout (excluded for Seagate disks)
- **Attribute 197**: Current Pending Sector Count
- **Attribute 198**: Offline Uncorrectable Sector Count

The algorithm returns the **maximum** AFR across all attributes (since they're correlated, not independent).

## Automation

To run this periodically, add a cron job:

```bash
# Run every hour
0 * * * * /path/to/venv/bin/python /path/to/smart_logger.py >> /var/log/smart_logger.log 2>&1
```

Or use systemd timer, or your preferred scheduler.

## Troubleshooting

### InfluxDB Connection Issues

- Verify the URL and port are correct
- Check that the token has read permissions for the `metrics` bucket
- Ensure InfluxDB is accessible from the machine running this script

### No Data Returned

- Run `explore_influx.py` to check the data structure
- Verify Scrutiny is actively collecting data
- Check the time range in the query (`-5m` might be too short)

### PostgreSQL Insert Failures

- Ensure the schema was created (`schema.sql`)
- Check that the database user has INSERT permissions
- Verify data types match (especially JSONB for smart_attributes)

## License

MIT License - feel free to modify and use as needed!

## Credits

- BackBlaze failure probability tables extracted from [SnapRAID](https://github.com/amadvance/snapraid)
- Based on BackBlaze's 2014 hard drive reliability dataset