Complete rewrite: SnapRAID SMART logger with BackBlaze algorithm
Fetches SMART data from Scrutiny's InfluxDB and calculates failure probabilities using the exact BackBlaze tables from SnapRAID source. Key features: - Calculates both v12 (with attr 193) and v13 (without attr 193) algorithms - v12 matches SnapRAID pre-v13.0 (includes Load Cycle Count) - v13 matches modern SnapRAID v13.0+ (excludes Load Cycle Count) - Stores both values for trending and comparison - Correctly applies bit masks (16-bit for 187/188, 32-bit for others) - Annualizes monthly rates and applies Poisson distribution - Logs to PostgreSQL with device metadata Solved the mystery: SnapRAID v12 uses attribute 193 which was removed in v13.0. Load cycle count has massive impact on failure predictions for some drives (80% vs 4% difference). 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>
This commit is contained in:
Wes
Claude Sonnet 4.5
+321
@@ -0,0 +1,321 @@
|
|||||||
|
# SnapRAID SMART Logger - Investigation Notes
|
||||||
|
|
||||||
|
## Project Goal
|
||||||
|
|
||||||
|
Create a Python utility that:
|
||||||
|
1. Fetches SMART data from Scrutiny's InfluxDB
|
||||||
|
2. Calculates failure probability **exactly like SnapRAID does** (using BackBlaze algorithm)
|
||||||
|
3. Logs enhanced data to PostgreSQL
|
||||||
|
4. Eventually add alerts and automation
|
||||||
|
|
||||||
|
## Current Status: BLOCKED - Algorithm Mismatch
|
||||||
|
|
||||||
|
We successfully built the infrastructure but **cannot match SnapRAID's failure probability calculations**.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## The Mystery: 81% vs 4.63%
|
||||||
|
|
||||||
|
### What SnapRAID Reports for /dev/sdc (ZLW0A3QJ)
|
||||||
|
```
|
||||||
|
Temp Power Error FP Size
|
||||||
|
C OnDays Count TB Serial Device Disk
|
||||||
|
-----------------------------------------------------------------------
|
||||||
|
24 1968 0 81% 12.0 ZLW0A3QJ /dev/sdc d1
|
||||||
|
```
|
||||||
|
|
||||||
|
**Failure Probability: 81%**
|
||||||
|
|
||||||
|
### What smartctl -A Shows
|
||||||
|
```
|
||||||
|
ID# ATTRIBUTE_NAME VALUE WORST THRESH RAW_VALUE
|
||||||
|
5 Reallocated_Sector_Ct 100 100 010 0
|
||||||
|
187 Reported_Uncorrect 100 100 000 0
|
||||||
|
188 Command_Timeout 100 100 000 0
|
||||||
|
197 Current_Pending_Sector 100 100 000 0
|
||||||
|
198 Offline_Uncorrectable 100 100 000 0
|
||||||
|
```
|
||||||
|
|
||||||
|
**All critical attributes are PERFECT** (VALUE=100, RAW_VALUE=0)
|
||||||
|
|
||||||
|
### What Our BackBlaze Calculation Gives
|
||||||
|
Using the exact same BackBlaze tables extracted from SnapRAID source:
|
||||||
|
- With RAW_VALUE = 0: **Probability = 4.63%**
|
||||||
|
- With VALUE = 100: **Probability = 100%** (saturated, wrong)
|
||||||
|
|
||||||
|
**Neither matches SnapRAID's 81%!**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What We Know About SnapRAID's Algorithm
|
||||||
|
|
||||||
|
### Source Code Analysis
|
||||||
|
|
||||||
|
From `cmdline/device.c`:
|
||||||
|
|
||||||
|
```c
|
||||||
|
static double smart_afr(uint64_t* smart, const char* model)
|
||||||
|
{
|
||||||
|
double afr = 0;
|
||||||
|
uint64_t mask32 = 0xffffffffU;
|
||||||
|
uint64_t mask16 = 0xffffU;
|
||||||
|
|
||||||
|
// Attribute 5: Reallocated Sectors (32-bit mask)
|
||||||
|
if (smart[5] != SMART_UNASSIGNED) {
|
||||||
|
double r = smart_afr_value(SMART_5_R, SMART_5_STEP, smart[5] & mask32);
|
||||||
|
if (afr < r) afr = r;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Attribute 187: Reported Uncorrectable (16-bit mask)
|
||||||
|
if (smart[187] != SMART_UNASSIGNED) {
|
||||||
|
double r = smart_afr_value(SMART_187_R, SMART_187_STEP, smart[187] & mask16);
|
||||||
|
if (afr < r) afr = r;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Attribute 188: Command Timeout (16-bit, skipped for Seagate)
|
||||||
|
if (strncmp(model, "ST", 2) != 0 && smart[188] != SMART_UNASSIGNED) {
|
||||||
|
double r = smart_afr_value(SMART_188_R, SMART_188_STEP, smart[188] & mask16);
|
||||||
|
if (afr < r) afr = r;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Attribute 197: Current Pending Sectors (32-bit)
|
||||||
|
if (smart[197] != SMART_UNASSIGNED) {
|
||||||
|
double r = smart_afr_value(SMART_197_R, SMART_197_STEP, smart[197] & mask32);
|
||||||
|
if (afr < r) afr = r;
|
||||||
|
}
|
||||||
|
|
||||||
|
// Attribute 198: Offline Uncorrectable (32-bit)
|
||||||
|
if (smart[198] != SMART_UNASSIGNED) {
|
||||||
|
double r = smart_afr_value(SMART_198_R, SMART_198_STEP, smart[198] & mask32);
|
||||||
|
if (afr < r) afr = r;
|
||||||
|
}
|
||||||
|
|
||||||
|
return afr;
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Table Lookup Function
|
||||||
|
|
||||||
|
```c
|
||||||
|
static double smart_afr_value(double* tab, unsigned step, uint64_t value)
|
||||||
|
{
|
||||||
|
value /= step;
|
||||||
|
if (value >= SMART_MEASURES) value = SMART_MEASURES - 1;
|
||||||
|
return 365.0 / 30.0 * tab[value]; // Annualize monthly rate
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
### Final Probability Calculation
|
||||||
|
|
||||||
|
```c
|
||||||
|
// Poisson distribution: P(at least 1 failure) = 1 - e^(-AFR)
|
||||||
|
poisson_prob_n_or_more_failures(afr, 1) * 100
|
||||||
|
```
|
||||||
|
|
||||||
|
**Formula:** `P = (1 - e^(-AFR)) × 100`
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Key Findings
|
||||||
|
|
||||||
|
### 1. BackBlaze Tables Are Correct
|
||||||
|
We extracted all 5 tables (256 values each) from SnapRAID source:
|
||||||
|
- `SMART_5_R` (Reallocated Sectors)
|
||||||
|
- `SMART_187_R` (Reported Uncorrectable)
|
||||||
|
- `SMART_188_R` (Command Timeout)
|
||||||
|
- `SMART_197_R` (Current Pending Sectors)
|
||||||
|
- `SMART_198_R` (Offline Uncorrectable)
|
||||||
|
|
||||||
|
Tables match exactly - verified byte-for-byte.
|
||||||
|
|
||||||
|
### 2. Algorithm Is Correct
|
||||||
|
Our implementation:
|
||||||
|
1. ✅ Uses correct bit masks (16-bit for 187/188, 32-bit for 5/197/198)
|
||||||
|
2. ✅ Applies correct step size (all = 1)
|
||||||
|
3. ✅ Takes maximum AFR across attributes
|
||||||
|
4. ✅ Scales monthly to annual (365/30)
|
||||||
|
5. ✅ Applies Poisson: `1 - e^(-AFR)`
|
||||||
|
|
||||||
|
### 3. The Math Works Backwards
|
||||||
|
|
||||||
|
**If SnapRAID reports 81%, then:**
|
||||||
|
```
|
||||||
|
0.81 = 1 - e^(-AFR)
|
||||||
|
e^(-AFR) = 0.19
|
||||||
|
AFR = -ln(0.19) = 1.66
|
||||||
|
Monthly rate = 1.66 × (30/365) = 0.136
|
||||||
|
```
|
||||||
|
|
||||||
|
**Looking at SMART_187_R table:**
|
||||||
|
- `table[1] = 0.1287` → AFR = 1.565 → **P = 79%** ✓
|
||||||
|
- `table[2] = 0.1579` → AFR = 1.920 → **P = 85%** ✓
|
||||||
|
|
||||||
|
**Conclusion: SnapRAID is using an attribute value of 1 or 2 for one of the attributes!**
|
||||||
|
|
||||||
|
### 4. But We Can't Find That Value!
|
||||||
|
|
||||||
|
From both smartctl AND InfluxDB:
|
||||||
|
- RAW_VALUE = 0 for all 5 attributes
|
||||||
|
- VALUE (normalized) = 100 for all 5 attributes
|
||||||
|
- WORST = 100 for all 5 attributes
|
||||||
|
- No errors, no degradation visible
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Current Hypothesis
|
||||||
|
|
||||||
|
SnapRAID must be:
|
||||||
|
|
||||||
|
1. **Extracting a specific byte** from RAW_VALUE that contains 1-2?
|
||||||
|
2. **Using a different field** we haven't checked (FLAGS? WHEN_FAILED?)?
|
||||||
|
3. **Calculating a derived value** from multiple fields?
|
||||||
|
4. **Using a different version** of the algorithm than GitHub master?
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## What We Need
|
||||||
|
|
||||||
|
### Immediate Next Step
|
||||||
|
Check if RAW_VALUE has byte-level data we're missing:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
smartctl -A /dev/sdc | grep -E "^( 5|187|188|197|198)"
|
||||||
|
```
|
||||||
|
|
||||||
|
Need to see if there are parenthetical byte breakdowns like:
|
||||||
|
```
|
||||||
|
194 Temperature_Celsius 24 (0 15 0 0 0)
|
||||||
|
^^^^^^^^^^^^^ Individual bytes
|
||||||
|
```
|
||||||
|
|
||||||
|
### Alternative Approaches
|
||||||
|
|
||||||
|
1. **Run SnapRAID with debug/verbose mode** to see what values it reads
|
||||||
|
2. **Find the smartctl parsing code** in SnapRAID source
|
||||||
|
3. **Check SnapRAID version** - might be using different algorithm
|
||||||
|
4. **Contact SnapRAID author** if needed
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Project Files
|
||||||
|
|
||||||
|
### Core Implementation
|
||||||
|
- `smart_logger_v3.py` - Main logger (currently uses Scrutiny's failure rates)
|
||||||
|
- `backblaze_tables.py` - BackBlaze failure rate tables (256 values × 5 attributes)
|
||||||
|
- `schema_v2.sql` - PostgreSQL schema with devices + smart_metrics tables
|
||||||
|
- `populate_devices_final.sql` - Your 9 drives' metadata
|
||||||
|
|
||||||
|
### Debug Scripts
|
||||||
|
- `debug_smart_values.py` - Check InfluxDB raw SMART values
|
||||||
|
- `debug_scrutiny_failure_rate.py` - Check Scrutiny's calculated rates
|
||||||
|
- `debug_value_field.py` - Compare VALUE vs RAW_VALUE fields
|
||||||
|
- `test_backblaze_calc.py` - Test BackBlaze calculation with different inputs
|
||||||
|
|
||||||
|
### Setup
|
||||||
|
- `setup_database.sh` - PostgreSQL database creation
|
||||||
|
- `get_device_info.py` - Extract device WWNs from InfluxDB
|
||||||
|
- `requirements.txt` - Python dependencies
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Database Schema
|
||||||
|
|
||||||
|
### devices (manually populated)
|
||||||
|
```sql
|
||||||
|
device_wwn, device_path, serial_number, model, manufacturer,
|
||||||
|
capacity_bytes, size_tb, disk_role, notes
|
||||||
|
```
|
||||||
|
|
||||||
|
### smart_metrics (auto-populated by logger)
|
||||||
|
```sql
|
||||||
|
device_wwn, timestamp, temperature_celsius, power_on_days,
|
||||||
|
error_count, failure_probability_pct, smart_attributes (JSONB)
|
||||||
|
```
|
||||||
|
|
||||||
|
### Views
|
||||||
|
- `smart_latest` - Latest metrics per device with metadata
|
||||||
|
- `smart_high_risk` - Devices >50% failure probability
|
||||||
|
- `smart_summary` - SnapRAID-style summary table
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Database Connection Info
|
||||||
|
|
||||||
|
- **InfluxDB**: cyrion.c0smere.net:8086
|
||||||
|
- Org: scrutiny
|
||||||
|
- Bucket: metrics
|
||||||
|
- Token: (in .env)
|
||||||
|
|
||||||
|
- **PostgreSQL**: phlegethon.c0smere.net:5432
|
||||||
|
- Database: smart_monitoring
|
||||||
|
- User: smart_logger
|
||||||
|
- Password: saddog095
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Next Steps (After Solving Algorithm)
|
||||||
|
|
||||||
|
### Option 2 Features (After Option 1 Works)
|
||||||
|
|
||||||
|
1. **Alerts**
|
||||||
|
- Email/webhook when failure probability crosses thresholds
|
||||||
|
- Trend detection (rapidly increasing failure rate)
|
||||||
|
- Integration with monitoring systems
|
||||||
|
|
||||||
|
2. **SnapRAID Integration**
|
||||||
|
- Trigger scrub when high-risk drives detected
|
||||||
|
- Pre-scrub notifications
|
||||||
|
- Post-scrub verification
|
||||||
|
|
||||||
|
3. **Historical Analysis**
|
||||||
|
- Failure probability trends over time
|
||||||
|
- Predict when drives will cross risk thresholds
|
||||||
|
- Identify patterns across drive models/manufacturers
|
||||||
|
|
||||||
|
4. **Custom Queries**
|
||||||
|
- Drive health reports
|
||||||
|
- Lifetime statistics
|
||||||
|
- Comparison across drive families
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Comparison: SnapRAID vs Scrutiny vs Our Calculator
|
||||||
|
|
||||||
|
| Drive | SnapRAID | Scrutiny | Our Calc (raw=0) | Our Calc (val=100) |
|
||||||
|
|-------|----------|----------|------------------|---------------------|
|
||||||
|
| /dev/sdc (d1) | **81%** | 11% | 4.63% | 100% |
|
||||||
|
| /dev/sdb (d2) | 22% | 19% | 4.63% | 100% |
|
||||||
|
| /dev/sdd (d5) | 51% | 11% | 4.63% | 100% |
|
||||||
|
| /dev/sda (parity) | 49% | 16% | 4.63% | 100% |
|
||||||
|
| /dev/sdg (2-parity) | 30% | 12% | 4.63% | 100% |
|
||||||
|
| /dev/sdh (2-parity) | 4% | 5% | 4.63% | 100% |
|
||||||
|
| /dev/sde | 4% | 5% | 4.63% | 100% |
|
||||||
|
| /dev/sdf | 84% | 11% | 4.63% | 100% |
|
||||||
|
| /dev/sdi | 84% | 11% | 4.63% | 100% |
|
||||||
|
|
||||||
|
**Pattern:** SnapRAID shows high variance (4-84%), we show either 4.63% or 100%.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## Questions to Answer
|
||||||
|
|
||||||
|
1. **What exact value does SnapRAID read for attribute 5/187/188/197/198?**
|
||||||
|
2. **Is SnapRAID using RAW_VALUE, VALUE, WORST, or something else?**
|
||||||
|
3. **Could there be byte-level parsing we're missing?**
|
||||||
|
4. **What version of SnapRAID is running, and does it match GitHub master?**
|
||||||
|
5. **Is there a verbose mode to see what SnapRAID actually reads?**
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## References
|
||||||
|
|
||||||
|
- [SnapRAID GitHub](https://github.com/amadvance/snapraid)
|
||||||
|
- [SnapRAID device.c](https://github.com/amadvance/snapraid/blob/master/cmdline/device.c)
|
||||||
|
- [Scrutiny GitHub](https://github.com/AnalogJ/scrutiny)
|
||||||
|
- [BackBlaze Hard Drive Stats](https://www.backblaze.com/cloud-storage/resources/hard-drive-test-data)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
**Last Updated:** 2025-12-05
|
||||||
|
**Status:** 🔴 BLOCKED - Cannot match SnapRAID algorithm
|
||||||
|
**Blocker:** Unknown how SnapRAID calculates 81% with all attributes = 0/100
|
||||||
@@ -0,0 +1,15 @@
|
|||||||
|
# InfluxDB Configuration
|
||||||
|
INFLUXDB_URL=http://your-docker-host:8086
|
||||||
|
INFLUXDB_TOKEN=your-scrutiny-token
|
||||||
|
INFLUXDB_ORG=scrutiny
|
||||||
|
INFLUXDB_BUCKET=metrics
|
||||||
|
|
||||||
|
# Scrutiny API Configuration (defaults to INFLUXDB_URL with port 8080)
|
||||||
|
SCRUTINY_API_URL=http://your-docker-host:8080
|
||||||
|
|
||||||
|
# PostgreSQL Configuration
|
||||||
|
POSTGRES_HOST=localhost
|
||||||
|
POSTGRES_PORT=5432
|
||||||
|
POSTGRES_DB=smart_monitoring
|
||||||
|
POSTGRES_USER=postgres
|
||||||
|
POSTGRES_PASSWORD=your-password
|
||||||
+24
@@ -0,0 +1,24 @@
|
|||||||
|
# Python
|
||||||
|
venv/
|
||||||
|
__pycache__/
|
||||||
|
*.pyc
|
||||||
|
*.pyo
|
||||||
|
*.pyd
|
||||||
|
.Python
|
||||||
|
|
||||||
|
# Environment
|
||||||
|
.env
|
||||||
|
|
||||||
|
# IDE
|
||||||
|
.vscode/
|
||||||
|
.idea/
|
||||||
|
*.swp
|
||||||
|
*.swo
|
||||||
|
|
||||||
|
# Output files
|
||||||
|
explore_output.txt
|
||||||
|
*.log
|
||||||
|
|
||||||
|
# OS
|
||||||
|
.DS_Store
|
||||||
|
Thumbs.db
|
||||||
@@ -0,0 +1,182 @@
|
|||||||
|
# SnapRAID SMART Logger
|
||||||
|
|
||||||
|
A Python utility that collects SMART data from Scrutiny's InfluxDB, calculates BackBlaze-based failure probabilities (like SnapRAID does), and logs the enhanced data to PostgreSQL.
|
||||||
|
|
||||||
|
## Features
|
||||||
|
|
||||||
|
- ✅ Connects to Scrutiny's InfluxDB to fetch SMART attributes
|
||||||
|
- ✅ Implements SnapRAID's BackBlaze failure probability algorithm
|
||||||
|
- ✅ Calculates annual failure rate (AFR) for each disk
|
||||||
|
- ✅ Stores historical data in PostgreSQL with time-series support
|
||||||
|
- ✅ Provides views for latest metrics and high-risk devices
|
||||||
|
|
||||||
|
## Quick Start
|
||||||
|
|
||||||
|
### 1. Install Dependencies
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python3 -m venv venv
|
||||||
|
source venv/bin/activate
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Configure Environment
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cp .env.example .env
|
||||||
|
# Edit .env with your actual values
|
||||||
|
```
|
||||||
|
|
||||||
|
Required configuration:
|
||||||
|
- `INFLUXDB_URL`: Your Scrutiny InfluxDB URL (e.g., `http://192.168.1.100:8086`)
|
||||||
|
- `INFLUXDB_TOKEN`: Your Scrutiny InfluxDB token
|
||||||
|
- `POSTGRES_*`: Your PostgreSQL connection details
|
||||||
|
|
||||||
|
### 3. Set Up PostgreSQL Schema
|
||||||
|
|
||||||
|
```bash
|
||||||
|
psql -h <host> -U <user> -d <database> -f schema.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. Explore Scrutiny's InfluxDB Schema
|
||||||
|
|
||||||
|
**IMPORTANT**: Run this first to understand how Scrutiny stores SMART data:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python explore_influx.py
|
||||||
|
```
|
||||||
|
|
||||||
|
This will show you:
|
||||||
|
- Available measurements in the `metrics` bucket
|
||||||
|
- Field names for each measurement
|
||||||
|
- Sample data structure
|
||||||
|
- Tags used for device identification
|
||||||
|
|
||||||
|
### 5. Customize the Logger
|
||||||
|
|
||||||
|
Based on the output from `explore_influx.py`, update `smart_logger.py`:
|
||||||
|
|
||||||
|
1. **Update the InfluxDB query** (`fetch_smart_data` method):
|
||||||
|
- Set correct `_measurement` name
|
||||||
|
- Map field names to device properties (temp, serial, model, etc.)
|
||||||
|
- Map SMART attribute fields to IDs (5, 187, 188, 197, 198)
|
||||||
|
|
||||||
|
2. **Adjust device detection**:
|
||||||
|
- Update how device path, serial, and model are extracted
|
||||||
|
- Add disk role mapping if you want to match SnapRAID roles
|
||||||
|
|
||||||
|
### 6. Run the Logger
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python smart_logger.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Output Format
|
||||||
|
|
||||||
|
The logger will display a summary like SnapRAID:
|
||||||
|
|
||||||
|
```
|
||||||
|
Device: /dev/sda
|
||||||
|
Serial: ZX20AAMV
|
||||||
|
Model: WDC WD200EFAX-68FB5N0
|
||||||
|
Temp: 25°C
|
||||||
|
Power-On: 636 days
|
||||||
|
Failure Probability: 49.00%
|
||||||
|
|
||||||
|
================================================================================
|
||||||
|
Summary:
|
||||||
|
================================================================================
|
||||||
|
🔴 HIGH RISK /dev/sdc: 81.00% failure probability
|
||||||
|
🟢 OK /dev/sdb: 22.00% failure probability
|
||||||
|
```
|
||||||
|
|
||||||
|
## PostgreSQL Schema
|
||||||
|
|
||||||
|
### Tables
|
||||||
|
|
||||||
|
- **`smart_metrics`**: Main table storing all SMART data snapshots
|
||||||
|
- Includes temperature, power-on time, error counts
|
||||||
|
- Stores calculated failure probability
|
||||||
|
- Raw SMART attributes as JSONB for flexibility
|
||||||
|
|
||||||
|
### Views
|
||||||
|
|
||||||
|
- **`smart_latest`**: Latest metrics for each device
|
||||||
|
- **`smart_high_risk`**: Devices with >50% annual failure probability
|
||||||
|
|
||||||
|
### Example Queries
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Get latest status for all devices
|
||||||
|
SELECT device_path, serial_number, temperature_celsius,
|
||||||
|
failure_probability_pct, timestamp
|
||||||
|
FROM smart_latest
|
||||||
|
ORDER BY failure_probability_pct DESC;
|
||||||
|
|
||||||
|
-- Historical trend for a specific disk
|
||||||
|
SELECT timestamp, temperature_celsius, failure_probability_pct
|
||||||
|
FROM smart_metrics
|
||||||
|
WHERE serial_number = 'ZX20AAMV'
|
||||||
|
ORDER BY timestamp DESC
|
||||||
|
LIMIT 100;
|
||||||
|
|
||||||
|
-- Check specific SMART attribute over time
|
||||||
|
SELECT timestamp,
|
||||||
|
(smart_attributes->>'5')::int as reallocated_sectors,
|
||||||
|
(smart_attributes->>'197')::int as pending_sectors
|
||||||
|
FROM smart_metrics
|
||||||
|
WHERE device_path = '/dev/sda'
|
||||||
|
ORDER BY timestamp DESC;
|
||||||
|
```
|
||||||
|
|
||||||
|
## BackBlaze Algorithm
|
||||||
|
|
||||||
|
The failure probability calculation is based on SnapRAID's implementation, which uses BackBlaze's 2014 dataset (47,322 disk observations) to create lookup tables for these SMART attributes:
|
||||||
|
|
||||||
|
- **Attribute 5**: Reallocated Sectors Count
|
||||||
|
- **Attribute 187**: Reported Uncorrectable Errors
|
||||||
|
- **Attribute 188**: Command Timeout (excluded for Seagate disks)
|
||||||
|
- **Attribute 197**: Current Pending Sector Count
|
||||||
|
- **Attribute 198**: Offline Uncorrectable Sector Count
|
||||||
|
|
||||||
|
The algorithm returns the **maximum** AFR across all attributes (since they're correlated, not independent).
|
||||||
|
|
||||||
|
## Automation
|
||||||
|
|
||||||
|
To run this periodically, add a cron job:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run every hour
|
||||||
|
0 * * * * /path/to/venv/bin/python /path/to/smart_logger.py >> /var/log/smart_logger.log 2>&1
|
||||||
|
```
|
||||||
|
|
||||||
|
Or use systemd timer, or your preferred scheduler.
|
||||||
|
|
||||||
|
## Troubleshooting
|
||||||
|
|
||||||
|
### InfluxDB Connection Issues
|
||||||
|
|
||||||
|
- Verify the URL and port are correct
|
||||||
|
- Check that the token has read permissions for the `metrics` bucket
|
||||||
|
- Ensure InfluxDB is accessible from the machine running this script
|
||||||
|
|
||||||
|
### No Data Returned
|
||||||
|
|
||||||
|
- Run `explore_influx.py` to check the data structure
|
||||||
|
- Verify Scrutiny is actively collecting data
|
||||||
|
- Check the time range in the query (`-5m` might be too short)
|
||||||
|
|
||||||
|
### PostgreSQL Insert Failures
|
||||||
|
|
||||||
|
- Ensure the schema was created (`schema.sql`)
|
||||||
|
- Check that the database user has INSERT permissions
|
||||||
|
- Verify data types match (especially JSONB for smart_attributes)
|
||||||
|
|
||||||
|
## License
|
||||||
|
|
||||||
|
MIT License - feel free to modify and use as needed!
|
||||||
|
|
||||||
|
## Credits
|
||||||
|
|
||||||
|
- BackBlaze failure probability tables extracted from [SnapRAID](https://github.com/amadvance/snapraid)
|
||||||
|
- Based on BackBlaze's 2014 hard drive reliability dataset
|
||||||
@@ -0,0 +1,95 @@
|
|||||||
|
# Scrutiny InfluxDB Schema Notes
|
||||||
|
|
||||||
|
Based on analysis of your `explore_output.txt`, here's what I discovered:
|
||||||
|
|
||||||
|
## InfluxDB Structure
|
||||||
|
|
||||||
|
### Measurements
|
||||||
|
- `smart` - SMART attribute data
|
||||||
|
- `temp` - Temperature data (simpler format)
|
||||||
|
|
||||||
|
### Tags (Device Identifiers)
|
||||||
|
- `device_wwn` - World Wide Name, unique identifier (e.g., `0x5000c500744487c5`)
|
||||||
|
- `device_protocol` - Protocol type (`ATA`, `NVME`, etc.)
|
||||||
|
|
||||||
|
### Fields in `smart` Measurement
|
||||||
|
|
||||||
|
**Basic Metrics:**
|
||||||
|
- `temp` - Temperature in Celsius
|
||||||
|
- `power_on_hours` - Total power-on hours
|
||||||
|
- `power_cycle_count` - Number of power cycles
|
||||||
|
|
||||||
|
**SMART Attributes** (pattern: `attr.{id}.{property}`):
|
||||||
|
|
||||||
|
For each SMART attribute ID (e.g., 5, 187, 188, 194, 197, 198), Scrutiny stores:
|
||||||
|
- `attr.{id}.attribute_id` - The attribute ID itself
|
||||||
|
- `attr.{id}.raw_value` - **Raw value (what we need for calculations!)**
|
||||||
|
- `attr.{id}.raw_string` - String representation
|
||||||
|
- `attr.{id}.transformed_value` - Normalized value (0-100)
|
||||||
|
- `attr.{id}.value` - Current value
|
||||||
|
- `attr.{id}.thresh` - Threshold
|
||||||
|
- `attr.{id}.worst` - Worst value seen
|
||||||
|
- `attr.{id}.status` - Status indicator
|
||||||
|
- `attr.{id}.status_reason` - Status explanation
|
||||||
|
- `attr.{id}.failure_rate` - Scrutiny's own failure rate calculation
|
||||||
|
- `attr.{id}.when_failed` - Failure timestamp (if applicable)
|
||||||
|
|
||||||
|
## Key SMART Attributes for Failure Prediction
|
||||||
|
|
||||||
|
BackBlaze analysis (used by SnapRAID) focuses on these attributes:
|
||||||
|
|
||||||
|
| ID | Name | Description |
|
||||||
|
|-----|------|-------------|
|
||||||
|
| 5 | Reallocated Sectors Count | Sectors remapped due to errors |
|
||||||
|
| 187 | Reported Uncorrectable Errors | Errors that couldn't be corrected |
|
||||||
|
| 188 | Command Timeout | Commands that timed out |
|
||||||
|
| 197 | Current Pending Sector Count | Sectors waiting for remapping |
|
||||||
|
| 198 | Offline Uncorrectable Sector Count | Errors found during offline scan |
|
||||||
|
|
||||||
|
## Your Devices (from explore_output.txt)
|
||||||
|
|
||||||
|
Found 5 devices with these WWNs:
|
||||||
|
1. `0x5000c500744487c5`
|
||||||
|
2. `0x5000c500c455c45e`
|
||||||
|
3. `0x5000c500c46b0832`
|
||||||
|
4. `0x5000c500c570663e`
|
||||||
|
5. `0x5000c500e584b9c8`
|
||||||
|
|
||||||
|
## Device Metadata Location
|
||||||
|
|
||||||
|
**Important:** Serial number, model, device path (`/dev/sdX`) are **NOT** stored in InfluxDB!
|
||||||
|
|
||||||
|
They are stored in:
|
||||||
|
1. **Scrutiny's SQLite database** (`/opt/scrutiny/config/scrutiny.db`)
|
||||||
|
2. **Scrutiny's REST API** endpoints:
|
||||||
|
- `/api/summary` - All devices with metadata
|
||||||
|
- `/api/device/{wwn}/smart` - Individual device data
|
||||||
|
|
||||||
|
## Solution Used
|
||||||
|
|
||||||
|
The `smart_logger_v2.py` script:
|
||||||
|
1. **Fetches device metadata** from Scrutiny's `/api/summary` endpoint
|
||||||
|
2. **Fetches SMART data** from InfluxDB (faster than API for historical queries)
|
||||||
|
3. **Calculates failure probability** using BackBlaze tables
|
||||||
|
4. **Exports to PostgreSQL** with full metadata
|
||||||
|
|
||||||
|
## Example InfluxDB Query
|
||||||
|
|
||||||
|
```flux
|
||||||
|
from(bucket: "metrics")
|
||||||
|
|> range(start: -1h)
|
||||||
|
|> filter(fn: (r) => r._measurement == "smart")
|
||||||
|
|> filter(fn: (r) =>
|
||||||
|
r._field == "temp" or
|
||||||
|
r._field == "power_on_hours" or
|
||||||
|
r._field == "attr.5.raw_value" or
|
||||||
|
r._field == "attr.187.raw_value" or
|
||||||
|
r._field == "attr.188.raw_value" or
|
||||||
|
r._field == "attr.197.raw_value" or
|
||||||
|
r._field == "attr.198.raw_value"
|
||||||
|
)
|
||||||
|
|> last()
|
||||||
|
|> pivot(rowKey:["device_wwn"], columnKey: ["_field"], valueColumn: "_value")
|
||||||
|
```
|
||||||
|
|
||||||
|
This groups all fields by device WWN, making it easy to get all SMART data per device.
|
||||||
@@ -0,0 +1,170 @@
|
|||||||
|
# Setup Guide - Smart Logger V3
|
||||||
|
|
||||||
|
This version uses a **PostgreSQL devices table** for metadata instead of the Scrutiny API. Much simpler!
|
||||||
|
|
||||||
|
## Quick Setup
|
||||||
|
|
||||||
|
### 1. Create the Database Schema
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# Run the new schema (includes devices table)
|
||||||
|
psql -h <your-pg-host> -U <user> -d smart_monitoring -f schema_v2.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Get Your Device WWNs
|
||||||
|
|
||||||
|
Run the helper script to see all devices in InfluxDB:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python get_device_info.py
|
||||||
|
```
|
||||||
|
|
||||||
|
This will show:
|
||||||
|
```
|
||||||
|
Device 1:
|
||||||
|
WWN: 0x5000c500744487c5
|
||||||
|
Protocol: ATA
|
||||||
|
Temperature: 24°C
|
||||||
|
Power-On Time: 1968 days (47232 hours)
|
||||||
|
|
||||||
|
Device 2:
|
||||||
|
...
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Populate Device Metadata
|
||||||
|
|
||||||
|
Now match each WWN with its actual device info from `snapraid smart`:
|
||||||
|
|
||||||
|
```
|
||||||
|
Temp Power Error FP Size
|
||||||
|
C OnDays Count TB Serial Device Disk
|
||||||
|
-----------------------------------------------------------------------
|
||||||
|
24 1968 0 81% 12.0 ZLW0A3QJ /dev/sdc d1
|
||||||
|
25 637 0 22% 20.0 ZX22EJ3Y /dev/sdb d2
|
||||||
|
21 1014 0 51% 12.0 ZTN1BNEZ /dev/sdd d5
|
||||||
|
25 636 0 49% 20.0 ZX20AAMV /dev/sda parity
|
||||||
|
25 2540 0 30% 10.0 2TJ97M6D /dev/sdg 2-parity
|
||||||
|
28 2354 1 4% 10.0 7JJVJ65C /dev/sdh 2-parity
|
||||||
|
32 1983 0 4% 3.0 YVGGG6EC /dev/sde -
|
||||||
|
30 2072 0 84% 8.0 ZA1GX9F4 /dev/sdf -
|
||||||
|
- - 0 - - 25243X801438 /dev/nvme0n1 -
|
||||||
|
32 2074 0 84% 8.0 ZA1GW0XM /dev/sdi -
|
||||||
|
```
|
||||||
|
|
||||||
|
Match the power-on days from `get_device_info.py` with the power-on days from `snapraid smart` to identify each WWN.
|
||||||
|
|
||||||
|
For example:
|
||||||
|
- **WWN 0x5000c500744487c5** has **1968 days** → matches **/dev/sdc** (ZLW0A3QJ, d1)
|
||||||
|
|
||||||
|
### 4. Edit populate_devices.sql
|
||||||
|
|
||||||
|
Update `populate_devices.sql` with your actual device information:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000c500744487c5',
|
||||||
|
'/dev/sdc',
|
||||||
|
'ZLW0A3QJ',
|
||||||
|
'WDC WD120EFAX-68FB5N0', -- Get from smartctl -i /dev/sdc
|
||||||
|
'Western Digital',
|
||||||
|
12000000000000,
|
||||||
|
12.0,
|
||||||
|
'd1',
|
||||||
|
'Data drive 1'
|
||||||
|
);
|
||||||
|
```
|
||||||
|
|
||||||
|
Repeat for all 9 drives.
|
||||||
|
|
||||||
|
### 5. Load the Data
|
||||||
|
|
||||||
|
```bash
|
||||||
|
psql -h <your-pg-host> -U <user> -d smart_monitoring -f populate_devices.sql
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. Run the Logger!
|
||||||
|
|
||||||
|
```bash
|
||||||
|
python smart_logger_v3.py
|
||||||
|
```
|
||||||
|
|
||||||
|
## Expected Output
|
||||||
|
|
||||||
|
```
|
||||||
|
Connecting to InfluxDB at http://cyrion.c0smere.net:8086...
|
||||||
|
✓ Connected to InfluxDB
|
||||||
|
Connecting to PostgreSQL...
|
||||||
|
✓ Connected to PostgreSQL
|
||||||
|
Loading device metadata from database...
|
||||||
|
✓ Loaded metadata for 9 device(s)
|
||||||
|
Querying InfluxDB for SMART data...
|
||||||
|
✓ Found data for 9 device(s)
|
||||||
|
|
||||||
|
/dev/sdc
|
||||||
|
WWN: 0x5000c500744487c5
|
||||||
|
Serial: ZLW0A3QJ
|
||||||
|
Model: WDC WD120EFAX
|
||||||
|
Role: d1
|
||||||
|
Temp: 24°C
|
||||||
|
Power-On: 1968 days
|
||||||
|
Size: 12.0 TB
|
||||||
|
SMART Attrs: [5, 187, 197, 198]
|
||||||
|
Failure Probability: 81.00%
|
||||||
|
|
||||||
|
...
|
||||||
|
|
||||||
|
================================================================================
|
||||||
|
SnapRAID-style SMART Summary
|
||||||
|
================================================================================
|
||||||
|
Temp Power Error FP Size
|
||||||
|
C OnDays Count TB Serial Device Disk
|
||||||
|
--------------------------------------------------------------------------------
|
||||||
|
24 1968 0 81% 12.0 ZLW0A3QJ /dev/sdc d1
|
||||||
|
25 637 0 22% 20.0 ZX22EJ3Y /dev/sdb d2
|
||||||
|
21 1014 0 51% 12.0 ZTN1BNEZ /dev/sdd d5
|
||||||
|
25 636 0 49% 20.0 ZX20AAMV /dev/sda parity
|
||||||
|
25 2540 0 30% 10.0 2TJ97M6D /dev/sdg 2-parity
|
||||||
|
```
|
||||||
|
|
||||||
|
## Querying the Data
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Latest status for all drives
|
||||||
|
SELECT * FROM smart_summary;
|
||||||
|
|
||||||
|
-- Historical data for a specific drive
|
||||||
|
SELECT timestamp, temperature_celsius, failure_probability_pct
|
||||||
|
FROM smart_metrics sm
|
||||||
|
JOIN devices d ON sm.device_wwn = d.device_wwn
|
||||||
|
WHERE d.device_path = '/dev/sdc'
|
||||||
|
ORDER BY timestamp DESC
|
||||||
|
LIMIT 100;
|
||||||
|
|
||||||
|
-- High-risk drives
|
||||||
|
SELECT * FROM smart_high_risk;
|
||||||
|
```
|
||||||
|
|
||||||
|
## Automation
|
||||||
|
|
||||||
|
Add to cron to run hourly:
|
||||||
|
|
||||||
|
```cron
|
||||||
|
0 * * * * cd /home/wes/projects/snapraid_smart_logger && /home/wes/projects/snapraid_smart_logger/venv/bin/python smart_logger_v3.py >> /var/log/smart_logger.log 2>&1
|
||||||
|
```
|
||||||
|
|
||||||
|
## Updating Device Info
|
||||||
|
|
||||||
|
If you add/remove drives, just update the `devices` table:
|
||||||
|
|
||||||
|
```sql
|
||||||
|
-- Add a new drive
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role)
|
||||||
|
VALUES ('0xNEW_WWN', '/dev/sdj', 'NEWSERIAL', 'MODEL', 'MANUFACTURER', 12000000000000, 12.0, 'd6');
|
||||||
|
|
||||||
|
-- Remove a drive
|
||||||
|
DELETE FROM devices WHERE device_wwn = '0xOLD_WWN';
|
||||||
|
|
||||||
|
-- Update drive role
|
||||||
|
UPDATE devices SET disk_role = 'parity' WHERE device_path = '/dev/sda';
|
||||||
|
```
|
||||||
@@ -0,0 +1,283 @@
|
|||||||
|
"""
|
||||||
|
BackBlaze SMART failure rate tables extracted from SnapRAID source code.
|
||||||
|
Based on BackBlaze's 2014 dataset (47,322 disk observations).
|
||||||
|
Tables contain 256 monthly failure rates, scaled to annual rates using: AFR = (365/30) * monthly_rate
|
||||||
|
"""
|
||||||
|
|
||||||
|
import math
|
||||||
|
|
||||||
|
# SMART Attribute 5: Reallocated Sectors Count
|
||||||
|
SMART_5_STEP = 1
|
||||||
|
SMART_5_R = [
|
||||||
|
0.0026, 0.0748, 0.0919, 0.1013, 0.1079, 0.1137, 0.1194, 0.1235, 0.1301, 0.1398,
|
||||||
|
0.1453, 0.1490, 0.1528, 0.1566, 0.1595, 0.1635, 0.1656, 0.1701, 0.1718, 0.1740,
|
||||||
|
0.1762, 0.1787, 0.1808, 0.1833, 0.1858, 0.1885, 0.1901, 0.1915, 0.1934, 0.1958,
|
||||||
|
0.1975, 0.1993, 0.2014, 0.2048, 0.2068, 0.2088, 0.2109, 0.2120, 0.2137, 0.2160,
|
||||||
|
0.2173, 0.2214, 0.2226, 0.2237, 0.2262, 0.2277, 0.2292, 0.2304, 0.2338, 0.2369,
|
||||||
|
0.2381, 0.2396, 0.2411, 0.2427, 0.2445, 0.2462, 0.2472, 0.2488, 0.2496, 0.2504,
|
||||||
|
0.2514, 0.2525, 0.2535, 0.2544, 0.2554, 0.2571, 0.2583, 0.2601, 0.2622, 0.2631,
|
||||||
|
0.2635, 0.2644, 0.2659, 0.2675, 0.2682, 0.2692, 0.2701, 0.2707, 0.2712, 0.2726,
|
||||||
|
0.2745, 0.2767, 0.2778, 0.2784, 0.2800, 0.2814, 0.2834, 0.2839, 0.2851, 0.2877,
|
||||||
|
0.2883, 0.2891, 0.2900, 0.2907, 0.2916, 0.2934, 0.2950, 0.2969, 0.2975, 0.2983,
|
||||||
|
0.2999, 0.3006, 0.3013, 0.3021, 0.3033, 0.3054, 0.3066, 0.3074, 0.3082, 0.3094,
|
||||||
|
0.3106, 0.3112, 0.3120, 0.3137, 0.3141, 0.3145, 0.3151, 0.3159, 0.3169, 0.3174,
|
||||||
|
0.3181, 0.3194, 0.3215, 0.3219, 0.3231, 0.3234, 0.3237, 0.3242, 0.3255, 0.3270,
|
||||||
|
0.3283, 0.3286, 0.3289, 0.3304, 0.3315, 0.3322, 0.3347, 0.3361, 0.3382, 0.3384,
|
||||||
|
0.3395, 0.3398, 0.3401, 0.3405, 0.3411, 0.3431, 0.3435, 0.3442, 0.3447, 0.3450,
|
||||||
|
0.3455, 0.3464, 0.3472, 0.3486, 0.3497, 0.3501, 0.3509, 0.3517, 0.3531, 0.3535,
|
||||||
|
0.3540, 0.3565, 0.3569, 0.3576, 0.3579, 0.3584, 0.3590, 0.3594, 0.3599, 0.3621,
|
||||||
|
0.3627, 0.3642, 0.3649, 0.3655, 0.3658, 0.3667, 0.3683, 0.3699, 0.3704, 0.3707,
|
||||||
|
0.3711, 0.3715, 0.3718, 0.3721, 0.3727, 0.3740, 0.3744, 0.3748, 0.3753, 0.3756,
|
||||||
|
0.3761, 0.3766, 0.3775, 0.3794, 0.3801, 0.3804, 0.3813, 0.3817, 0.3823, 0.3831,
|
||||||
|
0.3847, 0.3875, 0.3881, 0.3886, 0.3890, 0.3893, 0.3896, 0.3900, 0.3907, 0.3923,
|
||||||
|
0.3925, 0.3933, 0.3936, 0.3961, 0.3971, 0.3981, 0.3989, 0.4007, 0.4012, 0.4018,
|
||||||
|
0.4023, 0.4027, 0.4041, 0.4048, 0.4056, 0.4073, 0.4079, 0.4086, 0.4104, 0.4107,
|
||||||
|
0.4109, 0.4112, 0.4118, 0.4133, 0.4139, 0.4144, 0.4146, 0.4148, 0.4164, 0.4165,
|
||||||
|
0.4174, 0.4191, 0.4197, 0.4201, 0.4204, 0.4210, 0.4213, 0.4216, 0.4221, 0.4231,
|
||||||
|
0.4235, 0.4237, 0.4239, 0.4241, 0.4244, 0.4249,
|
||||||
|
]
|
||||||
|
|
||||||
|
# SMART Attribute 187: Reported Uncorrectable Errors
|
||||||
|
SMART_187_STEP = 1
|
||||||
|
SMART_187_R = [
|
||||||
|
0.0039, 0.1287, 0.1579, 0.1776, 0.1905, 0.2013, 0.2226, 0.3263, 0.3612, 0.3869,
|
||||||
|
0.4086, 0.4292, 0.4559, 0.5278, 0.5593, 0.5847, 0.6124, 0.6345, 0.6517, 0.6995,
|
||||||
|
0.7308, 0.7541, 0.7814, 0.8122, 0.8306, 0.8839, 0.9100, 0.9505, 0.9906, 1.0254,
|
||||||
|
1.0483, 1.1060, 1.1280, 1.1624, 1.1895, 1.2138, 1.2452, 1.2864, 1.3120, 1.3369,
|
||||||
|
1.3705, 1.3894, 1.4055, 1.4218, 1.4434, 1.4670, 1.4834, 1.4993, 1.5174, 1.5400,
|
||||||
|
1.5572, 1.5689, 1.5808, 1.6198, 1.6346, 1.6405, 1.6570, 1.6618, 1.6755, 1.6877,
|
||||||
|
1.7100, 1.7258, 1.7347, 1.7814, 1.7992, 1.8126, 1.8225, 1.8269, 1.8341, 1.8463,
|
||||||
|
1.8765, 1.8850, 1.9005, 1.9281, 1.9398, 1.9618, 1.9702, 1.9905, 2.0099, 2.0480,
|
||||||
|
2.0565, 2.0611, 2.0709, 2.0846, 2.0895, 2.0958, 2.1008, 2.1055, 2.1097, 2.1235,
|
||||||
|
2.1564, 2.1737, 2.1956, 2.1989, 2.2015, 2.2148, 2.2355, 2.2769, 2.2940, 2.3045,
|
||||||
|
2.3096, 2.3139, 2.3344, 2.3669, 2.3779, 2.3941, 2.4036, 2.4396, 2.4473, 2.4525,
|
||||||
|
2.4656, 2.4762, 2.4787, 2.5672, 2.5732, 2.5755, 2.5794, 2.5886, 2.6100, 2.6144,
|
||||||
|
2.6341, 2.6614, 2.6679, 2.6796, 2.6847, 2.6872, 2.6910, 2.6934, 2.6995, 2.7110,
|
||||||
|
2.7179, 2.7204, 2.7232, 2.7282, 2.7355, 2.7375, 2.7422, 2.7558, 2.7580, 2.7643,
|
||||||
|
2.7767, 2.7770, 2.8016, 2.9292, 2.9294, 2.9337, 2.9364, 2.9409, 2.9436, 2.9457,
|
||||||
|
2.9466, 2.9498, 2.9543, 2.9570, 2.9573, 2.9663, 2.9708, 2.9833, 2.9859, 2.9895,
|
||||||
|
2.9907, 2.9932, 2.9935, 3.0021, 3.0035, 3.0079, 3.0103, 3.0126, 3.0151, 3.0266,
|
||||||
|
3.0288, 3.0320, 3.0330, 3.0343, 3.0373, 3.0387, 3.0438, 3.0570, 3.0579, 3.0616,
|
||||||
|
3.0655, 3.0728, 3.0771, 3.0794, 3.0799, 3.0812, 3.1769, 3.1805, 3.1819, 3.1860,
|
||||||
|
3.1869, 3.2004, 3.2016, 3.2025, 3.2070, 3.2129, 3.2173, 3.2205, 3.2254, 3.2263,
|
||||||
|
3.2300, 3.2413, 3.2543, 3.2580, 3.2595, 3.2611, 3.2624, 3.2787, 3.2798, 3.2809,
|
||||||
|
3.2823, 3.2833, 3.2834, 3.2853, 3.2866, 3.3332, 3.3580, 3.3595, 3.3625, 3.3631,
|
||||||
|
3.3667, 3.3702, 3.3737, 3.3742, 3.3747, 3.3769, 3.3775, 3.3791, 3.3809, 3.3813,
|
||||||
|
3.3814, 3.3822, 3.3827, 3.3828, 3.3833, 3.3833, 3.3843, 3.3882, 3.3963, 3.4047,
|
||||||
|
3.4057, 3.4213, 3.4218, 3.4230, 3.4231, 3.4240, 3.4262, 3.4283, 3.4283, 3.4288,
|
||||||
|
3.4293, 3.4302, 3.4317, 3.4478, 3.4486, 3.4520,
|
||||||
|
]
|
||||||
|
|
||||||
|
# SMART Attribute 188: Command Timeout
|
||||||
|
SMART_188_STEP = 1
|
||||||
|
SMART_188_R = [
|
||||||
|
0.0025, 0.0129, 0.0182, 0.0215, 0.0236, 0.0257, 0.0279, 0.0308, 0.0341, 0.0382,
|
||||||
|
0.0430, 0.0491, 0.0565, 0.0658, 0.0770, 0.0906, 0.1037, 0.1197, 0.1355, 0.1525,
|
||||||
|
0.1686, 0.1864, 0.2011, 0.2157, 0.2281, 0.2404, 0.2505, 0.2591, 0.2676, 0.2766,
|
||||||
|
0.2827, 0.2913, 0.2999, 0.3100, 0.3185, 0.3298, 0.3361, 0.3446, 0.3506, 0.3665,
|
||||||
|
0.3699, 0.3820, 0.3890, 0.4059, 0.4108, 0.4255, 0.4290, 0.4424, 0.4473, 0.4617,
|
||||||
|
0.4667, 0.4770, 0.4829, 0.4977, 0.4997, 0.5102, 0.5137, 0.5283, 0.5316, 0.5428,
|
||||||
|
0.5480, 0.5597, 0.5634, 0.5791, 0.5826, 0.5929, 0.5945, 0.6025, 0.6102, 0.6175,
|
||||||
|
0.6245, 0.6313, 0.6421, 0.6468, 0.6497, 0.6557, 0.6570, 0.6647, 0.6698, 0.6769,
|
||||||
|
0.6849, 0.6884, 0.6925, 0.7025, 0.7073, 0.7161, 0.7223, 0.7256, 0.7280, 0.7411,
|
||||||
|
0.7445, 0.7530, 0.7628, 0.7755, 0.7900, 0.8006, 0.8050, 0.8098, 0.8132, 0.8192,
|
||||||
|
0.8230, 0.8293, 0.8356, 0.8440, 0.8491, 0.8672, 0.8766, 0.8907, 0.8934, 0.8992,
|
||||||
|
0.9062, 0.9111, 0.9209, 0.9290, 0.9329, 0.9378, 0.9385, 0.9402, 0.9427, 0.9448,
|
||||||
|
0.9459, 0.9568, 0.9626, 0.9628, 0.9730, 0.9765, 0.9797, 0.9825, 0.9873, 0.9902,
|
||||||
|
0.9926, 0.9991, 1.0031, 1.0044, 1.0062, 1.0120, 1.0148, 1.0188, 1.0218, 1.0231,
|
||||||
|
1.0249, 1.0277, 1.0335, 1.0355, 1.0417, 1.0467, 1.0474, 1.0510, 1.0529, 1.0532,
|
||||||
|
1.0562, 1.0610, 1.0702, 1.0708, 1.0800, 1.0804, 1.0845, 1.1120, 1.1191, 1.1225,
|
||||||
|
1.1264, 1.1265, 1.1335, 1.1347, 1.1479, 1.1479, 1.1519, 1.1545, 1.1645, 1.1646,
|
||||||
|
1.1647, 1.1649, 1.1678, 1.1713, 1.1723, 1.1733, 1.1736, 1.1736, 1.1738, 1.1739,
|
||||||
|
1.1739, 1.1741, 1.1741, 1.1746, 1.1746, 1.1748, 1.1750, 1.1760, 1.1794, 1.1854,
|
||||||
|
1.1908, 1.1912, 1.1912, 1.1971, 1.2033, 1.2033, 1.2120, 1.2166, 1.2185, 1.2185,
|
||||||
|
1.2189, 1.2211, 1.2226, 1.2234, 1.2320, 1.2345, 1.2345, 1.2347, 1.2350, 1.2350,
|
||||||
|
1.2407, 1.2408, 1.2408, 1.2408, 1.2409, 1.2460, 1.2518, 1.2519, 1.2519, 1.2519,
|
||||||
|
1.2520, 1.2520, 1.2521, 1.2521, 1.2521, 1.2593, 1.2745, 1.2760, 1.2772, 1.2831,
|
||||||
|
1.2833, 1.2890, 1.2906, 1.3166, 1.3201, 1.3202, 1.3202, 1.3202, 1.3204, 1.3204,
|
||||||
|
1.3314, 1.3422, 1.3423, 1.3441, 1.3491, 1.3583, 1.3602, 1.3606, 1.3636, 1.3650,
|
||||||
|
1.3661, 1.3703, 1.3708, 1.3716, 1.3730, 1.3731,
|
||||||
|
]
|
||||||
|
|
||||||
|
# SMART Attribute 197: Current Pending Sector Count
|
||||||
|
SMART_197_STEP = 1
|
||||||
|
SMART_197_R = [
|
||||||
|
0.0028, 0.2972, 0.3883, 0.4363, 0.4644, 0.4813, 0.4948, 0.5051, 0.5499, 0.8535,
|
||||||
|
0.8678, 0.8767, 0.8882, 0.8933, 0.9012, 0.9076, 0.9368, 1.1946, 1.2000, 1.2110,
|
||||||
|
1.2177, 1.2305, 1.2385, 1.2447, 1.2699, 1.4713, 1.4771, 1.4802, 1.4887, 1.5292,
|
||||||
|
1.5384, 1.5442, 1.5645, 1.7700, 1.7755, 1.7778, 1.7899, 1.7912, 1.7991, 1.7998,
|
||||||
|
1.8090, 1.9974, 1.9992, 2.0088, 2.0132, 2.0146, 2.0161, 2.0171, 2.0273, 2.1845,
|
||||||
|
2.1866, 2.1877, 2.1900, 2.1922, 2.1944, 2.1974, 2.2091, 2.3432, 2.3459, 2.3463,
|
||||||
|
2.3468, 2.3496, 2.3503, 2.3533, 2.3593, 2.4604, 2.4606, 2.4609, 2.4612, 2.4620,
|
||||||
|
2.4626, 2.4638, 2.4689, 2.5575, 2.5581, 2.5586, 2.5586, 2.5588, 2.5602, 2.5602,
|
||||||
|
2.5648, 2.6769, 2.6769, 2.6769, 2.6794, 2.6805, 2.6811, 2.6814, 2.6862, 2.7742,
|
||||||
|
2.7755, 2.7771, 2.7780, 2.7790, 2.7797, 2.7807, 2.7871, 2.9466, 2.9478, 2.9492,
|
||||||
|
2.9612, 2.9618, 2.9624, 2.9628, 2.9669, 3.1467, 3.1481, 3.1494, 3.1499, 3.1504,
|
||||||
|
3.1507, 3.1509, 3.1532, 3.2675, 3.2681, 3.2703, 3.2712, 3.2714, 3.2726, 3.2726,
|
||||||
|
3.2743, 3.3376, 3.3379, 3.3382, 3.3397, 3.3403, 3.3410, 3.3410, 3.3429, 3.4052,
|
||||||
|
3.4052, 3.4052, 3.4052, 3.4052, 3.4053, 3.4053, 3.4075, 3.4616, 3.4616, 3.4616,
|
||||||
|
3.4616, 3.4616, 3.4616, 3.4620, 3.4634, 3.4975, 3.4975, 3.4975, 3.4975, 3.4979,
|
||||||
|
3.4979, 3.4979, 3.4998, 3.5489, 3.5489, 3.5489, 3.5489, 3.5489, 3.5493, 3.5497,
|
||||||
|
3.5512, 3.5827, 3.5828, 3.5828, 3.5828, 3.5828, 3.5828, 3.5828, 3.5844, 3.6251,
|
||||||
|
3.6251, 3.6251, 3.6267, 3.6267, 3.6271, 3.6271, 3.6279, 3.6562, 3.6562, 3.6563,
|
||||||
|
3.7206, 3.7242, 3.7332, 3.7332, 3.7346, 3.7548, 3.7548, 3.7553, 3.7576, 3.7581,
|
||||||
|
3.7586, 3.7587, 3.7600, 3.7773, 3.7812, 3.7836, 3.7841, 3.7842, 3.7851, 3.7856,
|
||||||
|
3.7876, 3.8890, 3.8890, 3.8890, 3.8890, 3.8890, 3.8890, 3.8890, 3.8897, 3.9111,
|
||||||
|
3.9114, 3.9114, 3.9114, 3.9114, 3.9114, 3.9114, 3.9126, 3.9440, 3.9440, 3.9440,
|
||||||
|
3.9440, 3.9440, 3.9498, 3.9498, 3.9509, 3.9783, 3.9783, 3.9784, 3.9784, 3.9784,
|
||||||
|
3.9784, 4.0012, 4.0019, 4.0406, 4.0413, 4.0413, 4.0413, 4.0413, 4.0414, 4.0414,
|
||||||
|
4.0421, 4.0552, 4.0552, 4.0558, 4.0558, 4.0558, 4.0558, 4.0558, 4.0563, 4.0753,
|
||||||
|
4.0753, 4.0760, 4.1131, 4.1131, 4.1131, 4.1131,
|
||||||
|
]
|
||||||
|
|
||||||
|
# SMART Attribute 193: Load Cycle Count (removed in SnapRAID v13.0)
|
||||||
|
# Included for compatibility with older SnapRAID versions
|
||||||
|
SMART_193_STEP = 649
|
||||||
|
SMART_193_R = [
|
||||||
|
0.0000, 0.0016, 0.0032, 0.0036, 0.0039,
|
||||||
|
0.0042, 0.0046, 0.0049, 0.0052, 0.0054,
|
||||||
|
0.0057, 0.0060, 0.0062, 0.0065, 0.0068,
|
||||||
|
0.0071, 0.0074, 0.0077, 0.0080, 0.0083,
|
||||||
|
0.0086, 0.0091, 0.0094, 0.0098, 0.0101,
|
||||||
|
0.0104, 0.0108, 0.0111, 0.0119, 0.0122,
|
||||||
|
0.0127, 0.0130, 0.0134, 0.0137, 0.0141,
|
||||||
|
0.0144, 0.0146, 0.0152, 0.0155, 0.0159,
|
||||||
|
0.0163, 0.0165, 0.0168, 0.0172, 0.0176,
|
||||||
|
0.0179, 0.0184, 0.0188, 0.0190, 0.0194,
|
||||||
|
0.0197, 0.0201, 0.0204, 0.0207, 0.0209,
|
||||||
|
0.0213, 0.0215, 0.0219, 0.0221, 0.0225,
|
||||||
|
0.0229, 0.0234, 0.0241, 0.0246, 0.0253,
|
||||||
|
0.0263, 0.0278, 0.0286, 0.0293, 0.0298,
|
||||||
|
0.0302, 0.0306, 0.0311, 0.0315, 0.0319,
|
||||||
|
0.0322, 0.0329, 0.0334, 0.0338, 0.0343,
|
||||||
|
0.0348, 0.0352, 0.0358, 0.0362, 0.0367,
|
||||||
|
0.0371, 0.0374, 0.0378, 0.0383, 0.0388,
|
||||||
|
0.0393, 0.0397, 0.0401, 0.0404, 0.0410,
|
||||||
|
0.0416, 0.0422, 0.0428, 0.0436, 0.0443,
|
||||||
|
0.0449, 0.0454, 0.0457, 0.0462, 0.0468,
|
||||||
|
0.0473, 0.0479, 0.0483, 0.0488, 0.0491,
|
||||||
|
0.0493, 0.0497, 0.0500, 0.0504, 0.0507,
|
||||||
|
0.0510, 0.0514, 0.0519, 0.0523, 0.0528,
|
||||||
|
0.0533, 0.0538, 0.0542, 0.0547, 0.0551,
|
||||||
|
0.0556, 0.0560, 0.0565, 0.0572, 0.0577,
|
||||||
|
0.0584, 0.0590, 0.0594, 0.0599, 0.0603,
|
||||||
|
0.0607, 0.0611, 0.0616, 0.0621, 0.0626,
|
||||||
|
0.0632, 0.0639, 0.0647, 0.0655, 0.0661,
|
||||||
|
0.0669, 0.0676, 0.0683, 0.0691, 0.0699,
|
||||||
|
0.0708, 0.0713, 0.0719, 0.0724, 0.0730,
|
||||||
|
0.0736, 0.0745, 0.0751, 0.0759, 0.0769,
|
||||||
|
0.0779, 0.0787, 0.0796, 0.0804, 0.0815,
|
||||||
|
0.0825, 0.0833, 0.0840, 0.0847, 0.0854,
|
||||||
|
0.0859, 0.0865, 0.0873, 0.0881, 0.0890,
|
||||||
|
0.0900, 0.0912, 0.0919, 0.0929, 0.0942,
|
||||||
|
0.0956, 0.0965, 0.0976, 0.0986, 0.0995,
|
||||||
|
0.1006, 0.1019, 0.1031, 0.1038, 0.1045,
|
||||||
|
0.1051, 0.1058, 0.1066, 0.1072, 0.1077,
|
||||||
|
0.1084, 0.1091, 0.1099, 0.1104, 0.1111,
|
||||||
|
0.1118, 0.1127, 0.1135, 0.1142, 0.1149,
|
||||||
|
0.1157, 0.1163, 0.1168, 0.1173, 0.1179,
|
||||||
|
0.1184, 0.1189, 0.1195, 0.1203, 0.1208,
|
||||||
|
0.1213, 0.1223, 0.1231, 0.1240, 0.1246,
|
||||||
|
0.1252, 0.1260, 0.1269, 0.1276, 0.1287,
|
||||||
|
0.1303, 0.1311, 0.1319, 0.1328, 0.1335,
|
||||||
|
0.1341, 0.1348, 0.1362, 0.1373, 0.1380,
|
||||||
|
0.1387, 0.1392, 0.1398, 0.1403, 0.1408,
|
||||||
|
0.1412, 0.1418, 0.1422, 0.1428, 0.1434,
|
||||||
|
0.1439, 0.1445, 0.1451, 0.1457, 0.1464,
|
||||||
|
0.1469, 0.1475, 0.1480, 0.1486, 0.1491,
|
||||||
|
0.1498,
|
||||||
|
]
|
||||||
|
|
||||||
|
# SMART Attribute 198: Offline Uncorrectable Sector Count
|
||||||
|
SMART_198_STEP = 1
|
||||||
|
SMART_198_R = [
|
||||||
|
0.0030, 0.5479, 0.5807, 0.5949, 0.6046, 0.6086, 0.6139, 0.6224, 0.6639, 1.0308,
|
||||||
|
1.0329, 1.0364, 1.0371, 1.0387, 1.0399, 1.0421, 1.0675, 1.3730, 1.3733, 1.3741,
|
||||||
|
1.3741, 1.3752, 1.3794, 1.3800, 1.3985, 1.6291, 1.6303, 1.6309, 1.6352, 1.6384,
|
||||||
|
1.6448, 1.6464, 1.6645, 1.8949, 1.8951, 1.8962, 1.9073, 1.9073, 1.9152, 1.9161,
|
||||||
|
1.9240, 2.1308, 2.1315, 2.1328, 2.1328, 2.1328, 2.1328, 2.1329, 2.1439, 2.3203,
|
||||||
|
2.3205, 2.3205, 2.3205, 2.3205, 2.3205, 2.3205, 2.3265, 2.4729, 2.4729, 2.4729,
|
||||||
|
2.4729, 2.4729, 2.4729, 2.4729, 2.4778, 2.5900, 2.5900, 2.5901, 2.5901, 2.5901,
|
||||||
|
2.5901, 2.5901, 2.5949, 2.6964, 2.6965, 2.6965, 2.6965, 2.6965, 2.6965, 2.6965,
|
||||||
|
2.7010, 2.8328, 2.8328, 2.8328, 2.8329, 2.8329, 2.8329, 2.8329, 2.8366, 2.9405,
|
||||||
|
2.9405, 2.9405, 2.9405, 2.9405, 2.9405, 2.9405, 2.9442, 3.1344, 3.1344, 3.1346,
|
||||||
|
3.1463, 3.1463, 3.1463, 3.1463, 3.1493, 3.3076, 3.3076, 3.3076, 3.3076, 3.3076,
|
||||||
|
3.3077, 3.3077, 3.3097, 3.4456, 3.4456, 3.4456, 3.4456, 3.4456, 3.4456, 3.4456,
|
||||||
|
3.4473, 3.5236, 3.5236, 3.5236, 3.5236, 3.5236, 3.5236, 3.5236, 3.5249, 3.6004,
|
||||||
|
3.6004, 3.6004, 3.6004, 3.6004, 3.6004, 3.6004, 3.6026, 3.6684, 3.6684, 3.6684,
|
||||||
|
3.6684, 3.6684, 3.6684, 3.6684, 3.6697, 3.7121, 3.7121, 3.7121, 3.7121, 3.7121,
|
||||||
|
3.7121, 3.7121, 3.7136, 3.7744, 3.7744, 3.7744, 3.7744, 3.7744, 3.7745, 3.7745,
|
||||||
|
3.7756, 3.8151, 3.8151, 3.8151, 3.8151, 3.8151, 3.8151, 3.8151, 3.8163, 3.8673,
|
||||||
|
3.8673, 3.8673, 3.8673, 3.8673, 3.8673, 3.8673, 3.8680, 3.9044, 3.9044, 3.9044,
|
||||||
|
3.9044, 3.9044, 3.9044, 3.9044, 3.9056, 3.9297, 3.9297, 3.9297, 3.9297, 3.9297,
|
||||||
|
3.9297, 3.9297, 3.9305, 3.9494, 3.9494, 3.9494, 3.9494, 3.9494, 3.9494, 3.9494,
|
||||||
|
3.9514, 4.0725, 4.0725, 4.0725, 4.0725, 4.0725, 4.0725, 4.0725, 4.0731, 4.0990,
|
||||||
|
4.0993, 4.0993, 4.0993, 4.0993, 4.0993, 4.0993, 4.1004, 4.1385, 4.1385, 4.1385,
|
||||||
|
4.1386, 4.1386, 4.1387, 4.1387, 4.1398, 4.1732, 4.2284, 4.2284, 4.2284, 4.2284,
|
||||||
|
4.2284, 4.2284, 4.2290, 4.2781, 4.2781, 4.2963, 4.2963, 4.2963, 4.2963, 4.2963,
|
||||||
|
4.2971, 4.3141, 4.3141, 4.3141, 4.3141, 4.3141, 4.3141, 4.3141, 4.3146, 4.3393,
|
||||||
|
4.3393, 4.3393, 4.3393, 4.3393, 4.3393, 4.3393,
|
||||||
|
]
|
||||||
|
|
||||||
|
|
||||||
|
def calculate_afr(smart_attributes: dict, is_seagate: bool = False, include_193: bool = False) -> float:
|
||||||
|
"""
|
||||||
|
Calculate Annual Failure Rate (AFR) based on SMART attributes.
|
||||||
|
|
||||||
|
Uses BackBlaze dataset to predict 1-year failure probability.
|
||||||
|
Returns the maximum AFR across all relevant SMART attributes.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
smart_attributes: Dict mapping SMART attribute IDs to their values
|
||||||
|
is_seagate: Whether this is a Seagate disk (affects attr 188 handling)
|
||||||
|
include_193: Include attribute 193 (Load Cycle Count) for SnapRAID pre-v13.0 compatibility
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
AFR as a decimal (e.g., 0.81 for 81%)
|
||||||
|
"""
|
||||||
|
afr_values = []
|
||||||
|
|
||||||
|
# Helper function to get AFR from a table
|
||||||
|
def get_afr_from_table(attr_id: int, table: list, step: int) -> float:
|
||||||
|
if attr_id not in smart_attributes:
|
||||||
|
return 0.0
|
||||||
|
|
||||||
|
value = smart_attributes[attr_id]
|
||||||
|
index = min(value // step, len(table) - 1)
|
||||||
|
monthly_rate = table[index]
|
||||||
|
# Annualize: convert monthly to annual (like SnapRAID does)
|
||||||
|
return (365.0 / 30.0) * monthly_rate
|
||||||
|
|
||||||
|
# Check each attribute
|
||||||
|
afr_values.append(get_afr_from_table(5, SMART_5_R, SMART_5_STEP))
|
||||||
|
afr_values.append(get_afr_from_table(187, SMART_187_R, SMART_187_STEP))
|
||||||
|
|
||||||
|
# Exclude attribute 188 for Seagate disks (false positives on SMR/IronWolf)
|
||||||
|
if not is_seagate:
|
||||||
|
afr_values.append(get_afr_from_table(188, SMART_188_R, SMART_188_STEP))
|
||||||
|
|
||||||
|
# Attribute 193 (Load Cycle Count) - removed in SnapRAID v13.0
|
||||||
|
# Include for compatibility with older SnapRAID versions
|
||||||
|
if include_193:
|
||||||
|
afr_values.append(get_afr_from_table(193, SMART_193_R, SMART_193_STEP))
|
||||||
|
|
||||||
|
afr_values.append(get_afr_from_table(197, SMART_197_R, SMART_197_STEP))
|
||||||
|
afr_values.append(get_afr_from_table(198, SMART_198_R, SMART_198_STEP))
|
||||||
|
|
||||||
|
# Return the maximum AFR (attributes are correlated, not independent)
|
||||||
|
return max(afr_values) if afr_values else 0.0
|
||||||
|
|
||||||
|
|
||||||
|
def afr_to_failure_probability(afr: float) -> float:
|
||||||
|
"""
|
||||||
|
Convert Annual Failure Rate to failure probability using Poisson distribution.
|
||||||
|
|
||||||
|
SnapRAID uses: P(at least 1 failure) = 1 - e^(-AFR)
|
||||||
|
|
||||||
|
Args:
|
||||||
|
afr: Annual Failure Rate (e.g., 1.64 for high risk)
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Failure probability as a decimal (e.g., 0.806 for 80.6%)
|
||||||
|
"""
|
||||||
|
return 1.0 - math.exp(-afr)
|
||||||
@@ -0,0 +1,68 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Check if Scrutiny has failure_rate fields we should be using
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
from influxdb_client import InfluxDBClient
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
load_dotenv()
|
||||||
|
|
||||||
|
def check_scrutiny_failure_rates():
|
||||||
|
"""Check Scrutiny's own failure rate calculations."""
|
||||||
|
url = os.getenv('INFLUXDB_URL')
|
||||||
|
token = os.getenv('INFLUXDB_TOKEN')
|
||||||
|
org = os.getenv('INFLUXDB_ORG', 'scrutiny')
|
||||||
|
bucket = os.getenv('INFLUXDB_BUCKET', 'metrics')
|
||||||
|
|
||||||
|
client = InfluxDBClient(url=url, token=token, org=org)
|
||||||
|
query_api = client.query_api()
|
||||||
|
|
||||||
|
# Get failure_rate fields
|
||||||
|
query = f'''
|
||||||
|
from(bucket: "{bucket}")
|
||||||
|
|> range(start: -1h)
|
||||||
|
|> filter(fn: (r) => r._measurement == "smart")
|
||||||
|
|> filter(fn: (r) => r._field =~ /failure_rate/)
|
||||||
|
|> last()
|
||||||
|
'''
|
||||||
|
|
||||||
|
print("=" * 80)
|
||||||
|
print("Scrutiny's Failure Rate Calculations")
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
result = query_api.query(query)
|
||||||
|
|
||||||
|
devices = {}
|
||||||
|
for table in result:
|
||||||
|
for record in table.records:
|
||||||
|
wwn = record.values.get('device_wwn')
|
||||||
|
field = record.get_field()
|
||||||
|
value = record.get_value()
|
||||||
|
|
||||||
|
if wwn not in devices:
|
||||||
|
devices[wwn] = {}
|
||||||
|
|
||||||
|
# Extract attribute ID
|
||||||
|
parts = field.split('.')
|
||||||
|
if len(parts) >= 2:
|
||||||
|
attr_id = parts[1]
|
||||||
|
devices[wwn][attr_id] = value
|
||||||
|
|
||||||
|
# Show per device
|
||||||
|
for wwn, attrs in sorted(devices.items()):
|
||||||
|
print(f"\nDevice {wwn}:")
|
||||||
|
max_rate = 0
|
||||||
|
for attr_id, rate in sorted(attrs.items(), key=lambda x: int(x[0])):
|
||||||
|
if rate and rate > 0:
|
||||||
|
print(f" Attr {attr_id:3s}: {rate:.4f}")
|
||||||
|
max_rate = max(max_rate, rate)
|
||||||
|
|
||||||
|
if max_rate > 0:
|
||||||
|
print(f" MAX: {max_rate:.4f} ({max_rate*100:.2f}%)")
|
||||||
|
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
check_scrutiny_failure_rates()
|
||||||
Executable
+85
@@ -0,0 +1,85 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Debug script to see actual SMART attribute values from InfluxDB
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
from influxdb_client import InfluxDBClient
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
load_dotenv()
|
||||||
|
|
||||||
|
def debug_smart_values():
|
||||||
|
"""Fetch and display raw SMART values for debugging."""
|
||||||
|
url = os.getenv('INFLUXDB_URL')
|
||||||
|
token = os.getenv('INFLUXDB_TOKEN')
|
||||||
|
org = os.getenv('INFLUXDB_ORG', 'scrutiny')
|
||||||
|
bucket = os.getenv('INFLUXDB_BUCKET', 'metrics')
|
||||||
|
|
||||||
|
client = InfluxDBClient(url=url, token=token, org=org)
|
||||||
|
query_api = client.query_api()
|
||||||
|
|
||||||
|
# Get all SMART attribute fields for one device
|
||||||
|
query = f'''
|
||||||
|
from(bucket: "{bucket}")
|
||||||
|
|> range(start: -1h)
|
||||||
|
|> filter(fn: (r) => r._measurement == "smart")
|
||||||
|
|> filter(fn: (r) => r.device_wwn == "0x5000c500c570663e")
|
||||||
|
|> filter(fn: (r) =>
|
||||||
|
r._field =~ /attr\\.5\\.(raw_value|transformed_value|value)/ or
|
||||||
|
r._field =~ /attr\\.187\\.(raw_value|transformed_value|value)/ or
|
||||||
|
r._field =~ /attr\\.188\\.(raw_value|transformed_value|value)/ or
|
||||||
|
r._field =~ /attr\\.197\\.(raw_value|transformed_value|value)/ or
|
||||||
|
r._field =~ /attr\\.198\\.(raw_value|transformed_value|value)/ or
|
||||||
|
r._field =~ /attr\\.9\\.(raw_value|transformed_value|value)/
|
||||||
|
)
|
||||||
|
|> last()
|
||||||
|
'''
|
||||||
|
|
||||||
|
print("=" * 80)
|
||||||
|
print("SMART Attribute Values for /dev/sdc (ZLW0A3QJ) - Should be 81% failure")
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
result = query_api.query(query)
|
||||||
|
|
||||||
|
values = {}
|
||||||
|
for table in result:
|
||||||
|
for record in table.records:
|
||||||
|
field = record.get_field()
|
||||||
|
value = record.get_value()
|
||||||
|
values[field] = value
|
||||||
|
|
||||||
|
# Group by attribute
|
||||||
|
attrs = {}
|
||||||
|
for field, value in sorted(values.items()):
|
||||||
|
parts = field.split('.')
|
||||||
|
if len(parts) >= 3:
|
||||||
|
attr_id = parts[1]
|
||||||
|
field_type = parts[2]
|
||||||
|
|
||||||
|
if attr_id not in attrs:
|
||||||
|
attrs[attr_id] = {}
|
||||||
|
attrs[attr_id][field_type] = value
|
||||||
|
|
||||||
|
# Display
|
||||||
|
for attr_id in sorted(attrs.keys(), key=int):
|
||||||
|
print(f"\nAttribute {attr_id}:")
|
||||||
|
attr_data = attrs[attr_id]
|
||||||
|
|
||||||
|
raw = attr_data.get('raw_value', 'N/A')
|
||||||
|
transformed = attr_data.get('transformed_value', 'N/A')
|
||||||
|
value = attr_data.get('value', 'N/A')
|
||||||
|
|
||||||
|
print(f" raw_value: {raw}")
|
||||||
|
if isinstance(raw, int):
|
||||||
|
print(f" hex: 0x{raw:012x}")
|
||||||
|
print(f" low byte: {raw & 0xFF}")
|
||||||
|
print(f" low word (16-bit): {raw & 0xFFFF}")
|
||||||
|
print(f" low 32-bit: {raw & 0xFFFFFFFF}")
|
||||||
|
print(f" transformed_value: {transformed}")
|
||||||
|
print(f" value: {value}")
|
||||||
|
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
debug_smart_values()
|
||||||
Executable
+79
@@ -0,0 +1,79 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Check the VALUE field (not raw_value) for SMART attributes
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
from influxdb_client import InfluxDBClient
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
load_dotenv()
|
||||||
|
|
||||||
|
url = os.getenv('INFLUXDB_URL')
|
||||||
|
token = os.getenv('INFLUXDB_TOKEN')
|
||||||
|
org = os.getenv('INFLUXDB_ORG', 'scrutiny')
|
||||||
|
bucket = os.getenv('INFLUXDB_BUCKET', 'metrics')
|
||||||
|
|
||||||
|
client = InfluxDBClient(url=url, token=token, org=org)
|
||||||
|
query_api = client.query_api()
|
||||||
|
|
||||||
|
# Get VALUE field (normalized 0-100) for critical attributes
|
||||||
|
query = f'''
|
||||||
|
from(bucket: "{bucket}")
|
||||||
|
|> range(start: -1h)
|
||||||
|
|> filter(fn: (r) => r._measurement == "smart")
|
||||||
|
|> filter(fn: (r) => r.device_wwn == "0x5000c500c570663e")
|
||||||
|
|> filter(fn: (r) =>
|
||||||
|
r._field == "attr.5.value" or
|
||||||
|
r._field == "attr.187.value" or
|
||||||
|
r._field == "attr.188.value" or
|
||||||
|
r._field == "attr.197.value" or
|
||||||
|
r._field == "attr.198.value" or
|
||||||
|
r._field == "attr.5.raw_value" or
|
||||||
|
r._field == "attr.187.raw_value" or
|
||||||
|
r._field == "attr.188.raw_value" or
|
||||||
|
r._field == "attr.197.raw_value" or
|
||||||
|
r._field == "attr.198.raw_value"
|
||||||
|
)
|
||||||
|
|> last()
|
||||||
|
'''
|
||||||
|
|
||||||
|
print("=" * 80)
|
||||||
|
print("/dev/sdc (ZLW0A3QJ) - should show 81% failure")
|
||||||
|
print("Comparing InfluxDB values to smartctl output")
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
result = query_api.query(query)
|
||||||
|
|
||||||
|
attrs = {}
|
||||||
|
for table in result:
|
||||||
|
for record in table.records:
|
||||||
|
field = record.get_field()
|
||||||
|
value = record.get_value()
|
||||||
|
|
||||||
|
parts = field.split('.')
|
||||||
|
if len(parts) >= 3:
|
||||||
|
attr_id = parts[1]
|
||||||
|
field_type = parts[2]
|
||||||
|
|
||||||
|
if attr_id not in attrs:
|
||||||
|
attrs[attr_id] = {}
|
||||||
|
attrs[attr_id][field_type] = value
|
||||||
|
|
||||||
|
print("\nInfluxDB Data:")
|
||||||
|
print("-" * 80)
|
||||||
|
for attr_id in sorted(attrs.keys(), key=int):
|
||||||
|
print(f"\nAttribute {attr_id}:")
|
||||||
|
print(f" value (normalized): {attrs[attr_id].get('value', 'N/A')}")
|
||||||
|
print(f" raw_value: {attrs[attr_id].get('raw_value', 'N/A')}")
|
||||||
|
|
||||||
|
print("\n" + "=" * 80)
|
||||||
|
print("From smartctl -A /dev/sdc:")
|
||||||
|
print("-" * 80)
|
||||||
|
print(" 5 Reallocated_Sector_Ct: VALUE=100, RAW_VALUE=0")
|
||||||
|
print("187 Reported_Uncorrect: VALUE=100, RAW_VALUE=0")
|
||||||
|
print("188 Command_Timeout: VALUE=100, RAW_VALUE=0")
|
||||||
|
print("197 Current_Pending_Sector: VALUE=100, RAW_VALUE=0")
|
||||||
|
print("198 Offline_Uncorrectable: VALUE=100, RAW_VALUE=0")
|
||||||
|
|
||||||
|
client.close()
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
0x5000c500744487c5 = /dev/sda [parity]
|
||||||
|
0x5000c500c455c45e = /dev/sdf
|
||||||
|
0x5000c500c46b0832 = /dev/sdi
|
||||||
|
0x5000c500c570663e = /dev/sdc [d1]
|
||||||
|
0x5000c500e584b9c8 = /dev/sdd [d5]
|
||||||
|
0x5000c500e7a6a520 = /dev/sdb [d2]
|
||||||
|
0x5000cca234c69287 = /dev/sde
|
||||||
|
0x5000cca266e83d77 = /dev/sdh [2-parity]
|
||||||
|
0x5000cca26ae062f3 = /dev/sdg [2-parity]
|
||||||
|
25243x801438 = /dev/nvme0n1 [os]
|
||||||
Executable
+112
@@ -0,0 +1,112 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Explore Scrutiny's InfluxDB schema to understand how SMART data is stored.
|
||||||
|
Run this first to understand the data structure before running the main logger.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
from influxdb_client import InfluxDBClient
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
load_dotenv()
|
||||||
|
|
||||||
|
def explore_influxdb():
|
||||||
|
"""Connect to InfluxDB and explore the schema."""
|
||||||
|
url = os.getenv('INFLUXDB_URL')
|
||||||
|
token = os.getenv('INFLUXDB_TOKEN')
|
||||||
|
org = os.getenv('INFLUXDB_ORG', 'scrutiny')
|
||||||
|
bucket = os.getenv('INFLUXDB_BUCKET', 'metrics')
|
||||||
|
|
||||||
|
print(f"Connecting to InfluxDB at {url}")
|
||||||
|
print(f"Organization: {org}")
|
||||||
|
print(f"Bucket: {bucket}\n")
|
||||||
|
|
||||||
|
client = InfluxDBClient(url=url, token=token, org=org)
|
||||||
|
query_api = client.query_api()
|
||||||
|
|
||||||
|
# 1. List all measurements in the bucket
|
||||||
|
print("=" * 80)
|
||||||
|
print("STEP 1: Discovering measurements in bucket")
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
query_measurements = f'''
|
||||||
|
import "influxdata/influxdb/schema"
|
||||||
|
schema.measurements(bucket: "{bucket}")
|
||||||
|
'''
|
||||||
|
|
||||||
|
try:
|
||||||
|
result = query_api.query(query_measurements)
|
||||||
|
measurements = []
|
||||||
|
for table in result:
|
||||||
|
for record in table.records:
|
||||||
|
measurement = record.get_value()
|
||||||
|
measurements.append(measurement)
|
||||||
|
print(f" - {measurement}")
|
||||||
|
|
||||||
|
if not measurements:
|
||||||
|
print(" No measurements found!")
|
||||||
|
return
|
||||||
|
|
||||||
|
print(f"\nFound {len(measurements)} measurement(s)\n")
|
||||||
|
|
||||||
|
# 2. For each measurement, show field keys
|
||||||
|
print("=" * 80)
|
||||||
|
print("STEP 2: Discovering fields for each measurement")
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
for measurement in measurements[:5]: # Limit to first 5 measurements
|
||||||
|
print(f"\nMeasurement: {measurement}")
|
||||||
|
query_fields = f'''
|
||||||
|
import "influxdata/influxdb/schema"
|
||||||
|
schema.measurementFieldKeys(
|
||||||
|
bucket: "{bucket}",
|
||||||
|
measurement: "{measurement}"
|
||||||
|
)
|
||||||
|
'''
|
||||||
|
|
||||||
|
result = query_api.query(query_fields)
|
||||||
|
for table in result:
|
||||||
|
for record in table.records:
|
||||||
|
field = record.get_value()
|
||||||
|
print(f" Field: {field}")
|
||||||
|
|
||||||
|
# 3. Show sample data from the first measurement
|
||||||
|
print("\n" + "=" * 80)
|
||||||
|
print("STEP 3: Sample data from first measurement")
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
first_measurement = measurements[0]
|
||||||
|
query_sample = f'''
|
||||||
|
from(bucket: "{bucket}")
|
||||||
|
|> range(start: -24h)
|
||||||
|
|> filter(fn: (r) => r._measurement == "{first_measurement}")
|
||||||
|
|> limit(n: 10)
|
||||||
|
'''
|
||||||
|
|
||||||
|
print(f"\nSample records from '{first_measurement}':\n")
|
||||||
|
result = query_api.query(query_sample)
|
||||||
|
for table in result:
|
||||||
|
for record in table.records:
|
||||||
|
print(f"Time: {record.get_time()}")
|
||||||
|
print(f" Measurement: {record.get_measurement()}")
|
||||||
|
print(f" Field: {record.get_field()}")
|
||||||
|
print(f" Value: {record.get_value()}")
|
||||||
|
print(f" Tags: {record.values}")
|
||||||
|
print()
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"Error querying InfluxDB: {e}")
|
||||||
|
import traceback
|
||||||
|
traceback.print_exc()
|
||||||
|
|
||||||
|
finally:
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
if not os.getenv('INFLUXDB_URL'):
|
||||||
|
print("ERROR: Please create a .env file with your InfluxDB configuration")
|
||||||
|
print("Copy .env.example to .env and fill in your details")
|
||||||
|
exit(1)
|
||||||
|
|
||||||
|
explore_influxdb()
|
||||||
Executable
+89
@@ -0,0 +1,89 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
Helper script to extract WWN to device mappings from InfluxDB.
|
||||||
|
Run this to help populate the devices table.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
from influxdb_client import InfluxDBClient
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
load_dotenv()
|
||||||
|
|
||||||
|
def get_device_info():
|
||||||
|
"""Extract unique device WWNs and their latest data from InfluxDB."""
|
||||||
|
url = os.getenv('INFLUXDB_URL')
|
||||||
|
token = os.getenv('INFLUXDB_TOKEN')
|
||||||
|
org = os.getenv('INFLUXDB_ORG', 'scrutiny')
|
||||||
|
bucket = os.getenv('INFLUXDB_BUCKET', 'metrics')
|
||||||
|
|
||||||
|
print(f"Connecting to InfluxDB at {url}...")
|
||||||
|
client = InfluxDBClient(url=url, token=token, org=org)
|
||||||
|
query_api = client.query_api()
|
||||||
|
|
||||||
|
# Get unique device WWNs with their latest data
|
||||||
|
query = f'''
|
||||||
|
from(bucket: "{bucket}")
|
||||||
|
|> range(start: -24h)
|
||||||
|
|> filter(fn: (r) => r._measurement == "smart")
|
||||||
|
|> filter(fn: (r) =>
|
||||||
|
r._field == "temp" or
|
||||||
|
r._field == "power_on_hours"
|
||||||
|
)
|
||||||
|
|> last()
|
||||||
|
|> group(columns: ["device_wwn", "device_protocol"])
|
||||||
|
'''
|
||||||
|
|
||||||
|
print("\nDiscovered devices:\n")
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
result = query_api.query(query)
|
||||||
|
devices = {}
|
||||||
|
|
||||||
|
for table in result:
|
||||||
|
for record in table.records:
|
||||||
|
wwn = record.values.get('device_wwn')
|
||||||
|
protocol = record.values.get('device_protocol')
|
||||||
|
|
||||||
|
if wwn not in devices:
|
||||||
|
devices[wwn] = {
|
||||||
|
'wwn': wwn,
|
||||||
|
'protocol': protocol,
|
||||||
|
'temp': None,
|
||||||
|
'power_on_hours': None
|
||||||
|
}
|
||||||
|
|
||||||
|
field = record.get_field()
|
||||||
|
if field == 'temp':
|
||||||
|
devices[wwn]['temp'] = record.get_value()
|
||||||
|
elif field == 'power_on_hours':
|
||||||
|
devices[wwn]['power_on_hours'] = record.get_value()
|
||||||
|
|
||||||
|
# Print devices
|
||||||
|
for i, (wwn, info) in enumerate(sorted(devices.items()), 1):
|
||||||
|
power_on_days = int(info['power_on_hours']) // 24 if info['power_on_hours'] else 0
|
||||||
|
|
||||||
|
print(f"Device {i}:")
|
||||||
|
print(f" WWN: {wwn}")
|
||||||
|
print(f" Protocol: {info['protocol']}")
|
||||||
|
print(f" Temperature: {info['temp']}°C")
|
||||||
|
print(f" Power-On Time: {power_on_days} days ({info['power_on_hours']} hours)")
|
||||||
|
print()
|
||||||
|
|
||||||
|
print("=" * 80)
|
||||||
|
print("\nNow run 'snapraid smart' or 'smartctl -a /dev/sdX' on each device to get:")
|
||||||
|
print(" - Serial number")
|
||||||
|
print(" - Model name")
|
||||||
|
print(" - Capacity")
|
||||||
|
print(" - Device path (/dev/sdX)")
|
||||||
|
print("\nThen update populate_devices.sql with this information.")
|
||||||
|
|
||||||
|
client.close()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
if not os.getenv('INFLUXDB_URL'):
|
||||||
|
print("ERROR: Please create a .env file with your InfluxDB configuration")
|
||||||
|
exit(1)
|
||||||
|
|
||||||
|
get_device_info()
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
-- Populate devices table with your drive metadata
|
||||||
|
-- Replace the placeholder values with actual information from your drives
|
||||||
|
|
||||||
|
-- Device 1: WWN 0x5000c500744487c5
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000c500744487c5',
|
||||||
|
'/dev/sdc', -- Update with actual device path
|
||||||
|
'ZLW0A3QJ', -- Update with actual serial
|
||||||
|
'WDC WD120EFAX', -- Update with actual model
|
||||||
|
'Western Digital', -- Update with manufacturer
|
||||||
|
12000000000000, -- Update with capacity in bytes
|
||||||
|
12.0, -- Size in TB
|
||||||
|
'd1', -- SnapRAID role (d1, d2, parity, 2-parity, or '-' for unused)
|
||||||
|
'Data drive 1' -- Optional notes
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Device 2: WWN 0x5000c500c455c45e
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000c500c455c45e',
|
||||||
|
'/dev/sdb',
|
||||||
|
'ZX22EJ3Y',
|
||||||
|
'WDC WD200EFAX',
|
||||||
|
'Western Digital',
|
||||||
|
20000000000000,
|
||||||
|
20.0,
|
||||||
|
'd2',
|
||||||
|
'Data drive 2'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Device 3: WWN 0x5000c500c46b0832
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000c500c46b0832',
|
||||||
|
'/dev/sdd',
|
||||||
|
'ZTN1BNEZ',
|
||||||
|
'WDC WD120EFAX',
|
||||||
|
'Western Digital',
|
||||||
|
12000000000000,
|
||||||
|
12.0,
|
||||||
|
'd5',
|
||||||
|
'Data drive 5'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Device 4: WWN 0x5000c500c570663e
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000c500c570663e',
|
||||||
|
'/dev/sda',
|
||||||
|
'ZX20AAMV',
|
||||||
|
'WDC WD200EFAX',
|
||||||
|
'Western Digital',
|
||||||
|
20000000000000,
|
||||||
|
20.0,
|
||||||
|
'parity',
|
||||||
|
'Parity drive'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Device 5: WWN 0x5000c500e584b9c8
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000c500e584b9c8',
|
||||||
|
'/dev/sdg',
|
||||||
|
'2TJ97M6D',
|
||||||
|
'WDC WD100EFAX',
|
||||||
|
'Western Digital',
|
||||||
|
10000000000000,
|
||||||
|
10.0,
|
||||||
|
'2-parity',
|
||||||
|
'Second parity drive'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Add remaining 4 devices following the same pattern
|
||||||
|
-- Template:
|
||||||
|
-- INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
-- VALUES (
|
||||||
|
-- '0xXXXXXXXXXXXXXXXX', -- WWN from InfluxDB
|
||||||
|
-- '/dev/sdX', -- Device path
|
||||||
|
-- 'SERIAL', -- Serial number from snapraid smart or smartctl
|
||||||
|
-- 'MODEL', -- Model number
|
||||||
|
-- 'MANUFACTURER', -- Manufacturer name
|
||||||
|
-- 0, -- Capacity in bytes
|
||||||
|
-- 0.0, -- Size in TB
|
||||||
|
-- '-', -- SnapRAID role
|
||||||
|
-- '' -- Notes
|
||||||
|
-- );
|
||||||
@@ -0,0 +1,149 @@
|
|||||||
|
-- Populate devices table with actual drive data
|
||||||
|
-- Based on device_mappings.txt and snapraid smart output
|
||||||
|
|
||||||
|
-- Device 1: /dev/sda [parity]
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000c500744487c5',
|
||||||
|
'/dev/sda',
|
||||||
|
'ZX20AAMV',
|
||||||
|
'WDC WD200EFAX',
|
||||||
|
'Western Digital',
|
||||||
|
20000000000000,
|
||||||
|
20.0,
|
||||||
|
'parity',
|
||||||
|
'Parity drive - 636 days old'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Device 2: /dev/sdb [d2]
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000c500e7a6a520',
|
||||||
|
'/dev/sdb',
|
||||||
|
'ZX22EJ3Y',
|
||||||
|
'WDC WD200EFAX',
|
||||||
|
'Western Digital',
|
||||||
|
20000000000000,
|
||||||
|
20.0,
|
||||||
|
'd2',
|
||||||
|
'Data drive 2 - 637 days old'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Device 3: /dev/sdc [d1]
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000c500c570663e',
|
||||||
|
'/dev/sdc',
|
||||||
|
'ZLW0A3QJ',
|
||||||
|
'WDC WD120EFAX',
|
||||||
|
'Western Digital',
|
||||||
|
12000000000000,
|
||||||
|
12.0,
|
||||||
|
'd1',
|
||||||
|
'Data drive 1 - 1968 days old - HIGH RISK 81%'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Device 4: /dev/sdd [d5]
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000c500e584b9c8',
|
||||||
|
'/dev/sdd',
|
||||||
|
'ZTN1BNEZ',
|
||||||
|
'WDC WD120EFAX',
|
||||||
|
'Western Digital',
|
||||||
|
12000000000000,
|
||||||
|
12.0,
|
||||||
|
'd5',
|
||||||
|
'Data drive 5 - 1014 days old'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Device 5: /dev/sde (unused)
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000cca234c69287',
|
||||||
|
'/dev/sde',
|
||||||
|
'YVGGG6EC',
|
||||||
|
'WDC WD30EFRX',
|
||||||
|
'Western Digital',
|
||||||
|
3000000000000,
|
||||||
|
3.0,
|
||||||
|
'-',
|
||||||
|
'Unused drive - 1983 days old'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Device 6: /dev/sdf (unused)
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000c500c455c45e',
|
||||||
|
'/dev/sdf',
|
||||||
|
'ZA1GX9F4',
|
||||||
|
'WDC WD80EFAX',
|
||||||
|
'Western Digital',
|
||||||
|
8000000000000,
|
||||||
|
8.0,
|
||||||
|
'-',
|
||||||
|
'Unused drive - 2072 days old - HIGH RISK 84%'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Device 7: /dev/sdg [2-parity]
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000cca26ae062f3',
|
||||||
|
'/dev/sdg',
|
||||||
|
'2TJ97M6D',
|
||||||
|
'WDC WD100EFAX',
|
||||||
|
'Western Digital',
|
||||||
|
10000000000000,
|
||||||
|
10.0,
|
||||||
|
'2-parity',
|
||||||
|
'Second parity drive - 2540 days old'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Device 8: /dev/sdh [2-parity]
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000cca266e83d77',
|
||||||
|
'/dev/sdh',
|
||||||
|
'7JJVJ65C',
|
||||||
|
'WDC WD100EFAX',
|
||||||
|
'Western Digital',
|
||||||
|
10000000000000,
|
||||||
|
10.0,
|
||||||
|
'2-parity',
|
||||||
|
'Second parity drive - 2354 days old - 1 error reported'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Device 9: /dev/sdi (unused)
|
||||||
|
INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
VALUES (
|
||||||
|
'0x5000c500c46b0832',
|
||||||
|
'/dev/sdi',
|
||||||
|
'ZA1GW0XM',
|
||||||
|
'WDC WD80EFAX',
|
||||||
|
'Western Digital',
|
||||||
|
8000000000000,
|
||||||
|
8.0,
|
||||||
|
'-',
|
||||||
|
'Unused drive - 2074 days old - HIGH RISK 84%'
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Device 10: /dev/nvme0n1 [OS drive] - Note: WWN format different for NVMe
|
||||||
|
-- NVMe devices may not report WWN in the same format, skipping for now
|
||||||
|
-- You can add this manually if needed after checking actual WWN from InfluxDB
|
||||||
|
-- INSERT INTO devices (device_wwn, device_path, serial_number, model, manufacturer, capacity_bytes, size_tb, disk_role, notes)
|
||||||
|
-- VALUES (
|
||||||
|
-- '25243x801438', -- May need different format
|
||||||
|
-- '/dev/nvme0n1',
|
||||||
|
-- '25243X801438',
|
||||||
|
-- 'NVMe SSD',
|
||||||
|
-- 'Unknown',
|
||||||
|
-- NULL,
|
||||||
|
-- NULL,
|
||||||
|
-- 'os',
|
||||||
|
-- 'Operating system drive'
|
||||||
|
-- );
|
||||||
|
|
||||||
|
-- Verify the data
|
||||||
|
SELECT device_path, serial_number, size_tb, disk_role, notes
|
||||||
|
FROM devices
|
||||||
|
ORDER BY device_path;
|
||||||
@@ -0,0 +1,4 @@
|
|||||||
|
influxdb-client>=1.38.0
|
||||||
|
psycopg[binary]>=3.1.0
|
||||||
|
python-dotenv>=1.0.0
|
||||||
|
requests>=2.31.0
|
||||||
+58
@@ -0,0 +1,58 @@
|
|||||||
|
-- PostgreSQL schema for SMART data logging
|
||||||
|
|
||||||
|
-- Main table for SMART metrics snapshots
|
||||||
|
CREATE TABLE IF NOT EXISTS smart_metrics (
|
||||||
|
id BIGSERIAL PRIMARY KEY,
|
||||||
|
timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||||
|
device_path VARCHAR(255) NOT NULL,
|
||||||
|
serial_number VARCHAR(255) NOT NULL,
|
||||||
|
model VARCHAR(255),
|
||||||
|
disk_role VARCHAR(50), -- e.g., 'd1', 'd2', 'parity', '2-parity', or '-' for unused
|
||||||
|
|
||||||
|
-- Basic metrics (from SnapRAID smart output)
|
||||||
|
temperature_celsius INTEGER,
|
||||||
|
power_on_days INTEGER,
|
||||||
|
error_count INTEGER,
|
||||||
|
size_tb DECIMAL(10, 2),
|
||||||
|
|
||||||
|
-- Calculated failure probability
|
||||||
|
failure_probability_pct DECIMAL(5, 2), -- 0.00 to 100.00
|
||||||
|
|
||||||
|
-- Raw SMART attributes (key-value pairs as JSONB)
|
||||||
|
smart_attributes JSONB,
|
||||||
|
|
||||||
|
-- Indexes for efficient querying
|
||||||
|
CONSTRAINT unique_device_timestamp UNIQUE (device_path, timestamp)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Index on serial number for device history queries
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_smart_metrics_serial ON smart_metrics(serial_number);
|
||||||
|
|
||||||
|
-- Index on timestamp for time-range queries
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_smart_metrics_timestamp ON smart_metrics(timestamp DESC);
|
||||||
|
|
||||||
|
-- Index on device_path for per-device queries
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_smart_metrics_device ON smart_metrics(device_path);
|
||||||
|
|
||||||
|
-- Index on JSONB attributes for querying specific SMART values
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_smart_attributes_gin ON smart_metrics USING gin (smart_attributes);
|
||||||
|
|
||||||
|
-- View for latest metrics per device
|
||||||
|
CREATE OR REPLACE VIEW smart_latest AS
|
||||||
|
SELECT DISTINCT ON (device_path)
|
||||||
|
*
|
||||||
|
FROM smart_metrics
|
||||||
|
ORDER BY device_path, timestamp DESC;
|
||||||
|
|
||||||
|
-- View for high-risk devices (failure probability > 50%)
|
||||||
|
CREATE OR REPLACE VIEW smart_high_risk AS
|
||||||
|
SELECT *
|
||||||
|
FROM smart_latest
|
||||||
|
WHERE failure_probability_pct > 50.0
|
||||||
|
ORDER BY failure_probability_pct DESC;
|
||||||
|
|
||||||
|
COMMENT ON TABLE smart_metrics IS 'Historical SMART metrics collected from Scrutiny InfluxDB';
|
||||||
|
COMMENT ON COLUMN smart_metrics.failure_probability_pct IS 'BackBlaze-based 1-year failure probability (0-100%)';
|
||||||
|
COMMENT ON COLUMN smart_metrics.smart_attributes IS 'Raw SMART attributes as JSON (attribute_id -> value)';
|
||||||
|
COMMENT ON VIEW smart_latest IS 'Latest SMART metrics for each device';
|
||||||
|
COMMENT ON VIEW smart_high_risk IS 'Devices with >50% annual failure probability';
|
||||||
@@ -0,0 +1,72 @@
|
|||||||
|
-- Migration: Add v13.0 failure probability column
|
||||||
|
-- Adds a second failure probability column for the modern algorithm (without attr 193)
|
||||||
|
|
||||||
|
-- Add new column for v13.0+ algorithm (without attribute 193)
|
||||||
|
ALTER TABLE smart_metrics
|
||||||
|
ADD COLUMN IF NOT EXISTS failure_probability_v13_pct DECIMAL(5, 2);
|
||||||
|
|
||||||
|
-- Rename existing column to clarify it includes attribute 193 (pre-v13.0)
|
||||||
|
ALTER TABLE smart_metrics
|
||||||
|
RENAME COLUMN failure_probability_pct TO failure_probability_v12_pct;
|
||||||
|
|
||||||
|
-- Update views to include both columns
|
||||||
|
DROP VIEW IF EXISTS smart_latest CASCADE;
|
||||||
|
CREATE VIEW smart_latest AS
|
||||||
|
SELECT
|
||||||
|
m.id,
|
||||||
|
m.timestamp,
|
||||||
|
m.device_wwn,
|
||||||
|
d.device_path,
|
||||||
|
d.serial_number,
|
||||||
|
d.model,
|
||||||
|
d.manufacturer,
|
||||||
|
d.capacity_bytes,
|
||||||
|
d.size_tb,
|
||||||
|
d.disk_role,
|
||||||
|
m.temperature_celsius,
|
||||||
|
m.power_on_days,
|
||||||
|
m.error_count,
|
||||||
|
m.failure_probability_v12_pct, -- With attr 193 (matches your current SnapRAID)
|
||||||
|
m.failure_probability_v13_pct, -- Without attr 193 (modern SnapRAID v13.0+)
|
||||||
|
m.smart_attributes
|
||||||
|
FROM smart_metrics m
|
||||||
|
INNER JOIN devices d ON m.device_wwn = d.device_wwn
|
||||||
|
WHERE m.timestamp = (
|
||||||
|
SELECT MAX(m2.timestamp)
|
||||||
|
FROM smart_metrics m2
|
||||||
|
WHERE m2.device_wwn = m.device_wwn
|
||||||
|
);
|
||||||
|
|
||||||
|
DROP VIEW IF EXISTS smart_high_risk;
|
||||||
|
CREATE VIEW smart_high_risk AS
|
||||||
|
SELECT *
|
||||||
|
FROM smart_latest
|
||||||
|
WHERE failure_probability_v12_pct > 50 -- Using v12 since that matches your SnapRAID
|
||||||
|
ORDER BY failure_probability_v12_pct DESC;
|
||||||
|
|
||||||
|
DROP VIEW IF EXISTS smart_summary;
|
||||||
|
CREATE VIEW smart_summary AS
|
||||||
|
SELECT
|
||||||
|
device_path,
|
||||||
|
serial_number,
|
||||||
|
temperature_celsius,
|
||||||
|
power_on_days,
|
||||||
|
error_count,
|
||||||
|
failure_probability_v12_pct, -- With attr 193
|
||||||
|
failure_probability_v13_pct, -- Without attr 193
|
||||||
|
size_tb,
|
||||||
|
disk_role
|
||||||
|
FROM smart_latest
|
||||||
|
ORDER BY
|
||||||
|
CASE disk_role
|
||||||
|
WHEN 'parity' THEN 1
|
||||||
|
WHEN '2-parity' THEN 2
|
||||||
|
ELSE 3
|
||||||
|
END,
|
||||||
|
device_path;
|
||||||
|
|
||||||
|
COMMENT ON COLUMN smart_metrics.failure_probability_v12_pct IS
|
||||||
|
'Failure probability using SnapRAID <v13.0 algorithm (includes attribute 193 Load Cycle Count)';
|
||||||
|
|
||||||
|
COMMENT ON COLUMN smart_metrics.failure_probability_v13_pct IS
|
||||||
|
'Failure probability using SnapRAID v13.0+ algorithm (excludes attribute 193)';
|
||||||
+106
@@ -0,0 +1,106 @@
|
|||||||
|
-- PostgreSQL schema for SMART data logging (Version 2)
|
||||||
|
-- Now includes a devices metadata table
|
||||||
|
|
||||||
|
-- Device metadata table (populated manually)
|
||||||
|
CREATE TABLE IF NOT EXISTS devices (
|
||||||
|
device_wwn VARCHAR(50) PRIMARY KEY, -- e.g., '0x5000c500744487c5'
|
||||||
|
device_path VARCHAR(255) NOT NULL, -- e.g., '/dev/sda'
|
||||||
|
serial_number VARCHAR(255) NOT NULL,
|
||||||
|
model VARCHAR(255),
|
||||||
|
manufacturer VARCHAR(100),
|
||||||
|
capacity_bytes BIGINT,
|
||||||
|
size_tb DECIMAL(10, 2),
|
||||||
|
disk_role VARCHAR(50), -- e.g., 'd1', 'd2', 'parity', '2-parity', or '-' for unused
|
||||||
|
notes TEXT,
|
||||||
|
created_at TIMESTAMPTZ DEFAULT NOW(),
|
||||||
|
updated_at TIMESTAMPTZ DEFAULT NOW()
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Main table for SMART metrics snapshots
|
||||||
|
CREATE TABLE IF NOT EXISTS smart_metrics (
|
||||||
|
id BIGSERIAL PRIMARY KEY,
|
||||||
|
timestamp TIMESTAMPTZ NOT NULL DEFAULT NOW(),
|
||||||
|
device_wwn VARCHAR(50) NOT NULL, -- Foreign key to devices table
|
||||||
|
|
||||||
|
-- Basic metrics (from SnapRAID smart output)
|
||||||
|
temperature_celsius INTEGER,
|
||||||
|
power_on_days INTEGER,
|
||||||
|
error_count INTEGER,
|
||||||
|
|
||||||
|
-- Calculated failure probability
|
||||||
|
failure_probability_pct DECIMAL(5, 2), -- 0.00 to 100.00
|
||||||
|
|
||||||
|
-- Raw SMART attributes (key-value pairs as JSONB)
|
||||||
|
smart_attributes JSONB,
|
||||||
|
|
||||||
|
-- Foreign key constraint
|
||||||
|
CONSTRAINT fk_device FOREIGN KEY (device_wwn) REFERENCES devices(device_wwn),
|
||||||
|
|
||||||
|
-- Unique constraint
|
||||||
|
CONSTRAINT unique_device_timestamp UNIQUE (device_wwn, timestamp)
|
||||||
|
);
|
||||||
|
|
||||||
|
-- Indexes for efficient querying
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_smart_metrics_device_wwn ON smart_metrics(device_wwn);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_smart_metrics_timestamp ON smart_metrics(timestamp DESC);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_smart_attributes_gin ON smart_metrics USING gin (smart_attributes);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_devices_serial ON devices(serial_number);
|
||||||
|
CREATE INDEX IF NOT EXISTS idx_devices_path ON devices(device_path);
|
||||||
|
|
||||||
|
-- View for latest metrics per device with metadata
|
||||||
|
CREATE OR REPLACE VIEW smart_latest AS
|
||||||
|
SELECT DISTINCT ON (d.device_wwn)
|
||||||
|
d.device_wwn,
|
||||||
|
d.device_path,
|
||||||
|
d.serial_number,
|
||||||
|
d.model,
|
||||||
|
d.manufacturer,
|
||||||
|
d.size_tb,
|
||||||
|
d.disk_role,
|
||||||
|
sm.timestamp,
|
||||||
|
sm.temperature_celsius,
|
||||||
|
sm.power_on_days,
|
||||||
|
sm.error_count,
|
||||||
|
sm.failure_probability_pct,
|
||||||
|
sm.smart_attributes
|
||||||
|
FROM devices d
|
||||||
|
LEFT JOIN smart_metrics sm ON d.device_wwn = sm.device_wwn
|
||||||
|
ORDER BY d.device_wwn, sm.timestamp DESC NULLS LAST;
|
||||||
|
|
||||||
|
-- View for high-risk devices (failure probability > 50%)
|
||||||
|
CREATE OR REPLACE VIEW smart_high_risk AS
|
||||||
|
SELECT *
|
||||||
|
FROM smart_latest
|
||||||
|
WHERE failure_probability_pct > 50.0
|
||||||
|
ORDER BY failure_probability_pct DESC;
|
||||||
|
|
||||||
|
-- View for summary (like snapraid smart output)
|
||||||
|
CREATE OR REPLACE VIEW smart_summary AS
|
||||||
|
SELECT
|
||||||
|
device_path,
|
||||||
|
serial_number,
|
||||||
|
temperature_celsius as temp_c,
|
||||||
|
power_on_days,
|
||||||
|
error_count,
|
||||||
|
failure_probability_pct as fp_pct,
|
||||||
|
size_tb,
|
||||||
|
disk_role,
|
||||||
|
model,
|
||||||
|
timestamp as last_updated
|
||||||
|
FROM smart_latest
|
||||||
|
ORDER BY
|
||||||
|
CASE disk_role
|
||||||
|
WHEN 'parity' THEN 1
|
||||||
|
WHEN '2-parity' THEN 2
|
||||||
|
ELSE 3
|
||||||
|
END,
|
||||||
|
device_path;
|
||||||
|
|
||||||
|
-- Comments
|
||||||
|
COMMENT ON TABLE devices IS 'Device metadata (WWN, serial, model, etc.) - manually populated';
|
||||||
|
COMMENT ON TABLE smart_metrics IS 'Historical SMART metrics collected from Scrutiny InfluxDB';
|
||||||
|
COMMENT ON COLUMN smart_metrics.failure_probability_pct IS 'BackBlaze-based 1-year failure probability (0-100%)';
|
||||||
|
COMMENT ON COLUMN smart_metrics.smart_attributes IS 'Raw SMART attributes as JSON (attribute_id -> value)';
|
||||||
|
COMMENT ON VIEW smart_latest IS 'Latest SMART metrics for each device with metadata';
|
||||||
|
COMMENT ON VIEW smart_high_risk IS 'Devices with >50% annual failure probability';
|
||||||
|
COMMENT ON VIEW smart_summary IS 'Summary view matching snapraid smart output format';
|
||||||
Executable
+87
@@ -0,0 +1,87 @@
|
|||||||
|
#!/bin/bash
|
||||||
|
# Setup script for PostgreSQL database and user
|
||||||
|
|
||||||
|
set -e # Exit on error
|
||||||
|
|
||||||
|
# Configuration
|
||||||
|
PG_HOST="${1:-localhost}"
|
||||||
|
PG_ADMIN_USER="postgres"
|
||||||
|
PG_ADMIN_PASS="Lkxdjfhl1d"
|
||||||
|
|
||||||
|
DB_NAME="smart_monitoring"
|
||||||
|
DB_USER="smart_logger"
|
||||||
|
DB_PASS="${2:-change_me_123}" # Pass as second argument or use default
|
||||||
|
|
||||||
|
echo "PostgreSQL Setup for Smart Logger"
|
||||||
|
echo "=================================="
|
||||||
|
echo "Host: $PG_HOST"
|
||||||
|
echo "Database: $DB_NAME"
|
||||||
|
echo "User: $DB_USER"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
# Check if psql is installed
|
||||||
|
if ! command -v psql &> /dev/null; then
|
||||||
|
echo "ERROR: psql is not installed"
|
||||||
|
echo "Install with: sudo pacman -S postgresql"
|
||||||
|
exit 1
|
||||||
|
fi
|
||||||
|
|
||||||
|
echo "Step 1: Creating database..."
|
||||||
|
PGPASSWORD="$PG_ADMIN_PASS" psql -h "$PG_HOST" -U "$PG_ADMIN_USER" -d postgres <<EOF
|
||||||
|
-- Create database if it doesn't exist
|
||||||
|
SELECT 'CREATE DATABASE $DB_NAME'
|
||||||
|
WHERE NOT EXISTS (SELECT FROM pg_database WHERE datname = '$DB_NAME')\gexec
|
||||||
|
|
||||||
|
-- Create user if doesn't exist
|
||||||
|
DO \$\$
|
||||||
|
BEGIN
|
||||||
|
IF NOT EXISTS (SELECT FROM pg_user WHERE usename = '$DB_USER') THEN
|
||||||
|
CREATE USER $DB_USER WITH PASSWORD '$DB_PASS';
|
||||||
|
END IF;
|
||||||
|
END
|
||||||
|
\$\$;
|
||||||
|
|
||||||
|
-- Grant privileges
|
||||||
|
GRANT ALL PRIVILEGES ON DATABASE $DB_NAME TO $DB_USER;
|
||||||
|
EOF
|
||||||
|
|
||||||
|
echo "✓ Database and user created"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo "Step 2: Creating schema..."
|
||||||
|
PGPASSWORD="$PG_ADMIN_PASS" psql -h "$PG_HOST" -U "$PG_ADMIN_USER" -d "$DB_NAME" -f schema_v2.sql
|
||||||
|
|
||||||
|
echo "✓ Schema created"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo "Step 3: Granting permissions to $DB_USER..."
|
||||||
|
PGPASSWORD="$PG_ADMIN_PASS" psql -h "$PG_HOST" -U "$PG_ADMIN_USER" -d "$DB_NAME" <<EOF
|
||||||
|
-- Grant schema usage
|
||||||
|
GRANT USAGE ON SCHEMA public TO $DB_USER;
|
||||||
|
|
||||||
|
-- Grant table permissions
|
||||||
|
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO $DB_USER;
|
||||||
|
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO $DB_USER;
|
||||||
|
|
||||||
|
-- Grant default privileges for future objects
|
||||||
|
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON TABLES TO $DB_USER;
|
||||||
|
ALTER DEFAULT PRIVILEGES IN SCHEMA public GRANT ALL ON SEQUENCES TO $DB_USER;
|
||||||
|
EOF
|
||||||
|
|
||||||
|
echo "✓ Permissions granted"
|
||||||
|
echo ""
|
||||||
|
|
||||||
|
echo "=========================================="
|
||||||
|
echo "Setup Complete!"
|
||||||
|
echo "=========================================="
|
||||||
|
echo ""
|
||||||
|
echo "Connection details for your .env file:"
|
||||||
|
echo ""
|
||||||
|
echo "POSTGRES_HOST=$PG_HOST"
|
||||||
|
echo "POSTGRES_PORT=5432"
|
||||||
|
echo "POSTGRES_DB=$DB_NAME"
|
||||||
|
echo "POSTGRES_USER=$DB_USER"
|
||||||
|
echo "POSTGRES_PASSWORD=$DB_PASS"
|
||||||
|
echo ""
|
||||||
|
echo "Test connection with:"
|
||||||
|
echo "psql -h $PG_HOST -U $DB_USER -d $DB_NAME"
|
||||||
Executable
+255
@@ -0,0 +1,255 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
SMART Logger - Collect SMART data from Scrutiny InfluxDB,
|
||||||
|
calculate BackBlaze failure probability, and log to PostgreSQL.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Dict, List, Optional
|
||||||
|
import json
|
||||||
|
|
||||||
|
from influxdb_client import InfluxDBClient
|
||||||
|
import psycopg
|
||||||
|
from psycopg.rows import dict_row
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
from backblaze_tables import calculate_afr
|
||||||
|
|
||||||
|
load_dotenv()
|
||||||
|
|
||||||
|
|
||||||
|
class SmartLogger:
|
||||||
|
def __init__(self):
|
||||||
|
# InfluxDB configuration
|
||||||
|
self.influx_url = os.getenv('INFLUXDB_URL')
|
||||||
|
self.influx_token = os.getenv('INFLUXDB_TOKEN')
|
||||||
|
self.influx_org = os.getenv('INFLUXDB_ORG', 'scrutiny')
|
||||||
|
self.influx_bucket = os.getenv('INFLUXDB_BUCKET', 'metrics')
|
||||||
|
|
||||||
|
# PostgreSQL configuration
|
||||||
|
self.pg_host = os.getenv('POSTGRES_HOST', 'localhost')
|
||||||
|
self.pg_port = int(os.getenv('POSTGRES_PORT', '5432'))
|
||||||
|
self.pg_db = os.getenv('POSTGRES_DB', 'smart_monitoring')
|
||||||
|
self.pg_user = os.getenv('POSTGRES_USER', 'postgres')
|
||||||
|
self.pg_password = os.getenv('POSTGRES_PASSWORD')
|
||||||
|
|
||||||
|
self.influx_client = None
|
||||||
|
self.pg_conn = None
|
||||||
|
|
||||||
|
def connect_influxdb(self):
|
||||||
|
"""Connect to InfluxDB."""
|
||||||
|
print(f"Connecting to InfluxDB at {self.influx_url}...")
|
||||||
|
self.influx_client = InfluxDBClient(
|
||||||
|
url=self.influx_url,
|
||||||
|
token=self.influx_token,
|
||||||
|
org=self.influx_org
|
||||||
|
)
|
||||||
|
print("✓ Connected to InfluxDB")
|
||||||
|
|
||||||
|
def connect_postgres(self):
|
||||||
|
"""Connect to PostgreSQL."""
|
||||||
|
print(f"Connecting to PostgreSQL at {self.pg_host}:{self.pg_port}...")
|
||||||
|
self.pg_conn = psycopg.connect(
|
||||||
|
host=self.pg_host,
|
||||||
|
port=self.pg_port,
|
||||||
|
dbname=self.pg_db,
|
||||||
|
user=self.pg_user,
|
||||||
|
password=self.pg_password,
|
||||||
|
row_factory=dict_row
|
||||||
|
)
|
||||||
|
print("✓ Connected to PostgreSQL")
|
||||||
|
|
||||||
|
def fetch_smart_data(self) -> List[Dict]:
|
||||||
|
"""
|
||||||
|
Fetch latest SMART data from InfluxDB.
|
||||||
|
|
||||||
|
NOTE: This query needs to be customized based on your Scrutiny schema.
|
||||||
|
Run explore_influx.py first to understand the measurement and field names.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
List of device data dictionaries
|
||||||
|
"""
|
||||||
|
query_api = self.influx_client.query_api()
|
||||||
|
|
||||||
|
# TODO: Customize this query based on your Scrutiny InfluxDB schema
|
||||||
|
# This is a template - you'll need to adjust field names and tags
|
||||||
|
query = f'''
|
||||||
|
from(bucket: "{self.influx_bucket}")
|
||||||
|
|> range(start: -5m)
|
||||||
|
|> filter(fn: (r) => r._measurement == "smart")
|
||||||
|
|> last()
|
||||||
|
|> pivot(rowKey:["_time"], columnKey: ["_field"], valueColumn: "_value")
|
||||||
|
'''
|
||||||
|
|
||||||
|
print("Querying InfluxDB for SMART data...")
|
||||||
|
result = query_api.query(query)
|
||||||
|
|
||||||
|
devices = []
|
||||||
|
for table in result:
|
||||||
|
for record in table.records:
|
||||||
|
# Extract device information
|
||||||
|
# TODO: Adjust these field names based on your schema
|
||||||
|
device_data = {
|
||||||
|
'timestamp': record.get_time(),
|
||||||
|
'device_path': record.values.get('device', 'unknown'),
|
||||||
|
'serial_number': record.values.get('serial', 'unknown'),
|
||||||
|
'model': record.values.get('model', 'unknown'),
|
||||||
|
'temperature': record.values.get('temp', None),
|
||||||
|
'power_on_hours': record.values.get('power_on_hours', None),
|
||||||
|
'smart_attributes': {}
|
||||||
|
}
|
||||||
|
|
||||||
|
# Collect SMART attributes (IDs 5, 187, 188, 197, 198)
|
||||||
|
# TODO: Map your InfluxDB field names to SMART attribute IDs
|
||||||
|
for attr_id in [5, 187, 188, 197, 198]:
|
||||||
|
field_name = f'attr_{attr_id}' # Adjust based on your schema
|
||||||
|
if field_name in record.values:
|
||||||
|
device_data['smart_attributes'][attr_id] = record.values[field_name]
|
||||||
|
|
||||||
|
devices.append(device_data)
|
||||||
|
|
||||||
|
print(f"✓ Found {len(devices)} device(s)")
|
||||||
|
return devices
|
||||||
|
|
||||||
|
def process_device_data(self, devices: List[Dict]) -> List[Dict]:
|
||||||
|
"""
|
||||||
|
Process device data and calculate failure probabilities.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
devices: Raw device data from InfluxDB
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Processed device data with calculated AFR
|
||||||
|
"""
|
||||||
|
processed = []
|
||||||
|
|
||||||
|
for device in devices:
|
||||||
|
# Determine if this is a Seagate disk
|
||||||
|
model = device.get('model', '').lower()
|
||||||
|
is_seagate = 'seagate' in model
|
||||||
|
|
||||||
|
# Calculate failure probability
|
||||||
|
smart_attrs = device.get('smart_attributes', {})
|
||||||
|
afr = calculate_afr(smart_attrs, is_seagate=is_seagate)
|
||||||
|
failure_probability_pct = afr * 100 # Convert to percentage
|
||||||
|
|
||||||
|
# Calculate power-on days
|
||||||
|
power_on_hours = device.get('power_on_hours')
|
||||||
|
power_on_days = power_on_hours // 24 if power_on_hours else None
|
||||||
|
|
||||||
|
# Get error count (SMART attribute 199 or similar)
|
||||||
|
error_count = smart_attrs.get(199, 0) # TODO: Adjust based on your needs
|
||||||
|
|
||||||
|
processed_device = {
|
||||||
|
'timestamp': device.get('timestamp', datetime.now(timezone.utc)),
|
||||||
|
'device_path': device.get('device_path'),
|
||||||
|
'serial_number': device.get('serial_number'),
|
||||||
|
'model': device.get('model'),
|
||||||
|
'disk_role': None, # TODO: Map device to snapraid role if needed
|
||||||
|
'temperature_celsius': device.get('temperature'),
|
||||||
|
'power_on_days': power_on_days,
|
||||||
|
'error_count': error_count,
|
||||||
|
'size_tb': None, # TODO: Get disk size if available
|
||||||
|
'failure_probability_pct': round(failure_probability_pct, 2),
|
||||||
|
'smart_attributes': json.dumps(smart_attrs)
|
||||||
|
}
|
||||||
|
|
||||||
|
processed.append(processed_device)
|
||||||
|
|
||||||
|
# Print summary
|
||||||
|
print(f"\nDevice: {processed_device['device_path']}")
|
||||||
|
print(f" Serial: {processed_device['serial_number']}")
|
||||||
|
print(f" Model: {processed_device['model']}")
|
||||||
|
print(f" Temp: {processed_device['temperature_celsius']}°C")
|
||||||
|
print(f" Power-On: {processed_device['power_on_days']} days")
|
||||||
|
print(f" Failure Probability: {processed_device['failure_probability_pct']:.2f}%")
|
||||||
|
|
||||||
|
return processed
|
||||||
|
|
||||||
|
def save_to_postgres(self, devices: List[Dict]):
|
||||||
|
"""
|
||||||
|
Save processed device data to PostgreSQL.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
devices: Processed device data
|
||||||
|
"""
|
||||||
|
if not devices:
|
||||||
|
print("No devices to save")
|
||||||
|
return
|
||||||
|
|
||||||
|
print(f"\nSaving {len(devices)} device record(s) to PostgreSQL...")
|
||||||
|
|
||||||
|
insert_query = '''
|
||||||
|
INSERT INTO smart_metrics (
|
||||||
|
timestamp, device_path, serial_number, model, disk_role,
|
||||||
|
temperature_celsius, power_on_days, error_count, size_tb,
|
||||||
|
failure_probability_pct, smart_attributes
|
||||||
|
) VALUES (
|
||||||
|
%(timestamp)s, %(device_path)s, %(serial_number)s, %(model)s, %(disk_role)s,
|
||||||
|
%(temperature_celsius)s, %(power_on_days)s, %(error_count)s, %(size_tb)s,
|
||||||
|
%(failure_probability_pct)s, %(smart_attributes)s::jsonb
|
||||||
|
)
|
||||||
|
ON CONFLICT (device_path, timestamp) DO UPDATE SET
|
||||||
|
temperature_celsius = EXCLUDED.temperature_celsius,
|
||||||
|
power_on_days = EXCLUDED.power_on_days,
|
||||||
|
error_count = EXCLUDED.error_count,
|
||||||
|
failure_probability_pct = EXCLUDED.failure_probability_pct,
|
||||||
|
smart_attributes = EXCLUDED.smart_attributes
|
||||||
|
'''
|
||||||
|
|
||||||
|
with self.pg_conn.cursor() as cur:
|
||||||
|
for device in devices:
|
||||||
|
cur.execute(insert_query, device)
|
||||||
|
|
||||||
|
self.pg_conn.commit()
|
||||||
|
print("✓ Data saved to PostgreSQL")
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
"""Main execution flow."""
|
||||||
|
try:
|
||||||
|
self.connect_influxdb()
|
||||||
|
self.connect_postgres()
|
||||||
|
|
||||||
|
devices = self.fetch_smart_data()
|
||||||
|
processed = self.process_device_data(devices)
|
||||||
|
self.save_to_postgres(processed)
|
||||||
|
|
||||||
|
print("\n" + "=" * 80)
|
||||||
|
print("Summary:")
|
||||||
|
print("=" * 80)
|
||||||
|
for device in processed:
|
||||||
|
fp = device['failure_probability_pct']
|
||||||
|
status = "🔴 HIGH RISK" if fp > 50 else "🟡 MEDIUM" if fp > 20 else "🟢 OK"
|
||||||
|
print(f"{status} {device['device_path']}: {fp:.2f}% failure probability")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"ERROR: {e}")
|
||||||
|
import traceback
|
||||||
|
traceback.print_exc()
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
if self.influx_client:
|
||||||
|
self.influx_client.close()
|
||||||
|
if self.pg_conn:
|
||||||
|
self.pg_conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
# Validate environment variables
|
||||||
|
required_vars = ['INFLUXDB_URL', 'INFLUXDB_TOKEN', 'POSTGRES_PASSWORD']
|
||||||
|
missing = [var for var in required_vars if not os.getenv(var)]
|
||||||
|
|
||||||
|
if missing:
|
||||||
|
print(f"ERROR: Missing required environment variables: {', '.join(missing)}")
|
||||||
|
print("Please create a .env file based on .env.example")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
logger = SmartLogger()
|
||||||
|
logger.run()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
Executable
+335
@@ -0,0 +1,335 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
SMART Logger V2 - Optimized for Scrutiny's actual InfluxDB schema.
|
||||||
|
|
||||||
|
Fetches SMART data from Scrutiny's InfluxDB and device metadata from Scrutiny's API,
|
||||||
|
calculates BackBlaze failure probability, and logs to PostgreSQL.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from datetime import datetime, timezone, timedelta
|
||||||
|
from typing import Dict, List, Optional
|
||||||
|
import json
|
||||||
|
import requests
|
||||||
|
|
||||||
|
from influxdb_client import InfluxDBClient
|
||||||
|
import psycopg
|
||||||
|
from psycopg.rows import dict_row
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
from backblaze_tables import calculate_afr
|
||||||
|
|
||||||
|
load_dotenv()
|
||||||
|
|
||||||
|
|
||||||
|
class SmartLogger:
|
||||||
|
def __init__(self):
|
||||||
|
# InfluxDB configuration
|
||||||
|
self.influx_url = os.getenv('INFLUXDB_URL')
|
||||||
|
self.influx_token = os.getenv('INFLUXDB_TOKEN')
|
||||||
|
self.influx_org = os.getenv('INFLUXDB_ORG', 'scrutiny')
|
||||||
|
self.influx_bucket = os.getenv('INFLUXDB_BUCKET', 'metrics')
|
||||||
|
|
||||||
|
# Scrutiny API configuration
|
||||||
|
self.scrutiny_api_url = os.getenv('SCRUTINY_API_URL',
|
||||||
|
self.influx_url.replace(':8086', ':8080'))
|
||||||
|
|
||||||
|
# PostgreSQL configuration
|
||||||
|
self.pg_host = os.getenv('POSTGRES_HOST', 'localhost')
|
||||||
|
self.pg_port = int(os.getenv('POSTGRES_PORT', '5432'))
|
||||||
|
self.pg_db = os.getenv('POSTGRES_DB', 'smart_monitoring')
|
||||||
|
self.pg_user = os.getenv('POSTGRES_USER', 'postgres')
|
||||||
|
self.pg_password = os.getenv('POSTGRES_PASSWORD')
|
||||||
|
|
||||||
|
self.influx_client = None
|
||||||
|
self.pg_conn = None
|
||||||
|
self.device_metadata = {} # WWN -> device info mapping
|
||||||
|
|
||||||
|
def connect_influxdb(self):
|
||||||
|
"""Connect to InfluxDB."""
|
||||||
|
print(f"Connecting to InfluxDB at {self.influx_url}...")
|
||||||
|
self.influx_client = InfluxDBClient(
|
||||||
|
url=self.influx_url,
|
||||||
|
token=self.influx_token,
|
||||||
|
org=self.influx_org
|
||||||
|
)
|
||||||
|
print("✓ Connected to InfluxDB")
|
||||||
|
|
||||||
|
def connect_postgres(self):
|
||||||
|
"""Connect to PostgreSQL."""
|
||||||
|
print(f"Connecting to PostgreSQL at {self.pg_host}:{self.pg_port}...")
|
||||||
|
self.pg_conn = psycopg.connect(
|
||||||
|
host=self.pg_host,
|
||||||
|
port=self.pg_port,
|
||||||
|
dbname=self.pg_db,
|
||||||
|
user=self.pg_user,
|
||||||
|
password=self.pg_password,
|
||||||
|
row_factory=dict_row
|
||||||
|
)
|
||||||
|
print("✓ Connected to PostgreSQL")
|
||||||
|
|
||||||
|
def fetch_device_metadata(self):
|
||||||
|
"""
|
||||||
|
Fetch device metadata from Scrutiny API.
|
||||||
|
This gets serial, model, device path, etc.
|
||||||
|
"""
|
||||||
|
print("Fetching device metadata from Scrutiny API...")
|
||||||
|
try:
|
||||||
|
response = requests.get(f"{self.scrutiny_api_url}/api/summary")
|
||||||
|
response.raise_for_status()
|
||||||
|
data = response.json()
|
||||||
|
|
||||||
|
# Parse the summary response
|
||||||
|
if 'data' in data and 'summary' in data['data']:
|
||||||
|
for wwn, device_info in data['data']['summary'].items():
|
||||||
|
device = device_info.get('device', {})
|
||||||
|
self.device_metadata[wwn] = {
|
||||||
|
'serial_number': device.get('serial_number', 'unknown'),
|
||||||
|
'model': device.get('model_name', 'unknown'),
|
||||||
|
'device_name': device.get('device_name', 'unknown'),
|
||||||
|
'device_type': device.get('device_type', 'unknown'),
|
||||||
|
'manufacturer': device.get('manufacturer', 'unknown'),
|
||||||
|
'capacity': device.get('capacity', 0)
|
||||||
|
}
|
||||||
|
|
||||||
|
print(f"✓ Found metadata for {len(self.device_metadata)} device(s)")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"⚠ Warning: Could not fetch device metadata from API: {e}")
|
||||||
|
print(" Will use WWN as identifier")
|
||||||
|
|
||||||
|
def fetch_smart_data_from_influx(self) -> Dict[str, Dict]:
|
||||||
|
"""
|
||||||
|
Fetch latest SMART data from InfluxDB for all devices.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict mapping WWN to device data
|
||||||
|
"""
|
||||||
|
query_api = self.influx_client.query_api()
|
||||||
|
|
||||||
|
# Query to get latest data for each device
|
||||||
|
query = f'''
|
||||||
|
from(bucket: "{self.influx_bucket}")
|
||||||
|
|> range(start: -1h)
|
||||||
|
|> filter(fn: (r) => r._measurement == "smart")
|
||||||
|
|> filter(fn: (r) =>
|
||||||
|
r._field == "temp" or
|
||||||
|
r._field == "power_on_hours" or
|
||||||
|
r._field == "attr.5.raw_value" or
|
||||||
|
r._field == "attr.187.raw_value" or
|
||||||
|
r._field == "attr.188.raw_value" or
|
||||||
|
r._field == "attr.197.raw_value" or
|
||||||
|
r._field == "attr.198.raw_value" or
|
||||||
|
r._field == "attr.9.raw_value"
|
||||||
|
)
|
||||||
|
|> last()
|
||||||
|
'''
|
||||||
|
|
||||||
|
print("Querying InfluxDB for SMART data...")
|
||||||
|
result = query_api.query(query)
|
||||||
|
|
||||||
|
devices_data = {}
|
||||||
|
|
||||||
|
for table in result:
|
||||||
|
for record in table.records:
|
||||||
|
wwn = record.values.get('device_wwn')
|
||||||
|
if not wwn:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if wwn not in devices_data:
|
||||||
|
devices_data[wwn] = {
|
||||||
|
'timestamp': record.get_time(),
|
||||||
|
'wwn': wwn,
|
||||||
|
'protocol': record.values.get('device_protocol'),
|
||||||
|
'smart_attributes': {},
|
||||||
|
'temp': None,
|
||||||
|
'power_on_hours': None
|
||||||
|
}
|
||||||
|
|
||||||
|
field = record.get_field()
|
||||||
|
value = record.get_value()
|
||||||
|
|
||||||
|
# Map fields to device data
|
||||||
|
if field == 'temp':
|
||||||
|
devices_data[wwn]['temp'] = int(value) if value else None
|
||||||
|
elif field == 'power_on_hours':
|
||||||
|
devices_data[wwn]['power_on_hours'] = int(value) if value else None
|
||||||
|
elif field.startswith('attr.'):
|
||||||
|
# Extract attribute ID from field name like "attr.5.raw_value"
|
||||||
|
parts = field.split('.')
|
||||||
|
if len(parts) >= 2:
|
||||||
|
attr_id = int(parts[1])
|
||||||
|
# We want raw_value for calculations
|
||||||
|
if parts[-1] == 'raw_value':
|
||||||
|
# For most attributes, we want the raw value modulo 2^32
|
||||||
|
# (InfluxDB stores as int64, but SMART is typically 48-bit)
|
||||||
|
raw_val = int(value) if value else 0
|
||||||
|
# Mask to handle potential large values
|
||||||
|
devices_data[wwn]['smart_attributes'][attr_id] = raw_val & 0xFFFF
|
||||||
|
|
||||||
|
print(f"✓ Found data for {len(devices_data)} device(s)")
|
||||||
|
return devices_data
|
||||||
|
|
||||||
|
def process_device_data(self, devices_data: Dict[str, Dict]) -> List[Dict]:
|
||||||
|
"""
|
||||||
|
Process device data and calculate failure probabilities.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
devices_data: Dict mapping WWN to device data from InfluxDB
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Processed device data with calculated AFR
|
||||||
|
"""
|
||||||
|
processed = []
|
||||||
|
|
||||||
|
for wwn, data in devices_data.items():
|
||||||
|
# Get device metadata
|
||||||
|
metadata = self.device_metadata.get(wwn, {})
|
||||||
|
|
||||||
|
# Determine if this is a Seagate disk
|
||||||
|
model = metadata.get('model', '').lower()
|
||||||
|
manufacturer = metadata.get('manufacturer', '').lower()
|
||||||
|
is_seagate = 'seagate' in model or 'seagate' in manufacturer
|
||||||
|
|
||||||
|
# Calculate failure probability
|
||||||
|
smart_attrs = data.get('smart_attributes', {})
|
||||||
|
afr = calculate_afr(smart_attrs, is_seagate=is_seagate)
|
||||||
|
failure_probability_pct = afr * 100 # Convert to percentage
|
||||||
|
|
||||||
|
# Calculate power-on days
|
||||||
|
power_on_hours = data.get('power_on_hours')
|
||||||
|
power_on_days = power_on_hours // 24 if power_on_hours else None
|
||||||
|
|
||||||
|
# Calculate disk size in TB
|
||||||
|
capacity_bytes = metadata.get('capacity', 0)
|
||||||
|
size_tb = round(capacity_bytes / (1000**4), 1) if capacity_bytes else None
|
||||||
|
|
||||||
|
# Get device path/name
|
||||||
|
device_name = metadata.get('device_name', wwn)
|
||||||
|
|
||||||
|
processed_device = {
|
||||||
|
'timestamp': data.get('timestamp', datetime.now(timezone.utc)),
|
||||||
|
'device_path': device_name,
|
||||||
|
'serial_number': metadata.get('serial_number', wwn),
|
||||||
|
'model': metadata.get('model', 'unknown'),
|
||||||
|
'disk_role': None, # TODO: Map to snapraid role if needed
|
||||||
|
'temperature_celsius': data.get('temp'),
|
||||||
|
'power_on_days': power_on_days,
|
||||||
|
'error_count': smart_attrs.get(199, smart_attrs.get(187, 0)),
|
||||||
|
'size_tb': size_tb,
|
||||||
|
'failure_probability_pct': round(failure_probability_pct, 2),
|
||||||
|
'smart_attributes': json.dumps(smart_attrs)
|
||||||
|
}
|
||||||
|
|
||||||
|
processed.append(processed_device)
|
||||||
|
|
||||||
|
# Print summary
|
||||||
|
print(f"\nDevice: {processed_device['device_path']}")
|
||||||
|
print(f" Serial: {processed_device['serial_number']}")
|
||||||
|
print(f" Model: {processed_device['model']}")
|
||||||
|
print(f" Temp: {processed_device['temperature_celsius']}°C")
|
||||||
|
print(f" Power-On: {processed_device['power_on_days']} days")
|
||||||
|
print(f" Size: {processed_device['size_tb']} TB")
|
||||||
|
print(f" SMART Attrs: {list(smart_attrs.keys())}")
|
||||||
|
print(f" Failure Probability: {processed_device['failure_probability_pct']:.2f}%")
|
||||||
|
|
||||||
|
return processed
|
||||||
|
|
||||||
|
def save_to_postgres(self, devices: List[Dict]):
|
||||||
|
"""
|
||||||
|
Save processed device data to PostgreSQL.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
devices: Processed device data
|
||||||
|
"""
|
||||||
|
if not devices:
|
||||||
|
print("No devices to save")
|
||||||
|
return
|
||||||
|
|
||||||
|
print(f"\nSaving {len(devices)} device record(s) to PostgreSQL...")
|
||||||
|
|
||||||
|
insert_query = '''
|
||||||
|
INSERT INTO smart_metrics (
|
||||||
|
timestamp, device_path, serial_number, model, disk_role,
|
||||||
|
temperature_celsius, power_on_days, error_count, size_tb,
|
||||||
|
failure_probability_pct, smart_attributes
|
||||||
|
) VALUES (
|
||||||
|
%(timestamp)s, %(device_path)s, %(serial_number)s, %(model)s, %(disk_role)s,
|
||||||
|
%(temperature_celsius)s, %(power_on_days)s, %(error_count)s, %(size_tb)s,
|
||||||
|
%(failure_probability_pct)s, %(smart_attributes)s::jsonb
|
||||||
|
)
|
||||||
|
ON CONFLICT (device_path, timestamp) DO UPDATE SET
|
||||||
|
temperature_celsius = EXCLUDED.temperature_celsius,
|
||||||
|
power_on_days = EXCLUDED.power_on_days,
|
||||||
|
error_count = EXCLUDED.error_count,
|
||||||
|
failure_probability_pct = EXCLUDED.failure_probability_pct,
|
||||||
|
smart_attributes = EXCLUDED.smart_attributes
|
||||||
|
'''
|
||||||
|
|
||||||
|
with self.pg_conn.cursor() as cur:
|
||||||
|
for device in devices:
|
||||||
|
cur.execute(insert_query, device)
|
||||||
|
|
||||||
|
self.pg_conn.commit()
|
||||||
|
print("✓ Data saved to PostgreSQL")
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
"""Main execution flow."""
|
||||||
|
try:
|
||||||
|
self.connect_influxdb()
|
||||||
|
self.connect_postgres()
|
||||||
|
|
||||||
|
# Fetch device metadata from Scrutiny API
|
||||||
|
self.fetch_device_metadata()
|
||||||
|
|
||||||
|
# Fetch SMART data from InfluxDB
|
||||||
|
devices_data = self.fetch_smart_data_from_influx()
|
||||||
|
|
||||||
|
# Process and calculate failure probabilities
|
||||||
|
processed = self.process_device_data(devices_data)
|
||||||
|
|
||||||
|
# Save to PostgreSQL
|
||||||
|
self.save_to_postgres(processed)
|
||||||
|
|
||||||
|
# Print summary
|
||||||
|
print("\n" + "=" * 80)
|
||||||
|
print("Summary:")
|
||||||
|
print("=" * 80)
|
||||||
|
for device in sorted(processed, key=lambda x: x['failure_probability_pct'], reverse=True):
|
||||||
|
fp = device['failure_probability_pct']
|
||||||
|
status = "🔴 HIGH RISK" if fp > 50 else "🟡 MEDIUM" if fp > 20 else "🟢 OK"
|
||||||
|
print(f"{status} {device['device_path']:20s} {fp:5.2f}% | "
|
||||||
|
f"Temp: {device['temperature_celsius']:2d}°C | "
|
||||||
|
f"Age: {device['power_on_days']:5d} days | "
|
||||||
|
f"Size: {device['size_tb']:4.1f} TB")
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"ERROR: {e}")
|
||||||
|
import traceback
|
||||||
|
traceback.print_exc()
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
if self.influx_client:
|
||||||
|
self.influx_client.close()
|
||||||
|
if self.pg_conn:
|
||||||
|
self.pg_conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
# Validate environment variables
|
||||||
|
required_vars = ['INFLUXDB_URL', 'INFLUXDB_TOKEN', 'POSTGRES_PASSWORD']
|
||||||
|
missing = [var for var in required_vars if not os.getenv(var)]
|
||||||
|
|
||||||
|
if missing:
|
||||||
|
print(f"ERROR: Missing required environment variables: {', '.join(missing)}")
|
||||||
|
print("Please create a .env file based on .env.example")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
logger = SmartLogger()
|
||||||
|
logger.run()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
Executable
+387
@@ -0,0 +1,387 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""
|
||||||
|
SMART Logger V3 - Uses PostgreSQL devices table for metadata.
|
||||||
|
|
||||||
|
Fetches SMART data from Scrutiny's InfluxDB, looks up device metadata from
|
||||||
|
PostgreSQL devices table, calculates BackBlaze failure probability, and logs
|
||||||
|
the enhanced data to PostgreSQL.
|
||||||
|
"""
|
||||||
|
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
from datetime import datetime, timezone
|
||||||
|
from typing import Dict, List, Optional
|
||||||
|
import json
|
||||||
|
|
||||||
|
from influxdb_client import InfluxDBClient
|
||||||
|
import psycopg
|
||||||
|
from psycopg.rows import dict_row
|
||||||
|
from dotenv import load_dotenv
|
||||||
|
|
||||||
|
from backblaze_tables import calculate_afr, afr_to_failure_probability
|
||||||
|
|
||||||
|
load_dotenv()
|
||||||
|
|
||||||
|
|
||||||
|
class SmartLogger:
|
||||||
|
def __init__(self):
|
||||||
|
# InfluxDB configuration
|
||||||
|
self.influx_url = os.getenv('INFLUXDB_URL')
|
||||||
|
self.influx_token = os.getenv('INFLUXDB_TOKEN')
|
||||||
|
self.influx_org = os.getenv('INFLUXDB_ORG', 'scrutiny')
|
||||||
|
self.influx_bucket = os.getenv('INFLUXDB_BUCKET', 'metrics')
|
||||||
|
|
||||||
|
# PostgreSQL configuration
|
||||||
|
self.pg_host = os.getenv('POSTGRES_HOST', 'localhost')
|
||||||
|
self.pg_port = int(os.getenv('POSTGRES_PORT', '5432'))
|
||||||
|
self.pg_db = os.getenv('POSTGRES_DB', 'smart_monitoring')
|
||||||
|
self.pg_user = os.getenv('POSTGRES_USER', 'postgres')
|
||||||
|
self.pg_password = os.getenv('POSTGRES_PASSWORD')
|
||||||
|
|
||||||
|
self.influx_client = None
|
||||||
|
self.pg_conn = None
|
||||||
|
self.device_metadata = {} # WWN -> device info mapping
|
||||||
|
|
||||||
|
def connect_influxdb(self):
|
||||||
|
"""Connect to InfluxDB."""
|
||||||
|
print(f"Connecting to InfluxDB at {self.influx_url}...")
|
||||||
|
self.influx_client = InfluxDBClient(
|
||||||
|
url=self.influx_url,
|
||||||
|
token=self.influx_token,
|
||||||
|
org=self.influx_org
|
||||||
|
)
|
||||||
|
print("✓ Connected to InfluxDB")
|
||||||
|
|
||||||
|
def connect_postgres(self):
|
||||||
|
"""Connect to PostgreSQL."""
|
||||||
|
print(f"Connecting to PostgreSQL at {self.pg_host}:{self.pg_port}...")
|
||||||
|
self.pg_conn = psycopg.connect(
|
||||||
|
host=self.pg_host,
|
||||||
|
port=self.pg_port,
|
||||||
|
dbname=self.pg_db,
|
||||||
|
user=self.pg_user,
|
||||||
|
password=self.pg_password,
|
||||||
|
row_factory=dict_row
|
||||||
|
)
|
||||||
|
print("✓ Connected to PostgreSQL")
|
||||||
|
|
||||||
|
def load_device_metadata(self):
|
||||||
|
"""Load device metadata from PostgreSQL devices table."""
|
||||||
|
print("Loading device metadata from database...")
|
||||||
|
|
||||||
|
query = """
|
||||||
|
SELECT device_wwn, device_path, serial_number, model, manufacturer,
|
||||||
|
capacity_bytes, size_tb, disk_role, notes
|
||||||
|
FROM devices
|
||||||
|
"""
|
||||||
|
|
||||||
|
with self.pg_conn.cursor() as cur:
|
||||||
|
cur.execute(query)
|
||||||
|
for row in cur.fetchall():
|
||||||
|
wwn = row['device_wwn']
|
||||||
|
self.device_metadata[wwn] = dict(row)
|
||||||
|
|
||||||
|
print(f"✓ Loaded metadata for {len(self.device_metadata)} device(s)")
|
||||||
|
|
||||||
|
if not self.device_metadata:
|
||||||
|
print("\n⚠ WARNING: No devices found in the 'devices' table!")
|
||||||
|
print(" Please populate the devices table using populate_devices.sql")
|
||||||
|
print(" Run: psql -h <host> -U <user> -d <db> -f populate_devices.sql\n")
|
||||||
|
|
||||||
|
def fetch_smart_data_from_influx(self) -> Dict[str, Dict]:
|
||||||
|
"""
|
||||||
|
Fetch latest SMART data from InfluxDB for all devices.
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Dict mapping WWN to device data
|
||||||
|
"""
|
||||||
|
query_api = self.influx_client.query_api()
|
||||||
|
|
||||||
|
# Query to get latest data for each device
|
||||||
|
# Include attribute 193 for SnapRAID pre-v13.0 compatibility
|
||||||
|
query = f'''
|
||||||
|
from(bucket: "{self.influx_bucket}")
|
||||||
|
|> range(start: -1h)
|
||||||
|
|> filter(fn: (r) => r._measurement == "smart")
|
||||||
|
|> filter(fn: (r) =>
|
||||||
|
r._field == "temp" or
|
||||||
|
r._field == "power_on_hours" or
|
||||||
|
r._field =~ /failure_rate$/ or
|
||||||
|
r._field == "attr.5.raw_value" or
|
||||||
|
r._field == "attr.187.raw_value" or
|
||||||
|
r._field == "attr.188.raw_value" or
|
||||||
|
r._field == "attr.193.raw_value" or
|
||||||
|
r._field == "attr.197.raw_value" or
|
||||||
|
r._field == "attr.198.raw_value" or
|
||||||
|
r._field == "attr.9.raw_value"
|
||||||
|
)
|
||||||
|
|> last()
|
||||||
|
'''
|
||||||
|
|
||||||
|
print("Querying InfluxDB for SMART data...")
|
||||||
|
result = query_api.query(query)
|
||||||
|
|
||||||
|
devices_data = {}
|
||||||
|
|
||||||
|
for table in result:
|
||||||
|
for record in table.records:
|
||||||
|
wwn = record.values.get('device_wwn')
|
||||||
|
if not wwn:
|
||||||
|
continue
|
||||||
|
|
||||||
|
if wwn not in devices_data:
|
||||||
|
devices_data[wwn] = {
|
||||||
|
'timestamp': record.get_time(),
|
||||||
|
'wwn': wwn,
|
||||||
|
'protocol': record.values.get('device_protocol'),
|
||||||
|
'smart_attributes': {},
|
||||||
|
'failure_rates': {}, # Scrutiny's pre-calculated rates
|
||||||
|
'temp': None,
|
||||||
|
'power_on_hours': None
|
||||||
|
}
|
||||||
|
|
||||||
|
field = record.get_field()
|
||||||
|
value = record.get_value()
|
||||||
|
|
||||||
|
# Map fields to device data
|
||||||
|
if field == 'temp':
|
||||||
|
devices_data[wwn]['temp'] = int(value) if value else None
|
||||||
|
elif field == 'power_on_hours':
|
||||||
|
devices_data[wwn]['power_on_hours'] = int(value) if value else None
|
||||||
|
elif field.startswith('attr.'):
|
||||||
|
# Extract attribute ID from field name like "attr.5.raw_value"
|
||||||
|
parts = field.split('.')
|
||||||
|
if len(parts) >= 2:
|
||||||
|
# Try to parse as integer, skip if not numeric (e.g., NVMe attributes)
|
||||||
|
try:
|
||||||
|
attr_id = int(parts[1])
|
||||||
|
except ValueError:
|
||||||
|
continue # Skip non-numeric attribute IDs (NVMe, etc.)
|
||||||
|
|
||||||
|
# Collect raw_value for logging
|
||||||
|
if parts[-1] == 'raw_value':
|
||||||
|
raw_val = int(value) if value else 0
|
||||||
|
# Apply correct bit mask per attribute (like SnapRAID does)
|
||||||
|
# 16-bit mask for: 187 (Uncorrectable), 188 (Timeout)
|
||||||
|
# 32-bit mask for: 5 (Reallocated), 193 (Load Cycle), 197 (Pending), 198 (Offline)
|
||||||
|
if attr_id in [187, 188]:
|
||||||
|
devices_data[wwn]['smart_attributes'][attr_id] = raw_val & 0xFFFF
|
||||||
|
else:
|
||||||
|
devices_data[wwn]['smart_attributes'][attr_id] = raw_val & 0xFFFFFFFF
|
||||||
|
# Collect Scrutiny's failure_rate for each attribute
|
||||||
|
elif parts[-1] == 'failure_rate':
|
||||||
|
devices_data[wwn]['failure_rates'][attr_id] = float(value) if value else 0.0
|
||||||
|
|
||||||
|
print(f"✓ Found data for {len(devices_data)} device(s)")
|
||||||
|
return devices_data
|
||||||
|
|
||||||
|
def process_device_data(self, devices_data: Dict[str, Dict]) -> List[Dict]:
|
||||||
|
"""
|
||||||
|
Process device data and calculate failure probabilities.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
devices_data: Dict mapping WWN to device data from InfluxDB
|
||||||
|
|
||||||
|
Returns:
|
||||||
|
Processed device data with calculated AFR
|
||||||
|
"""
|
||||||
|
processed = []
|
||||||
|
|
||||||
|
for wwn, data in devices_data.items():
|
||||||
|
# Get device metadata from database
|
||||||
|
metadata = self.device_metadata.get(wwn, {})
|
||||||
|
|
||||||
|
if not metadata:
|
||||||
|
print(f"\n⚠ Warning: No metadata found for device {wwn}")
|
||||||
|
print(f" Add this device to the devices table using populate_devices.sql")
|
||||||
|
continue
|
||||||
|
|
||||||
|
# Get SMART attributes
|
||||||
|
smart_attrs = data.get('smart_attributes', {})
|
||||||
|
|
||||||
|
# Determine if Seagate (for attribute 188 handling)
|
||||||
|
model = metadata.get('model', '').lower()
|
||||||
|
manufacturer = metadata.get('manufacturer', '').lower()
|
||||||
|
is_seagate = 'seagate' in model or 'seagate' in manufacturer
|
||||||
|
|
||||||
|
# Calculate failure probability BOTH ways for trending
|
||||||
|
# v12 (with attr 193) - matches your current SnapRAID
|
||||||
|
afr_v12 = calculate_afr(smart_attrs, is_seagate=is_seagate, include_193=True)
|
||||||
|
prob_v12 = afr_to_failure_probability(afr_v12)
|
||||||
|
failure_probability_v12_pct = prob_v12 * 100
|
||||||
|
|
||||||
|
# v13 (without attr 193) - modern SnapRAID v13.0+
|
||||||
|
afr_v13 = calculate_afr(smart_attrs, is_seagate=is_seagate, include_193=False)
|
||||||
|
prob_v13 = afr_to_failure_probability(afr_v13)
|
||||||
|
failure_probability_v13_pct = prob_v13 * 100
|
||||||
|
|
||||||
|
# Calculate power-on days
|
||||||
|
power_on_hours = data.get('power_on_hours')
|
||||||
|
power_on_days = power_on_hours // 24 if power_on_hours else None
|
||||||
|
|
||||||
|
processed_device = {
|
||||||
|
'timestamp': data.get('timestamp', datetime.now(timezone.utc)),
|
||||||
|
'device_wwn': wwn,
|
||||||
|
'temperature_celsius': data.get('temp'),
|
||||||
|
'power_on_days': power_on_days,
|
||||||
|
'error_count': smart_attrs.get(199, smart_attrs.get(187, 0)),
|
||||||
|
'failure_probability_v12_pct': round(failure_probability_v12_pct, 2),
|
||||||
|
'failure_probability_v13_pct': round(failure_probability_v13_pct, 2),
|
||||||
|
'smart_attributes': json.dumps(smart_attrs)
|
||||||
|
}
|
||||||
|
|
||||||
|
processed.append(processed_device)
|
||||||
|
|
||||||
|
# Print summary with metadata
|
||||||
|
print(f"\n{metadata.get('device_path', wwn)}")
|
||||||
|
print(f" WWN: {wwn}")
|
||||||
|
print(f" Serial: {metadata.get('serial_number', 'unknown')}")
|
||||||
|
print(f" Model: {metadata.get('model', 'unknown')}")
|
||||||
|
print(f" Role: {metadata.get('disk_role', '-')}")
|
||||||
|
print(f" Temp: {processed_device['temperature_celsius']}°C")
|
||||||
|
print(f" Power-On: {processed_device['power_on_days']} days")
|
||||||
|
print(f" Size: {metadata.get('size_tb', '?')} TB")
|
||||||
|
print(f" SMART Attrs: {list(smart_attrs.keys())}")
|
||||||
|
print(f" Failure Probability (v12 w/193): {processed_device['failure_probability_v12_pct']:.2f}%")
|
||||||
|
print(f" Failure Probability (v13 no193): {processed_device['failure_probability_v13_pct']:.2f}%")
|
||||||
|
|
||||||
|
return processed
|
||||||
|
|
||||||
|
def save_to_postgres(self, devices: List[Dict]):
|
||||||
|
"""
|
||||||
|
Save processed device data to PostgreSQL.
|
||||||
|
|
||||||
|
Args:
|
||||||
|
devices: Processed device data
|
||||||
|
"""
|
||||||
|
if not devices:
|
||||||
|
print("\nNo devices to save")
|
||||||
|
return
|
||||||
|
|
||||||
|
print(f"\nSaving {len(devices)} device record(s) to PostgreSQL...")
|
||||||
|
|
||||||
|
insert_query = '''
|
||||||
|
INSERT INTO smart_metrics (
|
||||||
|
timestamp, device_wwn,
|
||||||
|
temperature_celsius, power_on_days, error_count,
|
||||||
|
failure_probability_v12_pct, failure_probability_v13_pct, smart_attributes
|
||||||
|
) VALUES (
|
||||||
|
%(timestamp)s, %(device_wwn)s,
|
||||||
|
%(temperature_celsius)s, %(power_on_days)s, %(error_count)s,
|
||||||
|
%(failure_probability_v12_pct)s, %(failure_probability_v13_pct)s, %(smart_attributes)s::jsonb
|
||||||
|
)
|
||||||
|
ON CONFLICT (device_wwn, timestamp) DO UPDATE SET
|
||||||
|
temperature_celsius = EXCLUDED.temperature_celsius,
|
||||||
|
power_on_days = EXCLUDED.power_on_days,
|
||||||
|
error_count = EXCLUDED.error_count,
|
||||||
|
failure_probability_v12_pct = EXCLUDED.failure_probability_v12_pct,
|
||||||
|
failure_probability_v13_pct = EXCLUDED.failure_probability_v13_pct,
|
||||||
|
smart_attributes = EXCLUDED.smart_attributes
|
||||||
|
'''
|
||||||
|
|
||||||
|
with self.pg_conn.cursor() as cur:
|
||||||
|
for device in devices:
|
||||||
|
cur.execute(insert_query, device)
|
||||||
|
|
||||||
|
self.pg_conn.commit()
|
||||||
|
print("✓ Data saved to PostgreSQL")
|
||||||
|
|
||||||
|
def print_summary(self):
|
||||||
|
"""Print a summary with both v12 and v13 failure calculations."""
|
||||||
|
print("\n" + "=" * 95)
|
||||||
|
print("SMART Summary (v12=with attr 193, v13=without attr 193)")
|
||||||
|
print("=" * 95)
|
||||||
|
print(f"{'Temp':>5} {'Power':>6} {'Error':>7} {'v12':>4} {'v13':>4} {'Size':>5}")
|
||||||
|
print(f"{'C':>5} {'OnDays':>6} {'Count':>7} {'FP':>4} {'FP':>4} {'TB':>5} "
|
||||||
|
f"{'Serial':<16} {'Device':<16} {'Disk':<10}")
|
||||||
|
print("-" * 95)
|
||||||
|
|
||||||
|
query = """
|
||||||
|
SELECT
|
||||||
|
device_path,
|
||||||
|
serial_number,
|
||||||
|
temperature_celsius,
|
||||||
|
power_on_days,
|
||||||
|
error_count,
|
||||||
|
failure_probability_v12_pct,
|
||||||
|
failure_probability_v13_pct,
|
||||||
|
size_tb,
|
||||||
|
disk_role
|
||||||
|
FROM smart_summary
|
||||||
|
ORDER BY
|
||||||
|
CASE disk_role
|
||||||
|
WHEN 'parity' THEN 1
|
||||||
|
WHEN '2-parity' THEN 2
|
||||||
|
ELSE 3
|
||||||
|
END,
|
||||||
|
device_path
|
||||||
|
"""
|
||||||
|
|
||||||
|
with self.pg_conn.cursor() as cur:
|
||||||
|
cur.execute(query)
|
||||||
|
for row in cur.fetchall():
|
||||||
|
temp = row['temperature_celsius'] or 0
|
||||||
|
days = row['power_on_days'] or 0
|
||||||
|
errors = row['error_count'] or 0
|
||||||
|
fp_v12 = int(row['failure_probability_v12_pct'] or 0)
|
||||||
|
fp_v13 = int(row['failure_probability_v13_pct'] or 0)
|
||||||
|
size = row['size_tb'] or 0
|
||||||
|
serial = (row['serial_number'] or '')[:16]
|
||||||
|
device = (row['device_path'] or '')[:16]
|
||||||
|
role = row['disk_role'] or '-'
|
||||||
|
|
||||||
|
print(f"{temp:5d} {days:6d} {errors:7d} {fp_v12:3d}% {fp_v13:3d}% {size:5.1f} "
|
||||||
|
f"{serial:<16} {device:<16} {role:<10}")
|
||||||
|
|
||||||
|
print()
|
||||||
|
|
||||||
|
def run(self):
|
||||||
|
"""Main execution flow."""
|
||||||
|
try:
|
||||||
|
self.connect_influxdb()
|
||||||
|
self.connect_postgres()
|
||||||
|
|
||||||
|
# Load device metadata from PostgreSQL
|
||||||
|
self.load_device_metadata()
|
||||||
|
|
||||||
|
# Fetch SMART data from InfluxDB
|
||||||
|
devices_data = self.fetch_smart_data_from_influx()
|
||||||
|
|
||||||
|
# Process and calculate failure probabilities
|
||||||
|
processed = self.process_device_data(devices_data)
|
||||||
|
|
||||||
|
# Save to PostgreSQL
|
||||||
|
self.save_to_postgres(processed)
|
||||||
|
|
||||||
|
# Print SnapRAID-style summary
|
||||||
|
self.print_summary()
|
||||||
|
|
||||||
|
except Exception as e:
|
||||||
|
print(f"ERROR: {e}")
|
||||||
|
import traceback
|
||||||
|
traceback.print_exc()
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
finally:
|
||||||
|
if self.influx_client:
|
||||||
|
self.influx_client.close()
|
||||||
|
if self.pg_conn:
|
||||||
|
self.pg_conn.close()
|
||||||
|
|
||||||
|
|
||||||
|
def main():
|
||||||
|
# Validate environment variables
|
||||||
|
required_vars = ['INFLUXDB_URL', 'INFLUXDB_TOKEN', 'POSTGRES_PASSWORD']
|
||||||
|
missing = [var for var in required_vars if not os.getenv(var)]
|
||||||
|
|
||||||
|
if missing:
|
||||||
|
print(f"ERROR: Missing required environment variables: {', '.join(missing)}")
|
||||||
|
print("Please create a .env file based on .env.example")
|
||||||
|
sys.exit(1)
|
||||||
|
|
||||||
|
logger = SmartLogger()
|
||||||
|
logger.run()
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == '__main__':
|
||||||
|
main()
|
||||||
@@ -0,0 +1,90 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Test attribute 193 calculation"""
|
||||||
|
|
||||||
|
import math
|
||||||
|
|
||||||
|
SMART_193_R = [
|
||||||
|
0.0000, 0.0016, 0.0032, 0.0036, 0.0039,
|
||||||
|
0.0042, 0.0046, 0.0049, 0.0052, 0.0054,
|
||||||
|
0.0057, 0.0060, 0.0062, 0.0065, 0.0068,
|
||||||
|
0.0071, 0.0074, 0.0077, 0.0080, 0.0083,
|
||||||
|
0.0086, 0.0091, 0.0094, 0.0098, 0.0101,
|
||||||
|
0.0104, 0.0108, 0.0111, 0.0119, 0.0122,
|
||||||
|
0.0127, 0.0130, 0.0134, 0.0137, 0.0141,
|
||||||
|
0.0144, 0.0146, 0.0152, 0.0155, 0.0159,
|
||||||
|
0.0163, 0.0165, 0.0168, 0.0172, 0.0176,
|
||||||
|
0.0179, 0.0184, 0.0188, 0.0190, 0.0194,
|
||||||
|
0.0197, 0.0201, 0.0204, 0.0207, 0.0209,
|
||||||
|
0.0213, 0.0215, 0.0219, 0.0221, 0.0225,
|
||||||
|
0.0229, 0.0234, 0.0241, 0.0246, 0.0253,
|
||||||
|
0.0263, 0.0278, 0.0286, 0.0293, 0.0298,
|
||||||
|
0.0302, 0.0306, 0.0311, 0.0315, 0.0319,
|
||||||
|
0.0322, 0.0329, 0.0334, 0.0338, 0.0343,
|
||||||
|
0.0348, 0.0352, 0.0358, 0.0362, 0.0367,
|
||||||
|
0.0371, 0.0374, 0.0378, 0.0383, 0.0388,
|
||||||
|
0.0393, 0.0397, 0.0401, 0.0404, 0.0410,
|
||||||
|
0.0416, 0.0422, 0.0428, 0.0436, 0.0443,
|
||||||
|
0.0449, 0.0454, 0.0457, 0.0462, 0.0468,
|
||||||
|
0.0473, 0.0479, 0.0483, 0.0488, 0.0491,
|
||||||
|
0.0493, 0.0497, 0.0500, 0.0504, 0.0507,
|
||||||
|
0.0510, 0.0514, 0.0519, 0.0523, 0.0528,
|
||||||
|
0.0533, 0.0538, 0.0542, 0.0547, 0.0551,
|
||||||
|
0.0556, 0.0560, 0.0565, 0.0572, 0.0577,
|
||||||
|
0.0584, 0.0590, 0.0594, 0.0599, 0.0603,
|
||||||
|
0.0607, 0.0611, 0.0616, 0.0621, 0.0626,
|
||||||
|
0.0632, 0.0639, 0.0647, 0.0655, 0.0661,
|
||||||
|
0.0669, 0.0676, 0.0683, 0.0691, 0.0699,
|
||||||
|
0.0708, 0.0713, 0.0719, 0.0724, 0.0730,
|
||||||
|
0.0736, 0.0745, 0.0751, 0.0759, 0.0769,
|
||||||
|
0.0779, 0.0787, 0.0796, 0.0804, 0.0815,
|
||||||
|
0.0825, 0.0833, 0.0840, 0.0847, 0.0854,
|
||||||
|
0.0859, 0.0865, 0.0873, 0.0881, 0.0890,
|
||||||
|
0.0900, 0.0912, 0.0919, 0.0929, 0.0942,
|
||||||
|
0.0956, 0.0965, 0.0976, 0.0986, 0.0995,
|
||||||
|
0.1006, 0.1019, 0.1031, 0.1038, 0.1045,
|
||||||
|
0.1051, 0.1058, 0.1066, 0.1072, 0.1077,
|
||||||
|
0.1084, 0.1091, 0.1099, 0.1104, 0.1111,
|
||||||
|
0.1118, 0.1127, 0.1135, 0.1142, 0.1149,
|
||||||
|
0.1157, 0.1163, 0.1168, 0.1173, 0.1179,
|
||||||
|
0.1184, 0.1189, 0.1195, 0.1203, 0.1208,
|
||||||
|
0.1213, 0.1223, 0.1231, 0.1240, 0.1246,
|
||||||
|
0.1252, 0.1260, 0.1269, 0.1276, 0.1287,
|
||||||
|
0.1303, 0.1311, 0.1319, 0.1328, 0.1335,
|
||||||
|
0.1341, 0.1348, 0.1362, 0.1373, 0.1380,
|
||||||
|
0.1387, 0.1392, 0.1398, 0.1403, 0.1408,
|
||||||
|
0.1412, 0.1418, 0.1422, 0.1428, 0.1434,
|
||||||
|
0.1439, 0.1445, 0.1451, 0.1457, 0.1464,
|
||||||
|
0.1469, 0.1475, 0.1480, 0.1486, 0.1491,
|
||||||
|
0.1498,
|
||||||
|
]
|
||||||
|
|
||||||
|
SMART_193_STEP = 649
|
||||||
|
|
||||||
|
# User's drive data
|
||||||
|
raw_value = 150091
|
||||||
|
|
||||||
|
# Calculate table index
|
||||||
|
table_index = raw_value // SMART_193_STEP
|
||||||
|
print(f"Attribute 193 RAW_VALUE: {raw_value}")
|
||||||
|
print(f"STEP: {SMART_193_STEP}")
|
||||||
|
print(f"Table index: {raw_value} // {SMART_193_STEP} = {table_index}")
|
||||||
|
|
||||||
|
# Get monthly rate
|
||||||
|
if table_index >= len(SMART_193_R):
|
||||||
|
table_index = len(SMART_193_R) - 1
|
||||||
|
|
||||||
|
monthly_rate = SMART_193_R[table_index]
|
||||||
|
print(f"Monthly failure rate (SMART_193_R[{table_index}]): {monthly_rate}")
|
||||||
|
|
||||||
|
# Annualize
|
||||||
|
afr = (365.0 / 30.0) * monthly_rate
|
||||||
|
print(f"Annualized: (365/30) * {monthly_rate} = {afr:.4f}")
|
||||||
|
|
||||||
|
# Apply Poisson
|
||||||
|
prob = (1 - math.exp(-afr)) * 100
|
||||||
|
print(f"Poisson P(failure): (1 - e^(-{afr:.4f})) * 100 = {prob:.2f}%")
|
||||||
|
|
||||||
|
print("\n" + "="*60)
|
||||||
|
print(f"SnapRAID reports: 81%")
|
||||||
|
print(f"Our calculation: {prob:.2f}%")
|
||||||
|
print("="*60)
|
||||||
@@ -0,0 +1,44 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Test BackBlaze calculation with different values"""
|
||||||
|
|
||||||
|
import math
|
||||||
|
from backblaze_tables import SMART_5_R, SMART_187_R, SMART_188_R, SMART_197_R, SMART_198_R
|
||||||
|
|
||||||
|
def poisson_prob(rate):
|
||||||
|
"""Calculate P(at least 1 failure) = 1 - e^(-rate)"""
|
||||||
|
return 1 - math.exp(-rate)
|
||||||
|
|
||||||
|
def test_calculation(attr_value):
|
||||||
|
"""Test with a given attribute value"""
|
||||||
|
# Scale monthly to annual (like snapraid does)
|
||||||
|
afr_5 = (365.0 / 30.0) * SMART_5_R[min(attr_value, 255)]
|
||||||
|
afr_187 = (365.0 / 30.0) * SMART_187_R[min(attr_value, 255)]
|
||||||
|
afr_188 = (365.0 / 30.0) * SMART_188_R[min(attr_value, 255)]
|
||||||
|
afr_197 = (365.0 / 30.0) * SMART_197_R[min(attr_value, 255)]
|
||||||
|
afr_198 = (365.0 / 30.0) * SMART_198_R[min(attr_value, 255)]
|
||||||
|
|
||||||
|
# Take maximum (like snapraid)
|
||||||
|
max_afr = max(afr_5, afr_187, afr_188, afr_197, afr_198)
|
||||||
|
|
||||||
|
# Apply Poisson
|
||||||
|
prob = poisson_prob(max_afr) * 100
|
||||||
|
|
||||||
|
return max_afr, prob
|
||||||
|
|
||||||
|
print("Testing different attribute values:")
|
||||||
|
print("=" * 80)
|
||||||
|
|
||||||
|
for test_val in [0, 50, 100, 150, 200, 255]:
|
||||||
|
afr, prob = test_calculation(test_val)
|
||||||
|
print(f"Attribute value {test_val:3d}: AFR={afr:.4f}, Probability={prob:.2f}%")
|
||||||
|
|
||||||
|
print("\n" + "=" * 80)
|
||||||
|
print("What we're seeing from /dev/sdc:")
|
||||||
|
print(" All critical attributes: RAW_VALUE=0, VALUE=100")
|
||||||
|
print(" SnapRAID reports: 81%")
|
||||||
|
print(" Our calculation with value=0: ", end="")
|
||||||
|
afr0, prob0 = test_calculation(0)
|
||||||
|
print(f"{prob0:.2f}%")
|
||||||
|
print(" Our calculation with value=100:", end="")
|
||||||
|
afr100, prob100 = test_calculation(100)
|
||||||
|
print(f"{prob100:.2f}%")
|
||||||
Reference in New Issue
Block a user