Added some more documentation to ETL scripts

stm_etl/AS400_Invoice_Interface.py

@@ -1,9 +1,44 @@
 '''
-Attempting to automatically close invoices in the AS400 using data from Jomar.. weee...
+AS400 Invoice Interface - Jomar to AS400 Invoice Synchronization
 
 Created on: 2017-11-20
-
 @author: Wes
+
+PURPOSE:
+    Pushes invoice data from Jomar ERP into the AS400 so that invoices can be
+    automatically closed on the AS400 side without manual entry.
+
+USAGE:
+    python AS400_Invoice_Interface.py <date>
+
+    The date argument (format matching Jomar's RDATLS field) selects which
+    invoices to extract. Typically run daily for the current invoice date.
+
+PROCESSING LOGIC:
+    1. Extract from Jomar (MSSQL via SQLAlchemy):
+        - Query joins invoice rolls (txuyre01), roll/tag master (TXRYRT00),
+          order header (TXUYUV00), and shipment table (TXUYLS00) to build a
+          row set of: invoice number, invoice line, ticket, AS400 order number,
+          invoiced yards, finished yards, ship date, and invoice date.
+        - Only pulls invoices matching the supplied date.
+        - Excludes FREIGHT items and sales return (SR) lines.
+
+    2. Load into AS400 (pyodbc via iSeries ODBC):
+        - Backs up existing rows from STMDATA.JSIINV to WES.JSIINV_BAK
+          (only when running against production library).
+        - Deletes all current rows from STMDATA.JSIINV (staging table).
+        - Inserts each extracted row into STMDATA.JSIINV.
+        - Rows missing a ticket number, order number, or with non-numeric
+          tickets (e.g. credits) are skipped and collected as exceptions.
+
+    3. Exception handling:
+        - Exception rows are written to a temporary CSV (AII_ERRORS_{date}.csv).
+        - CSV is emailed to RECIPIENTS, then deleted from disk.
+
+OUTPUT:
+    - STMDATA.JSIINV populated on the AS400 for downstream invoice closure.
+    - Log file at D:\AS400_Invoice_Interface.log
+    - Email alert with exception CSV if any rows were skipped.
 '''
 import pyodbc, time, logging, sys, os, StringIO, csv, sqlite3
 from sqlalchemy import create_engine

stm_etl/Finished_Goods_Interface.py

@@ -1,22 +1,76 @@
 '''
+Finished Goods Interface - AS400 to Jomar Inventory Synchronization
+
 Created on Jun 24, 2015
-
-If no arguments supplied script looks for CSV's in the folder specified by PATH.
-Can be run for single file by supplying file path as arg. This is the preferred way of doing things
-if the filewatcher powershell script is enabled to avoid collisions.
-
-Creates various inventory transactions needed to keep Jomar inventory up to date with what is entering
-shipping dept from the AS400
-
-Transaction data is written to TXRYBW10 and processed immediatley.
-See 'Inventory Control' -> 'Visibility' -> 'Inventory Errors' for transactions that failed to process
-
-TEST_MODE, when True:
-   * Prevents files being moved to the PROCESSED folder.
-   * Will _NOT_ e-mail Aaron
-   * Does not commit any transactions to the database
-
 @author: Wes
+
+PURPOSE:
+    Reads CSV files generated by the AS400 when inventory is scanned in shipping and creates
+    corresponding Jomar ERP inventory transactions to keep the two systems in sync.
+
+INPUT:
+    CSV files in D:\INTERFACE\ named as400_jsi_#####.csv (where ##### is a sequential transaction number).
+    Each row represents a roll/piece with columns:
+        [0] ORDER - AS400 order number (e.g. U17435)
+        [1] LINE - AS400 line number
+        [2] TICKET - Roll/piece ticket number (unique identifier)
+        [3] FINISHED_YARDS
+        [4] INVOICED_YARDS
+        [5] BIN - Physical bin location
+        [6] WAREHOUSE - Warehouse code (SH, BG, B8, etc.)
+        [7] WEIGHT - Roll weight in lbs
+        [8] SPAT - Pattern code
+        [9] SCOL - Color code
+
+    Can be run two ways:
+        - No arguments: processes all CSV files in PATH
+        - Single file path argument: processes just that file (preferred when using the PowerShell
+          file watcher to avoid collisions between concurrent runs)
+
+PROCESSING LOGIC (per row):
+    1. Validation checks (skip row to error list if any fail):
+        - Duplicate ticket within same transmission
+        - Invalid bin (HLD, or blank bin in SH warehouse)
+        - Finished yards <= 0
+        - Invoiced yards <= 0
+
+    2. Order/line lookup in Jomar:
+        - Query TXUYUV00 (order header) to map AS400 order -> Jomar order number
+        - Query TXUYUF01 (order detail) + TXMYAT03 (item info) to find Jomar line number
+        - Verify pattern and color match between AS400 and Jomar
+
+    3. Transaction generation based on roll state in Jomar (TXRYRT00 roll/tag master):
+        a. Roll does NOT exist in Jomar:
+            -> Create Production Receipt (08) transaction to add the roll
+            -> Sets warehouse/bin, yardage, weight, lot, NAFTA qualification
+            -> Pull orders (PULL_ORDERS list) get no order/line linkage
+
+        b. Roll ALREADY exists in Jomar:
+            - If shipped yards > 0: error (roll already shipped, can't modify)
+            - If yardage or weight differs from current values:
+                -> Create adjustment Production Receipt (08) with the delta
+            - If bin or warehouse differs:
+                -> Create Movement (06) transaction to relocate the roll
+            - If nothing differs: skip (log and ignore)
+
+    All transactions are written to TXRYBW10 (the finished goods interface table) and
+    processed immediately by Jomar.
+
+ERROR HANDLING:
+    - Rows that fail validation or lookup are collected in eRows
+    - Error rows are written to D:\INTERFACE\ERRORS\ERRORS_{lot}.csv
+    - Error CSV is emailed to ROLL_ERROR_RECIPIENTS
+    - If a file with a lower transaction number is still unprocessed when a newer file arrives,
+      the script alerts TX_ERROR_RECIPIENTS and exits (prevents out-of-order processing)
+
+OUTPUT:
+    - Processed CSV files are moved to D:\INTERFACE\PROCESSED\
+    - Log file written to D:\INTERFACE\Finished_Goods_Interface.log
+
+TEST_MODE (when True):
+    - Prevents files being moved to the PROCESSED folder
+    - Suppresses email notifications
+    - Rolls back all database transactions instead of committing
 '''
 import ConfigParser
 import csv, os, smtplib