wes/Python_Examples
1
0
Fork 0
Code Issues Pull Requests Releases Wiki Activity

Added some more documentation to ETL scripts

Browse Source
This commit is contained in:
wes
2026-02-11 07:21:01 -05:00
parent c4b970cbde
commit 24b2eca388
2 changed files with 107 additions and 18 deletions
Download Patch File Download Diff File Expand all files Collapse all files
stm_etl/AS400_Invoice_Interface.py
+37 -2
View File
@@ -1,9 +1,44 @@
'''
Attempting to automatically close invoices in the AS400 using data from Jomar.. weee...
AS400 Invoice Interface - Jomar to AS400 Invoice Synchronization
Created on: 2017-11-20
@author: Wes
PURPOSE:
Pushes invoice data from Jomar ERP into the AS400 so that invoices can be
automatically closed on the AS400 side without manual entry.
USAGE:
python AS400_Invoice_Interface.py <date>
The date argument (format matching Jomar's RDATLS field) selects which
invoices to extract. Typically run daily for the current invoice date.
PROCESSING LOGIC:
1. Extract from Jomar (MSSQL via SQLAlchemy):
- Query joins invoice rolls (txuyre01), roll/tag master (TXRYRT00),
order header (TXUYUV00), and shipment table (TXUYLS00) to build a
row set of: invoice number, invoice line, ticket, AS400 order number,
invoiced yards, finished yards, ship date, and invoice date.
- Only pulls invoices matching the supplied date.
- Excludes FREIGHT items and sales return (SR) lines.
2. Load into AS400 (pyodbc via iSeries ODBC):
- Backs up existing rows from STMDATA.JSIINV to WES.JSIINV_BAK
(only when running against production library).
- Deletes all current rows from STMDATA.JSIINV (staging table).
- Inserts each extracted row into STMDATA.JSIINV.
- Rows missing a ticket number, order number, or with non-numeric
tickets (e.g. credits) are skipped and collected as exceptions.
3. Exception handling:
- Exception rows are written to a temporary CSV (AII_ERRORS_{date}.csv).
- CSV is emailed to RECIPIENTS, then deleted from disk.
OUTPUT:
- STMDATA.JSIINV populated on the AS400 for downstream invoice closure.
- Log file at D:\AS400_Invoice_Interface.log
- Email alert with exception CSV if any rows were skipped.
'''
import pyodbc, time, logging, sys, os, StringIO, csv, sqlite3
from sqlalchemy import create_engine
stm_etl/Finished_Goods_Interface.py
+70 -16
View File
@@ -1,22 +1,76 @@
'''
Finished Goods Interface - AS400 to Jomar Inventory Synchronization
Created on Jun 24, 2015
If no arguments supplied script looks for CSV's in the folder specified by PATH.
Can be run for single file by supplying file path as arg. This is the preferred way of doing things
if the filewatcher powershell script is enabled to avoid collisions.
Creates various inventory transactions needed to keep Jomar inventory up to date with what is entering
shipping dept from the AS400
Transaction data is written to TXRYBW10 and processed immediatley.
See 'Inventory Control' -> 'Visibility' -> 'Inventory Errors' for transactions that failed to process
TEST_MODE, when True:
* Prevents files being moved to the PROCESSED folder.
* Will _NOT_ e-mail Aaron
* Does not commit any transactions to the database
@author: Wes
PURPOSE:
Reads CSV files generated by the AS400 when inventory is scanned in shipping and creates
corresponding Jomar ERP inventory transactions to keep the two systems in sync.
INPUT:
CSV files in D:\INTERFACE\ named as400_jsi_#####.csv (where ##### is a sequential transaction number).
Each row represents a roll/piece with columns:
[0] ORDER - AS400 order number (e.g. U17435)
[1] LINE - AS400 line number
[2] TICKET - Roll/piece ticket number (unique identifier)
[3] FINISHED_YARDS
[4] INVOICED_YARDS
[5] BIN - Physical bin location
[6] WAREHOUSE - Warehouse code (SH, BG, B8, etc.)
[7] WEIGHT - Roll weight in lbs
[8] SPAT - Pattern code
[9] SCOL - Color code
Can be run two ways:
- No arguments: processes all CSV files in PATH
- Single file path argument: processes just that file (preferred when using the PowerShell
file watcher to avoid collisions between concurrent runs)
PROCESSING LOGIC (per row):
1. Validation checks (skip row to error list if any fail):
- Duplicate ticket within same transmission
- Invalid bin (HLD, or blank bin in SH warehouse)
- Finished yards <= 0
- Invoiced yards <= 0
2. Order/line lookup in Jomar:
- Query TXUYUV00 (order header) to map AS400 order -> Jomar order number
- Query TXUYUF01 (order detail) + TXMYAT03 (item info) to find Jomar line number
- Verify pattern and color match between AS400 and Jomar
3. Transaction generation based on roll state in Jomar (TXRYRT00 roll/tag master):
a. Roll does NOT exist in Jomar:
-> Create Production Receipt (08) transaction to add the roll
-> Sets warehouse/bin, yardage, weight, lot, NAFTA qualification
-> Pull orders (PULL_ORDERS list) get no order/line linkage
b. Roll ALREADY exists in Jomar:
- If shipped yards > 0: error (roll already shipped, can't modify)
- If yardage or weight differs from current values:
-> Create adjustment Production Receipt (08) with the delta
- If bin or warehouse differs:
-> Create Movement (06) transaction to relocate the roll
- If nothing differs: skip (log and ignore)
All transactions are written to TXRYBW10 (the finished goods interface table) and
processed immediately by Jomar.
ERROR HANDLING:
- Rows that fail validation or lookup are collected in eRows
- Error rows are written to D:\INTERFACE\ERRORS\ERRORS_{lot}.csv
- Error CSV is emailed to ROLL_ERROR_RECIPIENTS
- If a file with a lower transaction number is still unprocessed when a newer file arrives,
the script alerts TX_ERROR_RECIPIENTS and exits (prevents out-of-order processing)
OUTPUT:
- Processed CSV files are moved to D:\INTERFACE\PROCESSED\
- Log file written to D:\INTERFACE\Finished_Goods_Interface.log
TEST_MODE (when True):
- Prevents files being moved to the PROCESSED folder
- Suppresses email notifications
- Rolls back all database transactions instead of committing
'''
import ConfigParser
import csv, os, smtplib