Radix-Skills/skills/data-science/erp-data-reconciliation/SKILL.md

---
name: erp-data-reconciliation
description: |
  Analyze and reconcile financial/accounting data between two ERP systems
  (e.g. SAP/Coop and Uniconta) using Excel exports. Covers file inspection,
  column mapping, Danish-format normalization, match-engine design, and
  iterative build workflow.
trigger: |
  User asks to reconcile, match, compare, or afstemme data between two
  accounting/ERP systems, typically via Excel files. Also relevant when
  importing/exporting financial ledgers, kontoudtog, vendor/customer
  statements, or bonus calculations across systems.
---

# ERP Data Reconciliation Skill

## Overview

Reconciling data between two ERP/accounting systems is a multi-phase task.
The typical deliverable is a webapp or script that can:
1. Import Excel files from both systems
2. Normalize dates, amounts, and text
3. Match postings between systems
4. Show unmatched, partial matches, and differences
5. Support manual approval/rejection
6. Generate reports

## Phase 1: File Analysis (CRITICAL — do not skip)

Before writing any import code, run a structured analysis of the Excel files.

### Steps
1. List all files in both source directories
2. For each file, identify:
   - Sheet names
   - Actual header row location (may NOT be row 1!)
   - Column names and data types
   - Date columns and their formats
   - Amount columns and their formats
   - Text columns
   - Reference/invoice/document columns
   - Null percentages
   - Duplicate rows
   - Data range (min/max dates)

### Key Pitfalls
- **Header row is NOT always row 1.** Uniconta exports often have account
  info in rows 1-2, blank row 3, headers in row 4.
- **Amounts mixed formats.** Danish Excel files often have amounts as both
  Excel floats AND as Danish text strings like `-2.543.803,93`.
  Always normalize using a robust parser:
  ```python
  def parse_danish_number(val):
      if pd.isna(val): return float('nan')
      if isinstance(val, (int, float)): return float(val)
      s = str(val).strip()
      if not s: return float('nan')
      if '.' in s and ',' in s:
          s = s.replace('.', '').replace(',', '.')
      elif ',' in s: s = s.replace(',', '.')
      return float(s)
  ```
- **Invoice/reference numbers may NOT match between systems.** This is the
  #1 cause of failed reconciliation. Always check overlap BEFORE assuming
  IDs can be used as join keys. Use amount overlap as a quick viability check.

### Analysis Script Template
Use or adapt `scripts/analyze_excel.py` from the Coop-Uniconta project:
- Reads all sheets in all files
- Prints column stats, null counts, sample values
- Detects Danish number formats
- Identifies empty/duplicate rows

## Phase 2: Determine Match Strategy

After analysis, document the match strategy BEFORE coding the engine.

### Decision Tree
1. **Do invoice/reference numbers overlap?** 
   - If yes → use `Faktura + abs(Amount) + Date` as PRIMARY signal (two-phase).
     First phase: Reference==Faktura + abs(Amount) within tolerance (±5 DKK) + Date±21d.
     Second phase: abs(Amount) + Date±21d for remaining rows.
   - If no → use `Amount + Date + Text similarity`
2. **Are dates exact or within tolerance?**
   - Same day → exact
   - ±1-3 days → high confidence partial
   - ±7 days → medium confidence
   - ±21 days → general match tolerance
   - >21 days → fuzzy/manual only
3. **Is text available on both sides?**
   - Both have text → Levenshtein/rapidfuzz similarity score
   - Only one side has text → amount+date only
4. **Do signs differ?**
   - Same sign → same direction transaction
   - Opposite sign → same transaction viewed from creditor vs debtor side.
     Match on `abs(amount)` but label "MODSAT" in output.  
     *Example: Coop RE (payment) is negative; Uniconta Faktura is positive.*

### Tolerance Matching (V6 innovation)

Instead of exact amount match (±0.01), use a tolerance (±5 DKK):
- Catches rounding differences between systems
- Catches partial payments (one system may record 440.000, the other 430.732)
- Uses bucket-based indexing for fast lookup:
  ```python
  AMT_TOL = 5.0
  bucket = round(abs_amount / AMT_TOL) * AMT_TOL
  # Search adjacent buckets: [bucket-AMT_TOL, bucket, bucket+AMT_TOL]
  ```

### Coop Fakturaliste + Uniconta Betalingsfordeling as the Simple Forward Workflow

When the user wants the process to be as simple as possible, prefer a forward-looking workflow based on:
1. `Payment_advice`
2. latest Uniconta Coop debtor/payment allocation export
3. the complete Coop invoice list

Model the invoice list and payment allocation as separate import sources. Use invoice number as the primary join between Coop invoices and Uniconta allocation lines, and calculate paid/open/partial status from payment/settlement lines rather than only from original invoice postings. Treat the payment allocation as an explanation layer between a bank deposit and many invoice/credit lines.

See `references/coop-invoice-payment-allocation.md` for the concrete Django model/import/status pattern, UI shape, and tests.

### Coop Payment Advice / Settlement Files as a Fourth Source

When the user provides Coop betalingsadviseringer/opgørelser for individual bank
payments, treat them as a separate reconciliation source, not just attachments.
They are often the missing bridge:

`Bank deposit → Coop payment advice total → advice invoice/credit lines → Coop ledger / Uniconta invoices`

For Coop-Uniconta style systems, the import UI should expose the full operational set, not just "Coop" and "Uniconta":
1. Uniconta Coop Debitor / payment allocation
2. Uniconta/ERP invoice list: all Coop invoices/credit notes for the year
3. Coop account statement / kontoudtog
4. Bank statement containing all Coop postings
5. Coop Payment_advice documents, uploadable continuously as they are collected

Recommended source/model names:
- `coop_settlement` or `coop_payment_advice`
- `PaymentAdvice` / `PaymentAdviceDocument` / `SettlementFile`: file name, unique id/bilag number, file type
  (PDF/XLSX), document date, payment date, total amount, period, parse status, matched bank transaction, raw
  text/raw JSON.
- `PaymentAdviceLine` / `SettlementLine`: invoice number, invoice date, company
  code, amount, currency, line type (invoice/credit/bonus/modregning/unknown),
  links to bank/Coop/Uniconta transactions, match status, explanation.

If the user does not yet have all payment advice documents, still allow upload now. Store raw PDF text or Excel row JSON with best-effort date/amount/reference/classification so later parsers can be tightened without re-requesting files.

Parser patterns observed:
- XLSX files may have header row 3 with columns `Fakturanummer`, `Fakturadato`,
  `Firmakode`, `Beløb`, `Valuta` and supplier metadata in rows 1-2.
- Text PDFs often extract as repeated triples:
  invoice number line, `dd.mm.yyyy firmakode` line, amount/`DKK` line.
- Some PDFs split sign into a separate `-` column/line; treat `123,45`, `-`,
  `DKK` as `-123,45`.
- Some PDFs are bank statement screenshots, not payment advice. Classify and do
  not parse them as settlement lines.
- Some files include manual middle calculations or special layouts. Keep raw text
  and mark parse confidence/status instead of silently forcing a schema.

Matching/reporting logic:
1. Match payment-advice total to bank deposit using ±5 DKK tolerance.
2. Then match advice lines to Uniconta by invoice number first, then amount/date.
3. Match advice lines to Coop ledger by reference/doc number; in the Coop-Uniconta
   project this was a very strong signal.
4. Treat payment-advice matches as an evidence/explanation layer, not as automatic
   ledger settlement. An invoice can remain open in Uniconta while Payment_advice
   proves Coop included it in a settlement, credit, bonus, modregning, or manual
   middle calculation.
5. Report per bank deposit/open item: bank date/amount, matched advice file,
   difference, line count, Uniconta hit count, Coop hit count, credit/bonus/
   modregning total, advice documents/vouchers/classifications, and status OK /
   partial / needs review.
6. Keep 2026 or out-of-period advice files in the database, but exclude or label
   them separately when current bank/Uniconta data only covers 2024-2025.

### Invoice Status: Net-based + Deduplication

**Critical pitfall:** The same payment may exist in BOTH systems (Coop RE and Uniconta Betaling). If you sum both, you get double the actual paid amount.

**Solution:** Calculate net payments per invoice:
```python
# Net Coop payments (NOT abs-sum!)
coop_net = sum(c['amount'] for c in coop 
               if c['reference'] == fnr and c['doc_type'] in ('RE','ZP'))
coop_paid = abs(coop_net) if coop_net < 0 else 0

# Net Uniconta payments
uni_net = sum(u['amount'] for u in uni 
            if u['faktura'] == fnr and u['konteringstype'] == 'Betaling')
uni_paid = abs(uni_net) if uni_net < 0 else 0

# Deduplicate: if both exist, use only one
if coop_paid > 0 and uni_paid > 0:
    if abs(coop_paid - uni_paid) <= AMT_TOL:
        betalt = coop_paid  # Same payment recorded twice
    else:
        betalt = max(coop_paid, uni_paid)  # Partial, take largest
```

**Also handle:** Positive Coop RE amounts are credit memos (modregninger) that should be SUBTRACTED from the net, not added with abs().

### Match Scoring Formula (default)
```
Score = 0
# Amount match (40%)
if abs(a-b) < AMT_TOL: Score += 40

# Date match (35%)
days = abs(d1 - d2).days
if days == 0: Score += 35
elif days <= 3: Score += 25
elif days <= 7: Score += 15
elif days <= 14: Score += 5

# Text match (25%)
if both_have_text:
    similarity = fuzzy_ratio(text1, text2)  # 0-100
    if similarity > 80: Score += 25
    elif similarity > 50: Score += 12

# Reference bonus (extra)
if reference_match: Score += 40

# Thresholds
if Score >= 85: exact_match, auto_approvable
if Score >= 60: partial_match, requires_manual
if Score >= 40: fuzzy_match, requires_manual
```

| Table | Key Fields |
|-------|-----------|
| `ImportBatch` | source (coop/uniconta), filename, imported_at, raw_row_count |
| `Posting` | batch_id, source, fiscal_year, posting_date, amount, original_text, normalized_text, reference, document_number, original_data (JSON) |
| `Match` | posting_a_id, posting_b_id, score, match_type (exact/partial/fuzzy/manual), status (proposed/approved/rejected), matched_by (auto/user), matched_at |
| `AuditLog` | user, action, match_id, old_status, new_status, comment, timestamp |

## Phase 4: Build Iteratively

Follow this order strictly:
1. Analysis scripts and documentation → `ANALYSE.md`
2. Import parser with preview → validate before commit
3. Normalization layer (dates, amounts, text)
4. Match engine (exact → partial → fuzzy)
5. Simple web UI (dashboard → import → matching table)
6. Manual approve/reject + audit log
7. Reports and exports
8. Bonus calculation (if needed)
9. Tests + README + Docker

### Revisor/Auditor Package Pattern

When the user asks for output that can be sent to an accountant/revisor, produce a deterministic evidence package rather than relying on dashboard screenshots. First verify and remove duplicate import runs after a DB backup; then export a Markdown summary plus an Excel workbook with source files, month totals, open invoice/credit residuals, unexplained bank items, ignored bank items, payment advice rows, and raw normalized postings. Use explicit status language such as `AFSTEMT_MED_ÅBNE_POSTER` when the bank layer reconciles but invoice/credit-note residuals remain.

See `references/auditor-pack-and-dedup.md` for the duplicate-import cleanup sequence, auditor workbook sheet shape, status semantics, and verification checklist.

### Idempotent SourceFile Imports

For accounting import UIs, especially Coop `Payment_advice` bulk uploads, make uploads idempotent. Add a SHA256 `content_hash` on the source-file model, compute it before creating a new row, skip identical reuploads, and update/reparse an existing same-named source file when the content changed. Wrap child-row replacement in a database transaction and test both duplicate-skip and update-replace paths so repeated user uploads do not inflate reconciliation totals.

See `references/idempotent-source-file-imports.md` for the Django model/view/parser pattern, UI counters, bulk upload setting, and regression checklist.

## Danish-specific Conventions

- Dates: use ISO format internally (`YYYY-MM-DD`), but display in Danish
  (`dd-MM-yyyy`) per user preference. Parse ambiguous Danish text dates with day-first semantics (`dayfirst=True`).
- Amounts: always normalize to float/Decimal; display with 2 decimals, Danish comma.
- Fiscal year: calendar year (1/1 – 31/12) unless specified otherwise.
- For Coop workflows, model invoice year/month separately from payment year/month: Coop can have ~3 months of credit, so December invoices may be paid in March of the following fiscal year. Reports must not treat that as an error by default.
- Text: normalize by lowercasing, stripping extra spaces, removing punctuation before fuzzy comparison.
- Regulation/bonus/fee/fine classification should be explicit and reviewable. Start with keyword classification (`bonus`, `rabat`, `gebyr`, `bøde`, `regulering`, `korrektion`, `modregning`) but keep raw data and manual review because wording/layout varies.

## Django Accounting UI Display Formatting

When the reconciliation deliverable is a Django webapp, treat amount display as part of the reconciliation contract rather than a cosmetic detail. Use one shared template filter/localization utility for money columns in dashboards, match tables, invoice/payment status pages, and reports.

For Danish accounting UIs, monetary amounts should normally render with `.` thousands separators, `,` decimal separators, exactly two decimals, and a leading minus sign for negative values (for example `1.234.567,89`, `-1.234.567,89`, `0,00`). Prefer `Decimal`-based formatting over Python floats. Replace monetary `|floatformat:2` usage, but do not reformat percentages, match scores, counts, or raw imported/audit JSON unless explicitly requested.

Minimum verification for UI formatting changes:
- Add a unit/template test for the formatting helper.
- Add at least one rendered-page regression test proving formatted money appears and raw `1234567.89`-style output is absent.
- Confirm templates using the filter load its tag module, and restart the web container if needed.

See `references/django-accounting-ui-formatting.md` for the concrete Django template-filter implementation, search targets, Docker test commands, and pitfalls.

## References
- See `references/coop-uniconta-analysis.md` for the concrete discovery
  patterns from the Coop ↔ Uniconta project (header row locations,
  column mappings, SAP document type codes, etc.).
- See `references/coop-uniconta-match-engine.md` for the 5-phase match engine
  implementation, tolerance matching, net-based invoice status, and score formulas.
- See `references/bank-reconciliation.md` for three-source reconciliation extending
  Coop ↔ Uniconta with bank CSV — month-level validation, bonus reconciliation,
  and many-to-one (bank payment = sum of Coop invoices) matching strategy.
- See `references/coop-payment-advice.md` for using Coop betalingsadviseringer
  (PDF/XLSX settlement files) as the bridge between bank deposits and invoice
  lines, including parser patterns, coverage metrics, and reporting shape.
- See `references/three-way-django-analysis-engine.md` for Django implementation
  structure: separate analysis module, batch import command, three-way report
  outputs, interpretation pitfalls, and minimum tests.
- See `references/coop-invoice-payment-allocation.md` for the simplified forward
  workflow using Coop invoice lists + Uniconta payment allocation + Payment_advice,
  including separate models, invoice paid/open status logic, employee-expense
  exceptions, UI badges, and minimum tests.
- See `references/coop-uniconta-import-ui-and-payment-advice.md` for the five-source
  import UI pattern, flexible PDF/Excel Payment_advice raw import, monthly dashboard
  fields, 3-month Coop credit handling, and regression tests.
- See `references/auditor-pack-and-dedup.md` for revisor-ready exports, duplicate
  import cleanup after DB backup, status semantics such as `AFSTEMT_MED_ÅBNE_POSTER`,
  and Excel/Markdown verification.
- See `references/idempotent-source-file-imports.md` for SHA256-based idempotent
  accounting source-file uploads, update-vs-skip semantics, Django transaction
  boundaries, bulk upload limits, and regression tests.
- See `references/django-accounting-ui-formatting.md` for Django money-formatting
  implementation details, template search targets, regression tests, and pitfalls.