Radix-Skills/skills/data-science/erp-data-reconciliation/SKILL.md

name, description, trigger

name	description	trigger
erp-data-reconciliation	Analyze and reconcile financial/accounting data between two ERP systems (e.g. SAP/Coop and Uniconta) using Excel exports. Covers file inspection, column mapping, Danish-format normalization, match-engine design, and iterative build workflow.	User asks to reconcile, match, compare, or afstemme data between two accounting/ERP systems, typically via Excel files. Also relevant when importing/exporting financial ledgers, kontoudtog, vendor/customer statements, or bonus calculations across systems.

ERP Data Reconciliation Skill

Overview

Reconciling data between two ERP/accounting systems is a multi-phase task. The typical deliverable is a webapp or script that can:

1. Import Excel files from both systems
2. Normalize dates, amounts, and text
3. Match postings between systems
4. Show unmatched, partial matches, and differences
5. Support manual approval/rejection
6. Generate reports

Phase 1: File Analysis (CRITICAL — do not skip)

Before writing any import code, run a structured analysis of the Excel files.

Steps

1. List all files in both source directories
2. For each file, identify:
   • Sheet names
   • Actual header row location (may NOT be row 1!)
   • Column names and data types
   • Date columns and their formats
   • Amount columns and their formats
   • Text columns
   • Reference/invoice/document columns
   • Null percentages
   • Duplicate rows
   • Data range (min/max dates)

Key Pitfalls

• Header row is NOT always row 1. Uniconta exports often have account info in rows 1-2, blank row 3, headers in row 4.
• Amounts mixed formats. Danish Excel files often have amounts as both Excel floats AND as Danish text strings like -2.543.803,93. Always normalize using a robust parser:

  def parse_danish_number(val):
      if pd.isna(val): return float('nan')
      if isinstance(val, (int, float)): return float(val)
      s = str(val).strip()
      if not s: return float('nan')
      if '.' in s and ',' in s:
          s = s.replace('.', '').replace(',', '.')
      elif ',' in s: s = s.replace(',', '.')
      return float(s)
  

• Invoice/reference numbers may NOT match between systems. This is the #1 cause of failed reconciliation. Always check overlap BEFORE assuming IDs can be used as join keys. Use amount overlap as a quick viability check.

Analysis Script Template

Use or adapt scripts/analyze_excel.py from the Coop-Uniconta project:

• Reads all sheets in all files
• Prints column stats, null counts, sample values
• Detects Danish number formats
• Identifies empty/duplicate rows

Phase 2: Determine Match Strategy

After analysis, document the match strategy BEFORE coding the engine.

Decision Tree

1. Do invoice/reference numbers overlap?
   • If yes → use Faktura + abs(Amount) + Date as PRIMARY signal (two-phase). First phase: Reference==Faktura + abs(Amount) within tolerance (±5 DKK) + Date±21d. Second phase: abs(Amount) + Date±21d for remaining rows.
   • If no → use Amount + Date + Text similarity
2. Are dates exact or within tolerance?
   • Same day → exact
   • ±1-3 days → high confidence partial
   • ±7 days → medium confidence
   • ±21 days → general match tolerance

   • 21 days → fuzzy/manual only

3. Is text available on both sides?
   • Both have text → Levenshtein/rapidfuzz similarity score
   • Only one side has text → amount+date only
4. Do signs differ?
   • Same sign → same direction transaction
   • Opposite sign → same transaction viewed from creditor vs debtor side. Match on abs(amount) but label "MODSAT" in output.
     Example: Coop RE (payment) is negative; Uniconta Faktura is positive.

Tolerance Matching (V6 innovation)

Instead of exact amount match (±0.01), use a tolerance (±5 DKK):

• Catches rounding differences between systems
• Catches partial payments (one system may record 440.000, the other 430.732)
• Uses bucket-based indexing for fast lookup:

  AMT_TOL = 5.0
  bucket = round(abs_amount / AMT_TOL) * AMT_TOL
  # Search adjacent buckets: [bucket-AMT_TOL, bucket, bucket+AMT_TOL]
  

Coop Fakturaliste + Uniconta Betalingsfordeling as the Simple Forward Workflow

When the user wants the process to be as simple as possible, prefer a forward-looking workflow based on:

1. Payment_advice
2. latest Uniconta Coop debtor/payment allocation export
3. the complete Coop invoice list

Model the invoice list and payment allocation as separate import sources. Use invoice number as the primary join between Coop invoices and Uniconta allocation lines, and calculate paid/open/partial status from payment/settlement lines rather than only from original invoice postings. Treat the payment allocation as an explanation layer between a bank deposit and many invoice/credit lines.

See references/coop-invoice-payment-allocation.md for the concrete Django model/import/status pattern, UI shape, and tests.

Coop Payment Advice / Settlement Files as a Fourth Source

When the user provides Coop betalingsadviseringer/opgørelser for individual bank payments, treat them as a separate reconciliation source, not just attachments. They are often the missing bridge:

Bank deposit → Coop payment advice total → advice invoice/credit lines → Coop ledger / Uniconta invoices

For Coop-Uniconta style systems, the import UI should expose the full operational set, not just "Coop" and "Uniconta":

1. Uniconta Coop Debitor / payment allocation
2. Uniconta/ERP invoice list: all Coop invoices/credit notes for the year
3. Coop account statement / kontoudtog
4. Bank statement containing all Coop postings
5. Coop Payment_advice documents, uploadable continuously as they are collected

Recommended source/model names:

• coop_settlement or coop_payment_advice
• PaymentAdvice / PaymentAdviceDocument / SettlementFile: file name, unique id/bilag number, file type (PDF/XLSX), document date, payment date, total amount, period, parse status, matched bank transaction, raw text/raw JSON.
• PaymentAdviceLine / SettlementLine: invoice number, invoice date, company code, amount, currency, line type (invoice/credit/bonus/modregning/unknown), links to bank/Coop/Uniconta transactions, match status, explanation.

If the user does not yet have all payment advice documents, still allow upload now. Store raw PDF text or Excel row JSON with best-effort date/amount/reference/classification so later parsers can be tightened without re-requesting files.

Parser patterns observed:

• XLSX files may have header row 3 with columns Fakturanummer, Fakturadato, Firmakode, Beløb, Valuta and supplier metadata in rows 1-2.
• Text PDFs often extract as repeated triples: invoice number line, dd.mm.yyyy firmakode line, amount/DKK line.
• Some PDFs split sign into a separate - column/line; treat 123,45, -, DKK as -123,45.
• Some PDFs are bank statement screenshots, not payment advice. Classify and do not parse them as settlement lines.
• Some files include manual middle calculations or special layouts. Keep raw text and mark parse confidence/status instead of silently forcing a schema.

Matching/reporting logic:

1. Match payment-advice total to bank deposit using ±5 DKK tolerance.
2. Then match advice lines to Uniconta by invoice number first, then amount/date.
3. Match advice lines to Coop ledger by reference/doc number; in the Coop-Uniconta project this was a very strong signal.
4. Treat payment-advice matches as an evidence/explanation layer, not as automatic ledger settlement. An invoice can remain open in Uniconta while Payment_advice proves Coop included it in a settlement, credit, bonus, modregning, or manual middle calculation.
5. Report per bank deposit/open item: bank date/amount, matched advice file, difference, line count, Uniconta hit count, Coop hit count, credit/bonus/ modregning total, advice documents/vouchers/classifications, and status OK / partial / needs review.
6. Keep 2026 or out-of-period advice files in the database, but exclude or label them separately when current bank/Uniconta data only covers 2024-2025.

Invoice Status: Net-based + Deduplication

Critical pitfall: The same payment may exist in BOTH systems (Coop RE and Uniconta Betaling). If you sum both, you get double the actual paid amount.

Solution: Calculate net payments per invoice:

# Net Coop payments (NOT abs-sum!)
coop_net = sum(c['amount'] for c in coop 
               if c['reference'] == fnr and c['doc_type'] in ('RE','ZP'))
coop_paid = abs(coop_net) if coop_net < 0 else 0

# Net Uniconta payments
uni_net = sum(u['amount'] for u in uni 
            if u['faktura'] == fnr and u['konteringstype'] == 'Betaling')
uni_paid = abs(uni_net) if uni_net < 0 else 0

# Deduplicate: if both exist, use only one
if coop_paid > 0 and uni_paid > 0:
    if abs(coop_paid - uni_paid) <= AMT_TOL:
        betalt = coop_paid  # Same payment recorded twice
    else:
        betalt = max(coop_paid, uni_paid)  # Partial, take largest

Also handle: Positive Coop RE amounts are credit memos (modregninger) that should be SUBTRACTED from the net, not added with abs().

Match Scoring Formula (default)

Score = 0
# Amount match (40%)
if abs(a-b) < AMT_TOL: Score += 40

# Date match (35%)
days = abs(d1 - d2).days
if days == 0: Score += 35
elif days <= 3: Score += 25
elif days <= 7: Score += 15
elif days <= 14: Score += 5

# Text match (25%)
if both_have_text:
    similarity = fuzzy_ratio(text1, text2)  # 0-100
    if similarity > 80: Score += 25
    elif similarity > 50: Score += 12

# Reference bonus (extra)
if reference_match: Score += 40

# Thresholds
if Score >= 85: exact_match, auto_approvable
if Score >= 60: partial_match, requires_manual
if Score >= 40: fuzzy_match, requires_manual

Table	Key Fields
ImportBatch	source (coop/uniconta), filename, imported_at, raw_row_count
Posting	batch_id, source, fiscal_year, posting_date, amount, original_text, normalized_text, reference, document_number, original_data (JSON)
Match	posting_a_id, posting_b_id, score, match_type (exact/partial/fuzzy/manual), status (proposed/approved/rejected), matched_by (auto/user), matched_at
AuditLog	user, action, match_id, old_status, new_status, comment, timestamp

Phase 4: Build Iteratively

Follow this order strictly:

1. Analysis scripts and documentation → ANALYSE.md
2. Import parser with preview → validate before commit
3. Normalization layer (dates, amounts, text)
4. Match engine (exact → partial → fuzzy)
5. Simple web UI (dashboard → import → matching table)
6. Manual approve/reject + audit log
7. Reports and exports
8. Bonus calculation (if needed)
9. Tests + README + Docker

Revisor/Auditor Package Pattern

When the user asks for output that can be sent to an accountant/revisor, produce a deterministic evidence package rather than relying on dashboard screenshots. First verify and remove duplicate import runs after a DB backup; then export a Markdown summary plus an Excel workbook with source files, month totals, open invoice/credit residuals, unexplained bank items, ignored bank items, payment advice rows, and raw normalized postings. Use explicit status language such as AFSTEMT_MED_ÅBNE_POSTER when the bank layer reconciles but invoice/credit-note residuals remain.

See references/auditor-pack-and-dedup.md for the duplicate-import cleanup sequence, auditor workbook sheet shape, status semantics, and verification checklist.

Idempotent SourceFile Imports

For accounting import UIs, especially Coop Payment_advice bulk uploads, make uploads idempotent. Add a SHA256 content_hash on the source-file model, compute it before creating a new row, skip identical reuploads, and update/reparse an existing same-named source file when the content changed. Wrap child-row replacement in a database transaction and test both duplicate-skip and update-replace paths so repeated user uploads do not inflate reconciliation totals.

See references/idempotent-source-file-imports.md for the Django model/view/parser pattern, UI counters, bulk upload setting, and regression checklist.

Danish-specific Conventions

• Dates: use ISO format internally (YYYY-MM-DD), but display in Danish (dd-MM-yyyy) per user preference. Parse ambiguous Danish text dates with day-first semantics (dayfirst=True).
• Amounts: always normalize to float/Decimal; display with 2 decimals, Danish comma.
• Fiscal year: calendar year (1/1 – 31/12) unless specified otherwise.
• For Coop workflows, model invoice year/month separately from payment year/month: Coop can have ~3 months of credit, so December invoices may be paid in March of the following fiscal year. Reports must not treat that as an error by default.
• Text: normalize by lowercasing, stripping extra spaces, removing punctuation before fuzzy comparison.
• Regulation/bonus/fee/fine classification should be explicit and reviewable. Start with keyword classification (bonus, rabat, gebyr, bøde, regulering, korrektion, modregning) but keep raw data and manual review because wording/layout varies.

Django Accounting UI Display Formatting

When the reconciliation deliverable is a Django webapp, treat amount display as part of the reconciliation contract rather than a cosmetic detail. Use one shared template filter/localization utility for money columns in dashboards, match tables, invoice/payment status pages, and reports.

For Danish accounting UIs, monetary amounts should normally render with . thousands separators, , decimal separators, exactly two decimals, and a leading minus sign for negative values (for example 1.234.567,89, -1.234.567,89, 0,00). Prefer Decimal-based formatting over Python floats. Replace monetary |floatformat:2 usage, but do not reformat percentages, match scores, counts, or raw imported/audit JSON unless explicitly requested.

Minimum verification for UI formatting changes:

• Add a unit/template test for the formatting helper.
• Add at least one rendered-page regression test proving formatted money appears and raw 1234567.89-style output is absent.
• Confirm templates using the filter load its tag module, and restart the web container if needed.

See references/django-accounting-ui-formatting.md for the concrete Django template-filter implementation, search targets, Docker test commands, and pitfalls.

References

• See references/coop-uniconta-analysis.md for the concrete discovery patterns from the Coop ↔ Uniconta project (header row locations, column mappings, SAP document type codes, etc.).
• See references/coop-uniconta-match-engine.md for the 5-phase match engine implementation, tolerance matching, net-based invoice status, and score formulas.
• See references/bank-reconciliation.md for three-source reconciliation extending Coop ↔ Uniconta with bank CSV — month-level validation, bonus reconciliation, and many-to-one (bank payment = sum of Coop invoices) matching strategy.
• See references/coop-payment-advice.md for using Coop betalingsadviseringer (PDF/XLSX settlement files) as the bridge between bank deposits and invoice lines, including parser patterns, coverage metrics, and reporting shape.
• See references/three-way-django-analysis-engine.md for Django implementation structure: separate analysis module, batch import command, three-way report outputs, interpretation pitfalls, and minimum tests.
• See references/coop-invoice-payment-allocation.md for the simplified forward workflow using Coop invoice lists + Uniconta payment allocation + Payment_advice, including separate models, invoice paid/open status logic, employee-expense exceptions, UI badges, and minimum tests.
• See references/coop-uniconta-import-ui-and-payment-advice.md for the five-source import UI pattern, flexible PDF/Excel Payment_advice raw import, monthly dashboard fields, 3-month Coop credit handling, and regression tests.
• See references/auditor-pack-and-dedup.md for revisor-ready exports, duplicate import cleanup after DB backup, status semantics such as AFSTEMT_MED_ÅBNE_POSTER, and Excel/Markdown verification.
• See references/idempotent-source-file-imports.md for SHA256-based idempotent accounting source-file uploads, update-vs-skip semantics, Django transaction boundaries, bulk upload limits, and regression tests.
• See references/django-accounting-ui-formatting.md for Django money-formatting implementation details, template search targets, regression tests, and pitfalls.