Data Handler: Control & Runs

How the IMS describes the pull server, how the pipeline builds it, and what each row in the runs table means.

Three Roles, One Record

The data handler is a record on the customer project's system in the Bilendo IMS. Three parties read and write it, each through its own door:

RoleDoorWhat it does
Bilendo employeeGET/POST/PUT …/data-handlersDescribes the desired state: schedule, company codes, objects, delivery method, SAP API base, retention — and the names of the three secrets. Never a value.
Deployment pipeline (Terraform)GET …/data-handlers/:id/desired-stateReads the desired state as JSON and creates the stack: EventBridge schedule, Step Functions, Lambdas, S3, DynamoDB. Then reports the status active.
Collector (Lambda, at runtime)GET …/data-handlers/:id/mapping · POST /api/data-handler-runsFetches the variant's field mapping at startup, transforms data with it, delivers to Bilendo, and reports each run back as a manifest.
flowchart LR
    subgraph IMS[Bilendo IMS]
        DH[(Data handler<br/>Desired state)]
        MAP[(Variant<br/>Field mapping)]
        RUNS[(Runs)]
    end
    PIPE[Terraform pipeline]
    subgraph AWS[AWS · Customer project]
        EB[EventBridge<br/>Schedule] --> SF[Step Functions]
        SF --> L[Lambda Collector / Transform / Deliver / Report]
        L --> S3[(S3 raw data)]
        L --> WM[(DynamoDB<br/>Watermark)]
        SM[(Secrets Manager)] -.Values.-> L
    end
    SAP[SAP BTP API Management<br/>→ S/4HANA Utilities FI-CA]
    BIL[Bilendo Loader<br/>API POST or SFTP]

    DH -- "desired-state (JSON)" --> PIPE
    PIPE -- "creates stack" --> AWS
    DH -. "Secret NAMES" .-> SM
    MAP -- "mapping" --> L
    L -- "OData, paginated, delta" --> SAP
    L -- "Delta batches ≤ 5,000 rows" --> BIL
    L -- "Manifest per run" --> RUNS

The IMS describes and observes; it does not operate. It holds no access to SAP or Bilendo, only the name under which the access credentials are stored in the customer's secret store. That's why the desired state is safe to read and can control a pipeline in a third-party account.

The Data Handler Lifecycle

The data handler's status answers whether the stack exists and whether it is healthy. Nobody sets two of the transitions by hand: a run with failed sets error, and the next successful run resets the status from error back to active. Pausing is a human decision and is never overwritten by a run.

stateDiagram-v2
    [*] --> draft : described in the IMS
    draft --> provisioning : pipeline reads desired state
    provisioning --> active : stack is up
    active --> error : run reports failed
    error --> active : run reports delivered
    active --> paused : manually
    error --> paused : manually
    paused --> active : manually
StatusMeaning
draftDescribed, but nothing built yet. The target state is complete once the SAP API base, delivery route, and the three secret names are set.
provisioningThe pipeline is working. The schedule is not yet active.
activeThe stack runs on schedule; the last reported run was successful.
pausedSchedule turned off, stack remains. Intended for maintenance windows in SAP or Bilendo.
errorThe last run failed. The watermarks stay in place, and the window is delivered again on the next run.

One run, step by step

sequenceDiagram
    autonumber
    participant EB as EventBridge
    participant L as Lambda
    participant IMS as Bilendo IMS
    participant SAP as SAP BTP / FI-CA
    participant BIL as Bilendo Loader

    EB->>L: Schedule fires (run_id assigned)
    L->>IMS: GET mapping (field mapping of the variant)
    L->>L: Read watermark per object
    loop per object: debtors → contract accounts → items → payments → installments
        L->>SAP: OData $filter ≥ watermark − overlap, $top/$skip
        SAP-->>L: Page n
        L->>L: Raw data to S3, transform, clearing rules
        L->>BIL: Batch (≤ 5,000 rows, idempotency key)
        BIL-->>L: accepted / rejected per row
    end
    alt all batches confirmed
        L->>L: Advance watermark to start time
        L->>IMS: POST /data-handler-runs {status: delivered, received, stored, rejected}
    else one batch fails
        L->>IMS: POST /data-handler-runs {status: failed, error}
        Note over L: Watermarks stay in place
    end

The order of the objects is fixed: an item needs its debtor, an installment its contract. A run reports once per object — so the run table in the IMS shows multiple rows with the same run ID for a single run.

What the rows in the run table tell you

Each row is a manifest that the collector reported after one object of a run. A second report with the same run ID and the same object overwrites the first (this is how running becomes delivered or failed at the end).

ColumnMeaningWhat to look for
RunThe collector's run ID plus start time.Rows with the same ID belong to one schedule event.
ObjectWhat was delivered: debtors, contract accounts, open items, payments, installment plans.If an object from the target state is missing, the run did not reach it.
Windowwatermark_from → watermark_to: the delta window that was read (posting date or clearing date, respectively).A window that is larger than the interval between two runs means the previous run failed and was caught up.
Readreceived: rows that SAP delivered.0 on a business day is suspicious — check the delta filter or authorization.
Storedstored: rows that Bilendo accepted.Should be received minus rejections.
Rejectedrejected: rejected by Bilendo (required field empty, unknown debtor) · rejected_local: discarded before delivery (no mapping, rule applies).Local rejections point to the mapping in the IMS; Bilendo rejections point to the data in SAP.
Statusrunning · delivered · failed — with error text.failed turns the data handler error; the error text appears in the row.

A healthy day looks like this: for each schedule event, five rows with the same run ID, a window from the last execution to this one, "rejected" at 0 or a small, explainable number, and status "delivered."

What the IMS deliberately does not do