DATA INTEGRATION

The Kafka Connect Failure Hidden Behind a Green Process

A connector story about offsets, poisoned records and the difference between process health and data movement.

The worker answered its health check, yet customer updates had stopped reaching the downstream system forty minutes earlier. The failed unit was not the process; it was one task, retrying the same malformed record.

Start with one workload | Scale into a complete platform | Open Source Freedom | Enterprise-Grade Operations.

Published | April 2026

The moment the service stopped being simple

The worker answered its health check, yet customer updates had stopped reaching the downstream system forty minutes earlier. The failed unit was not the process; it was one task, retrying the same malformed record.

A restart looked tempting. It also risked hiding the offset position, replaying side effects and losing the evidence needed to explain which records had crossed the boundary.

The turning point

The team treated connector configuration, plugins, secrets, offsets, task state and the dead-letter path as one managed service. Recovery began with the data contract, not the restart button.

The team used OSS Manager to make managed distributed connector runtime reviewable and repeatable. The engine still owned its native behaviour; the control plane made intent, prerequisites, execution and evidence visible to the people responsible for the service.

What had to become explicit

• The service boundary: A distributed Kafka Connect worker group uses dedicated internal topics, approved plugins, protected connector secrets, monitoring, and change-controlled connector configuration.
• The operational controls: plugin allow-list, internal-topic replication, config validation, secret providers, task restart policy, offset preservation, schema compatibility, and backpressure controls.
• The capacity conversation: connector count, task count, record rate, transformation cost, payload size, source or sink latency, retry volume, and worker failure headroom.
• The human boundary: who may observe, who may approve, who may execute, and who decides whether the application is ready.
  

The architecture that changed the conversation

The diagram is deliberately centred on the decision the team had to make. It is not a product inventory. It shows where authority sits, what crosses the boundary, and where a failed assumption must stop the workflow.

The practical design choices

• Poisoned record. Bad reflex: Restart workers. Safer response: Quarantine with context.
• Sink pressure. Bad reflex: Increase retries. Safer response: Control backpressure.
• Plugin drift. Bad reflex: Copy a new JAR. Safer response: Validate approved artifact.
• Offset uncertainty. Bad reflex: Reset to latest. Safer response: Capture position and choose replay boundary.
• Schema break. Bad reflex: Patch the task. Safer response: Version the contract and prove compatibility.

These choices are intentionally small enough to review and test. They keep the architecture tied to operating behaviour instead of allowing a visually impressive diagram to hide unclear ownership.

What OSS Manager changed - and what it did not

OSS Manager brought discovery, planning, guarded execution and normalised status into one path for Kafka Connect. It did not replace the engine's correctness model or the application's responsibility for data semantics.

• Before change, validate versions, hosts, identities, artifacts, topology and the recovery boundary without mutating the service.
• During change, persist progress, expose stop conditions and prevent a partial result from being mistaken for success.
• After change, observe worker health; connector and task state; records in and out; error rate; retry queue; dead-letter topic; source lag; sink latency; rebalance time and run representative application journeys, not only process checks.
• For recovery, protect the last known-good authority and require an explicit decision before promotion, rollback or destructive cleanup.

The failure modes worth rehearsing

The test plan should make room for missing plugin, incompatible connector version, poisoned record, secret resolution failure, task crash loop, source overload, sink backpressure, and offset loss. The purpose is not to produce a longer checklist. It is to learn whether operators can recognise the failure and choose the safe next action while the evidence is incomplete.

A human operating model

L1 operators need a plain-language answer to what is healthy, what is delayed and whether a change is in progress. Platform engineers need topology and evidence. Application owners need to know what users will experience. A useful platform connects those views without giving every person the same privileges.

Where this pattern earns its place

Media and OTT

This pattern is relevant where teams need regional continuity, burst traffic, low-latency serving and predictable recovery. The architecture must still be adjusted for local data classification, recovery objectives, workload shape and application behaviour.

Retail

This pattern is relevant where teams need campaign peaks, catalogue freshness, transaction continuity and rapid rollback. The architecture must still be adjusted for local data classification, recovery objectives, workload shape and application behaviour.

Telecom

This pattern is relevant where teams need high event volume, distributed operations, identity boundaries and service assurance. The architecture must still be adjusted for local data classification, recovery objectives, workload shape and application behaviour.

The lesson we would carry into the next project

The most useful status view was the one that told an L1 operator what had stopped, where the record was parked, and which action was safe without exposing credentials.

The strongest open-source platforms are not the ones with the most automation. They are the ones where automation makes ownership, risk and recovery easier for people to understand.