EDA for the Rest of Us: Operational Challenges

This is the final post in the “EDA for the Rest of Us” series. We’ve covered Event Design Patterns, Communication Patterns, and Consumption Patterns. Now for the critical part: evolving your architecture as your business grows.

You’ve built a successful event-driven system. It’s processing millions of events, teams are moving fast, and the architecture is holding up beautifully. But businesses don’t stand still.

New products launch. Customer behavior shifts. That simple OrderPlaced event now needs to handle subscriptions, pre-orders, and multi-currency transactions. The payment team wants to add fraud scoring. The fulfillment team needs split shipments. And somehow, you need to evolve your events without breaking the 47 services that depend on them.

Welcome to the operational challenges of EDA - the practices that let your event-driven architecture grow with your business instead of becoming a legacy liability.

Table of Contents

  1. Growing Pains
  2. Managing Event Schemas Without Breaking Everything
    1. The Schema Evolution Challenge
    2. Schema Registry: Your Safety Net
    3. Practical Schema Versioning
  3. Building Event Catalogs People Actually Use
    1. Discovery That Works
    2. Ownership at Scale
  4. Monitoring Event Flows
  5. Debugging Distributed Event Processing
  6. Implementing Replay for Testing and Recovery
  7. Working Together: An Integrated Approach
  8. The Path to Operational Maturity
  9. References

Growing Pains

Building event-driven systems is just the beginning, the real challenge emerges as your system grows and evolves:

  • Business requirements change and your events need new fields without breaking existing consumers
  • Teams proliferate and need to discover and understand events across the organization
  • Services multiply and ownership becomes unclear as the original architects move on
  • Processing patterns that worked at 1K events/second break at 100K events/second
  • New regulations require audit trails and replay capabilities you didn’t plan for
  • What started as a simple event becomes mission-critical for dozens of services

These aren’t failures of your original design - they’re natural consequences of success. Let’s look at how to address these challenges gracefully.

Managing Event Schemas Without Breaking Everything

The Schema Evolution Challenge

Change is constant in a successful business, the question is not how to prevent it, but how to manage it gracefully. The OrderPlaced event that started with 5 fields now needs 15 to support new features. Some consumers need all that data, others are happy with the original fields. You need to add subscription support, but half your consumers will break if they see an unexpected order type.

This is what success looks like! Your event-driven architecture is enabling rapid feature development. The challenge is evolving schemas to support new capabilities while maintaining stability for existing consumers.

The core challenge is that event schemas need to evolve, but in distributed systems, you can’t coordinate all consumers to update simultaneously. With events, there’s an additional twist: consumers might be processing historical events hours or days later, so multiple schema versions need to coexist in the stream at the same time.

Schema Registry: Your Safety Net

A schema registry acts as the source of truth for all event schemas in your system. But here’s the key: it’s not just a storage system, it’s an enforcement mechanism.

Think of it as a bouncer at the event publishing door. Before any event goes out, it checks: “Is this schema registered? Does this event match the schema? Are you making breaking changes?” This catches problems at publish time, not at 3 AM when consumers start failing.

AWS offers a couple of options here. EventBridge Schema Registry works if you’re all-in on EventBridge, but it’s limited to JSON Schema and tightly coupled to EventBridge itself. For more flexibility, AWS Glue Schema Registry supports multiple formats (Avro, JSON Schema, Protobuf) and works across different streaming services like Kinesis and MSK. Many teams also run their own schema registry using tools like Confluent Schema Registry.

The key isn’t which registry you use - it’s that you use one. Too many teams treat schema registration as optional documentation. Make it mandatory. Every event type needs a schema. Every publish validates against that schema. No exceptions.

Practical Schema Versioning

The textbook says “use semantic versioning.” Reality says “our team doesn’t know what constitutes a breaking change.” Here’s what actually works:

  • Automated compatibility checking: Don’t rely on developers to know what’s breaking. Build tooling that compares schemas and tells you: “This change removes a required field - that’s breaking” or “This adds an optional field - that’s safe.”

  • Multi-version support: You can’t force all consumers to upgrade simultaneously. Support multiple schema versions in parallel. Tag events with their schema version. Route v1 events to v1 consumers, v2 to v2. This gives teams time to migrate.

  • Deprecation windows: When you must make breaking changes, announce them, provide migration guides, and set deprecation dates. Monitor which consumers are still using old versions. Reach out to teams proactively.

graph LR
    P[Order Service v2] -->|OrderPlaced v2| EB[EventBridge]
    EB -->|v2 events| C1[Payment Service<br/>supports v1 & v2]
    EB -->|v2 events| C2[Inventory Service<br/>supports v2 only]
    EB -->|v2→v1 adapter| C3[Legacy Analytics<br/>supports v1 only]

    style P fill:#e1f5e1
    style C3 fill:#ffe1e1

Building Event Catalogs People Actually Use

Discovery That Works

Without event discovery, teams face a common dilemma: they either spend hours hunting through documentation trying to find if an event already exists, or they give up and create their own - leading to duplicate events, inconsistent naming, and point-to-point integrations that defeat the purpose of EDA.

Your event catalog needs to answer key questions: What events exist? What do they contain? Who publishes them? Who consumes them? How often are they published? What do they look like in practice?

A static wiki page isn’t a catalog - it’s where event documentation goes to die. But the solution isn’t to build a complex auto-discovery system that nobody will maintain. Focus on the essentials:

  • Clear documentation structure: Each event should have a consistent format - what it represents, when it’s published, example payloads (not from production!), and schema definition. Make it easy to add new events.

  • Domain organization: Group events by business domain, not by technical service. The “Order” domain might include events from multiple services, but they’re logically related.

  • Version history: Track schema changes over time. When did we add that field? Why? Who approved it? This history becomes invaluable during debugging.

  • Example payloads: Provide realistic but synthetic examples. Show edge cases and optional fields. Developers need to understand the data shape without exposing production data.

  • Simple search: Let teams find events by name, domain, or publishing service. Nothing fancy - just make discovery possible.

Tools like EventCatalog provide a purpose-built solution for this. It’s an open-source documentation tool specifically designed for event-driven architectures, offering features like event versioning, domain organization, and visual flow diagrams. Remember: a manually maintained catalog that’s actually updated is infinitely more valuable than an auto-generated one that’s always out of date.

Ownership at Scale

The most critical information in your catalog? Ownership. “Who owns the OrderPlaced event?” Without clear ownership, events become orphans. Nobody maintains them, nobody fixes them, and eventually, nobody trusts them.

Your catalog needs to capture ownership that goes beyond a name in a spreadsheet:

  • Who owns the schema
  • Which team maintains the publisher
  • Who are the consumers
  • Which domain the event belongs to
  • How to contact the owning team (Slack channel, email, runbook)

Align event ownership with business domains. The team that owns the order domain owns all order-related events. This creates clear boundaries and makes ownership sustainable as your organization grows. Tools like EventCatalog make this easier by treating ownership as a first-class concept, not an afterthought. Every event has an owner, every owner has contact information, and every schema change has an approval trail.

Monitoring Event Flows

CloudWatch tells you events are flowing. It doesn’t tell you if they’re flowing correctly. There’s a difference between “EventBridge processed 1M events” and “customers received their order confirmations.”

Stop monitoring technical metrics and start monitoring business outcomes:

  • End-to-end transaction tracking: An order should trigger OrderPlaced → PaymentProcessed → InventoryReserved → OrderShipped. Monitor the complete flow, not individual hops.
  • Business SLAs, not technical SLAs: “99% of orders complete within 5 minutes” matters more than “EventBridge latency p99 < 100ms.”
  • Anomaly detection on business patterns: Alert when order-to-shipment time degrades, not when queue depth increases. The queue might be fine - the business process might be broken.

Consumer lag is critical, but raw numbers don’t tell the story. 1000 messages behind means different things for different consumers:

  • Order confirmations: Customer-facing impact, every minute matters
  • Inventory sync: Accuracy impact, lag creates oversell risk
  • Analytics pipeline: Delayed dashboards, usually not critical

Map consumers to business functions. Monitor and alert based on business impact, not just message counts.

sequenceDiagram
    participant C as Customer
    participant OS as Order Service
    participant PS as Payment Service
    participant IS as Inventory Service
    participant SS as Shipping Service

    C->>OS: Place Order
    OS->>OS: OrderPlaced event
    OS-->>PS:
    OS-->>IS:
    PS->>PS: PaymentProcessed event
    IS->>IS: InventoryReserved event
    PS-->>SS:
    IS-->>SS:
    SS->>SS: OrderShipped event

    Note over OS,SS: Monitor complete flow, not individual hops

Debugging Distributed Event Processing

Traditional distributed tracing assumes synchronous calls. In event-driven systems, the “request” might complete hours before the “response” starts processing. Your trace needs to survive this temporal disconnect.

Every event needs trace context that survives the journey through queues, streams, and storage. But here’s the catch: most tracing systems expect synchronous handoffs. You need to:

  • Embed trace context in event payloads, not just headers
  • Generate new spans when events are consumed
  • Correlate spans across time using business correlation IDs
  • Handle replay scenarios where events are reprocessed
graph LR
    subgraph "Synchronous Call"
        A[Service A<br/>TraceID: 123] -->|HTTP + Headers| B[Service B<br/>TraceID: 123]
    end

    subgraph "Async Event Flow"
        C[Producer<br/>TraceID: 456] -->|Event + Context in Payload| Q[Queue/Stream]
        Q -.->|Hours Later| D[Consumer<br/>TraceID: 456]
    end

    style Q fill:#ffe5b4

When debugging, you need to reconstruct what happened across time and services:

  1. Find all events with a correlation ID
  2. Query all services that processed those events
  3. Build a timeline showing the flow
  4. Identify gaps where events went missing
  5. Spot bottlenecks where processing stalled

This is harder than it sounds. Events might be processed out of order. Services might process events multiple times. Failed events might be retried hours later. Your tracing needs to handle this chaos.

Implementing Replay for Testing and Recovery

“It worked in staging!” Then production happens. Event replay is your time machine - the ability to take production events and replay them in a safe environment to debug issues, test fixes, or recover from failures.

The power of replay comes with risks. Replaying payment events could charge customers twice. Replaying order events could duplicate shipments. You need safety controls:

  • Isolated environments: Replay in sandboxes that can’t affect production
  • Service mocking: External services should be mocked during replay
  • Output suppression: Don’t send emails, SMS, or push notifications
  • State management: Decide whether to preserve or reset state between replays
graph TD
    subgraph "Production"
        PE[EventBridge] -->|Archive| EA[EventBridge Archive]
        PE --> KDS[Kinesis Data Streams]
        KDS -->|Firehose| S3[S3 Event Lake]
    end

    subgraph "Replay Environment"
        EA -->|Replay API| RE[Replay EventBridge]
        S3 -->|Lambda Replay| RE
        RE --> TL[Test Lambda Functions]
        TL --> TDB[(Test DynamoDB)]
        TL -.->|Blocked| Ext[External APIs ❌]
    end

    style PE fill:#ff9900
    style RE fill:#232f3e,color:#fff

Beyond debugging, replay enables powerful testing:

  • Regression testing: Replay last week’s events through your new code
  • Load testing: Replay Black Friday traffic to test scalability
  • Chaos testing: Replay events with simulated failures
  • Migration validation: Ensure new services handle historical events

The key is making replay a first-class operational capability, not an afterthought.

Working Together: An Integrated Approach

These operational capabilities work together as a system:

graph TD
    Issue[Production Issue] --> Cat{Event<br/>Catalog}
    Cat -->|Find Owner| Team[Alert Team]
    Cat -->|Event Details| Context[Schema & Consumers]

    Team --> Diag[Diagnostics]
    Context --> Diag

    Diag --> SR[Check Schema<br/>Registry]
    Diag --> Mon[Business<br/>Monitoring]
    Diag --> Trace[Distributed<br/>Tracing]

    SR --> Root[Root Cause]
    Mon --> Root
    Trace --> Root

    Root --> Replay[Test Fix<br/>via Replay]
    Replay --> Deploy[Deploy with<br/>Confidence]

    style Issue fill:#ffebee
    style Deploy fill:#e8f5e9
  1. Schema Registry prevents breaking changes from reaching production
  2. Event Catalog helps teams discover events and find owners when issues arise
  3. Business Monitoring tells you when events are actually broken (not just slow)
  4. Distributed Tracing helps you figure out why they’re broken
  5. Event Replay lets you safely test your fixes

Here’s a real-world scenario: A schema change breaks production. The monitoring alerts the team. They use the catalog to identify the event owner and affected consumers. Distributed tracing shows where events are failing. They test the fix using replay in a sandbox. Once verified, they deploy with confidence.

Without these operational capabilities, you’re flying blind. With them, you can run event-driven systems with confidence.

The Path to Operational Maturity

After four posts in this series, here’s what I hope you take away:

Start with the basics, but plan for operational maturity from day one. You don’t need to solve all these challenges immediately, but you need to know they’re coming. That schema registry you’re postponing? You’ll need it the first time a schema change breaks production.

Build incrementally:

  1. Basic monitoring and alerting
  2. Schema registry and versioning
  3. Event catalog and discovery
  4. Distributed tracing and replay

Measure what matters to the business, not just what’s easy to measure. “Events processed per second” is less important than “orders completed without errors.”

Invest in debugging capabilities early. The time to build replay infrastructure isn’t during a production incident.

Event-driven architecture promises loose coupling and scalability. Understanding and addressing these operational challenges is what lets you deliver on that promise while adapting to changing business needs. Without proper operational practices, your flexible architecture calcifies into a distributed monolith that nobody dares to change.

Remember: successful architectures evolve. The patterns in this series - from event design through communication, consumption, and operations - work together to create systems that can grow and adapt while maintaining stability.

Start simple. Address challenges as they arise. But always, always build for evolution.

Good luck out there. May your events flow smoothly, your schemas evolve gracefully, and your architecture grow stronger with each change.

What operational challenges have you faced with event-driven systems? How did you solve them? Share your war stories in the comments below.

References

Operational Practices

  1. EventCatalog - Open Source Event Documentation - Purpose-built tool for documenting event-driven architectures
  2. AsyncAPI Specification - Standard for documenting async APIs
  3. AWS Glue Schema Registry - Multi-format schema management for AWS
  4. EventBridge Schema Registry - Native schema discovery for EventBridge
  5. OpenTelemetry for Event-Driven Systems - Semantic conventions for async messaging
  6. Distributed Tracing with OpenTelemetry - Vendor-neutral tracing

Event Replay and Recovery

  1. EventBridge Archive and Replay - Built-in replay capabilities
  2. Dead Letter Queue Pattern - Essential error handling pattern

Schema Evolution and Versioning

  1. Schema Evolution in Event-Driven Systems - Compatibility strategies
  2. CloudEvents - Specification for event data standardization

Monitoring and Debugging

  1. OpenTelemetry Context Propagation - Maintaining trace context across async boundaries
  2. Distributed Tracing in Event-Driven Systems - Challenges and solutions
  3. OpenTelemetry Collector - Vendor-agnostic telemetry pipeline

Core Patterns and Best Practices

  1. Event-Driven Architecture Patterns - IBM’s comprehensive pattern catalog
  2. Enterprise Integration Patterns - The foundational reference
  3. My TOP Patterns for Event Driven Architecture - Real-world patterns from CodeOpinion

AWS Resources

  1. Building Event-Driven Architectures on AWS - Official AWS guide
  2. Choosing Between Messaging Services - EventBridge vs Kinesis vs SNS/SQS
  3. AWS EventBridge Best Practices - Production recommendations

Books

  1. Building Event-Driven Microservices by Adam Bellemare - O’Reilly
  2. Designing Event-Driven Systems by Ben Stopford - Free PDF