1. The hub: The router pipeline
The Hub is a lightweight Dataflow job that acts as a traffic controller. It reads from unified source topics, parses the Tenant ID or Business Domain, and fans the data out into isolated buffers. This keeps the entry point simple and robust.
2. The buffer: Durable isolation
Pub/Sub topics are introduced between the Hub and the Spokes. These act as a durable shock absorber, preventing a slow downstream sink from backing up the original source.
3. The spokes: Isolated execution
Instead of one giant pipeline, multiple, smaller Dataflow instances are deployed and categorized by workload:
-
Tier 1 (high-priority): Dedicated pipelines with high resource allocation for critical tenants.
-
Shared tiers: Grouped pipelines for smaller tenants to optimize costs.
-
Domain specific: Specialized pipelines for complex logic (e.g., separating distinct business domains) to isolate code complexity.
Comparative Benefits at a Glance
|
Feature |
Monolithic (legacy) |
Hub-and-spoke |
|
Fault tolerance |
One failing DB stops everything |
Failures isolated to specific spoke |
|
Blast radius |
100% |
< 5% (Isolated to one spoke) |
|
Resource scaling |
Scaled for “worst-case” |
Independent scaling per tenant load |
|
Maintenance |
Global updates affect everyone |
Update one domain without touching others |
Pro-Tips for Implementation
Moving to this architecture is more than just shifting boxes on a diagram. Additional “Spoke” level optimizations are recommended for maximum stability:
-
Implement Dead Letter Queues (DLQ): Do not let a single SQL exception stall the pipeline. Route failed records to storage (like BigQuery or Google Cloud Storage) for later investigation.
-
Strict connection pooling: Databases have connection limits. Use a thread-safe singleton pattern and set a low MaximumPoolSize (e.g., 1-2) per worker to avoid exhausting the database during autoscaling.
-
Asynchronous I/O: Use the GroupIntoBatches transform to buffer writes, which reduces the connection overhead that often triggers database-induced latency.
Conclusion
By adopting a sharded approach, platforms can guarantee that a “noisy neighbor” is no longer a threat to the neighborhood. This architecture provides the isolation needed to maintain strict SLAs while allowing for independent scaling and safer deployments
To learn more about implementing a sharded hub-and-spoke architecture, explore the Dataflow documentation.
