An industrial IoT platform needed to stabilize a failing Apache Druid environment and migrate to StarRocks – at the same time. Here is how both tracks ran in parallel and what the migration actually involved.
An industrial IoT platform was running its analytical workloads on Apache Druid. The system ingested telemetry continuously, ran derived analytics, and served queries over recent and historical data.
Over time, the workload changed. Druid was still the production system, but it was becoming harder to operate. Ingestion incidents required manual recovery. Compaction and reindexing needed close attention. Historical nodes were under pressure. A custom versioning extension, originally built to support correcting analytical records, had become a major source of operational complexity.
Deep.BI evaluated the customer’s update-heavy workload and recommended StarRocks as the long-term target because its Primary Key tables were a better fit for the way corrected analytical records needed to be handled. After reviewing the proposed architecture and migration approach, the customer accepted the recommendation and moved forward with the plan. Deep.BI helped on both sides: stabilize Druid first, then design and support the StarRocks migration.
The customer had a real production system, not a clean migration lab.
Druid was ingesting data from Kafka. Some of the data was raw telemetry. Some of it was derived analytical data that could be recalculated later. When analytics were re-executed, older logical records had to be superseded by newer versions.
Druid is strong for append-heavy analytical workloads, but this use case needed update semantics. The customer had addressed that with a custom design. Each record carried version information. The customer also maintained an in-memory versioning map through a custom extension. The extension provided custom aggregators that used this map to determine which record versions were current, and queries had to use those custom aggregators to return correct results
That design was reasonable when it was introduced. It allowed the platform to keep using Druid while supporting corrected analytics. But the cost grew with the data volume. The versioning layer consumed memory, increased storage costs because multiple versions of the same logical records had to be retained, affected query design, complicated upgrades, and made troubleshooting harder.
The immediate goal was not to replace Druid in one move. The first goal was to make it stable enough to keep running.
The Druid work started with operational issues.
The customer had seen real-time ingestion incidents where tasks stopped progressing and Kafka lag grew quickly. In one case, lag exceeded 100 million records. Recovery required scaling MiddleManager resources and increasing task runtime limits so ingestion could catch up.
Deep.BI reviewed the incident and discussed possible recovery approaches with the customer. One topic that came up was the use of ingestion resets during troubleshooting. Rather than recommending resets as a routine operational practice, the focus was on understanding the state of ingestion first by reviewing task logs, Kafka lag, delayed events, and resource utilization.
The investigation focused on ingestion behavior, task capacity, and cluster sizing. The customer increased MiddleManager resources and adjusted task settings to help ingestion recover, and Deep.BI provided guidance on how those changes related to the observed backlog and task execution characteristics.
The customer also needed help with Druid versions.
Deep.BI reviewed the upgrade path, breaking changes, and configuration changes. The customer later completed the move to Druid v34 without major issues. That mattered because the existing version was carrying both operational risk and upgrade friction.
For issues that affected the customer before they could move further upstream, Deep.BI also provided a customer-specific patched Druid build. The purpose was practical: reduce risk in the current environment while the longer upgrade and migration path was being prepared.
This was not positioned as a permanent fork. It was a bridge. The customer needed a stable Druid environment first, because the migration could not succeed if production was constantly in recovery mode.
A large part of the Druid work focused on segment layout and data lifecycle.
The customer had already seen that partitioned data could perform better in development. The next step was to make partitioning and compaction reliable enough for production.
One problem was schema inconsistency. For instance, a field named count appeared both as a dimension and as a metric in different specs. That caused compaction failures. The fix was to align the Kafka supervisor spec, reindexing spec, and compaction spec so the column was defined consistently.
Another problem was shard type. Some compaction jobs produced dynamically partitioned or numbered shards instead of the intended partitioning strategy. Deep.BI advised the customer to define the partitioning strategy explicitly in compaction when consistent segment layout was required.
The team also reviewed dimension order. For Druid partitioning, the order of dimensions can matter. Historical reindexing, real-time ingestion, and compaction needed to produce compatible segment layouts. Otherwise, the customer could end up with data that looked equivalent at query time but behaved differently during compaction and pruning.
Druid MSQ Engine was introduced to remove obsolete versioned data. This became increasingly important as the volume of superseded records continued to grow, creating both performance issues and additional challenges for the planned migration. At its peak, some queries were scanning more than 100 times the amount of data actually needed because historical versions had accumulated over time. Implementing a cleanup process through reindexing reduced the amount of data that needed to be scanned, improved query performance, lowered storage costs, and made the eventual migration significantly more manageable.
The customer also reported compaction tasks that could run indefinitely.
Thread dump analysis suggested that a parent task was waiting for a subtask response. The subtask may have failed, lost communication, or been blocked by resource or deep storage behavior. Deep.BI reviewed the failure mode and identified an issue in the Druid code path used by the customer's environment. As part of the customer-specific Druid release, we provided a bug fix that addressed the task coordination behavior contributing to these stuck compaction scenarios, in addition to configuration and operational recommendations. We also advised moving away from legacy task runner behavior where applicable and validating the fix under production-like workloads.
The important point was not only the specific thread dump. The bigger issue was that the old ingestion and compaction model still required too much manual intervention. Druid could be stabilized, but the customer was still spending effort on problems caused by the fit between the workload and the architecture.
After the stabilization work, the main concern was still data updates.
The customer had derived analytical records that could be recalculated. In the old Druid design, the system wrote new versions and used a lookup to decide which version was active. That meant query correctness depended on lookup state and version filtering. Periodic cleanup was needed to remove old versions.
This created several problems:
This was the point where tuning was no longer enough. Druid could be made stable, but it was still not the best fit for update-heavy analytical records.
StarRocks was chosen because Primary Key tables offered a cleaner model. A logical record could be represented by a primary key. A corrected record could replace the previous version through the table’s update mechanism. The update behavior moved into the database instead of living in a custom Druid extension and query-time lookup layer.
The customer initially considered running StarRocks on EC2 with S3 in shared-data mode. Deep.BI reviewed the tradeoffs and recommended EKS because the customer already operated Kubernetes and could use the StarRocks operator and Helm chart.
The recommendation was not generic. It was based on the customer’s environment. With EKS, the team could manage deployment, scaling, service discovery, and upgrades more consistently. A direct EC2 deployment was possible, but it would have required more custom operations work.
Deep.BI also recommended running StarRocks on a dedicated node group. CPU and memory limits are visible in Kubernetes, but database workloads also depend on network and disk behavior. Keeping StarRocks separate from application workloads made performance easier to reason about and reduced noisy-neighbor risk.
The reviewed architecture used shared-data mode with S3 as durable storage and StarRocks compute nodes for execution and cache. That design reduced the need to replicate data across local disks, but it did not remove the need for cache planning. Cold reads from object storage and warm reads from local cache have different performance profiles.
The schema work was one of the most important parts of the migration.
The customer had two main data patterns. Raw time-series sensor data was mostly append-oriented. Derived analytical data needed updates because it could be recalculated. Treating both patterns as the same table design would have been a mistake.
For the derived data, Primary Key tables were the right starting point. The key design question was how to define row uniqueness. Some logical records were identified by a wider set of fields than would be practical as a primary key. The design therefore used a hash column as part of the uniqueness strategy.
Deep.BI also corrected a common migration assumption: Druid partitioning cannot be copied directly into StarRocks. StarRocks table design uses several different concepts:
The initial StarRocks ordering key was too broad. Too many high-cardinality columns in the sort key would increase load and sort cost without necessarily improving query performance. Deep.BI recommended a smaller sort key based on the most important filters and query patterns.
The indexing advice was also different from the original Druid mindset. StarRocks does maintain a prefix index, but it does not automatically create Druid-style indexes for every analytical dimension. Bitmap and bloom filter indexes needed to be chosen deliberately, tested, and tied to real predicates.
The customer already used Kafka as the ingestion path for Druid. For StarRocks, Routine Load was the simplest fit for continuous ingestion from existing topics.
Deep.BI reviewed whether a more complex connector setup was needed. For this workload, it was not. Routine Load covered the live ingestion path and kept the architecture understandable.
Historical migration was harder. A direct insert path from Druid to StarRocks was not the right operational model. The better approach was to use a migration service that read from Druid in controlled slices, transformed the records, and published them to Kafka. StarRocks then consumed the same way it consumed live data.
That gave the team one ingestion path for both history and live data. It also made cutover easier to control. Once historical migration caught up and live ingestion was running through StarRocks, the migration service could stop.
StarRocks was a better fit for updates, but it did not perform well automatically in every area. The first deployment needed review and correction.
The most obvious issue was a memory mismatch in the Frontend configuration. The pod limit was set to 4 GiB, while JAVA_OPTS configured the JVM heap at 8 GiB. Under load, that kind of mismatch can cause container kills and misleading stability problems. Deep.BI recommended either lowering the heap or increasing the pod limit with enough headroom.
Compute node scaling also needed work. The CN autoscaling policy was not defined, so the cluster would not automatically add compute capacity under query load. Deep.BI recommended setting CN replicas or configuring starRocksCnSpec.autoScalingPolicy explicitly. We also clarified that autoscaling does not replace cache and disk sizing. If local cache is too small, adding CPU alone will not fix cold-read behavior.
The shared-data cache design needed attention as well. CN local disks act as cache, while S3 remains durable storage. That means cache size, cache warmup, and working-set behavior can have a direct effect on query latency. The customer needed to size cache for the data that users query most often, not only for average storage volume.
Several production-safety issues were also identified in infrastructure code. forceDestroy: true on the S3 bucket was acceptable for development but unsafe for production. A hardcoded StarRocks secret needed to be rotated or moved into a proper secrets process. Backup planning had to include FE metadata, not only S3 data files.
The customer needed better visibility into query behavior.
Deep.BI recommended AuditLoader and SQL digest. AuditLoader stores query execution records in StarRocks. SQL digest groups structurally similar queries even when literal values differ. That makes it easier to see which query patterns are slow, frequent, or regressing after schema changes.
This was important because the migration was not only about making queries run. The customer needed to see which queries were expensive and why. Without that feedback loop, schema and index tuning would be guesswork.
The Druid environment became stable enough to keep production running while the StarRocks migration moved forward.
The Druid work covered incident response, upgrade planning, customer-specific patching, compaction fixes, partitioning guidance, schema alignment, task troubleshooting, metadata maintenance, and MSQ Engine cleanup guidance. The result was not that Druid became the perfect long-term system for the workload. The result was that Druid stopped being the immediate blocker.
The StarRocks work addressed the longer-term architecture. Primary Key tables gave the customer a better model for corrected analytical records. Deep.BI helped review the deployment model, schema, ingestion path, historical migration approach, indexing strategy, cache behavior, autoscaling, observability, and production hardening.
The migration was difficult in the areas that usually matter most: not syntax, but design and operations. Queries had to be rewritten, but the harder work was mapping Druid versioning semantics to StarRocks Primary Key tables, sizing shared-data cache, choosing sort keys and indexes, validating Kafka ingestion, protecting metadata, and making query performance visible.
By the end of the engagement, the customer had a stabilized Druid platform and a practical StarRocks migration path. Druid could continue serving production while StarRocks was tested and hardened. The update problem moved out of a custom extension and lookup-dependent query model and into a database design that supported updates directly.
This project was not a simple database replacement.
Druid still had to be stabilized because it was running production. StarRocks still had to be designed carefully because Primary Key tables solve update semantics, not every performance and operations problem by themselves.
The useful lesson is straightforward: stabilize the old system enough to reduce risk, then migrate the parts of the workload that no longer fit its model. In this case, Druid remained valuable for append-heavy analytics, but the customer’s derived and re-executed data needed a different update model. StarRocks gave them that model, and the work was in making the migration safe enough to execute.
Deep.BI supports teams on both sides of the move: stabilizing the Druid cluster that runs production today, and designing the StarRocks schema, ingestion, and deployment that replaces it.
