Healthcare · Data Operations
Modernizing enterprise data pipelines with Spark and Airflow.
Fortune 10 healthcare enterprise
A ground-up modernization of legacy data integration onto Spark and Airflow — smarter scheduling, event-driven triggers, and stronger validation — delivering roughly $5M in annual cost reduction.
Healthcare · Data Operations
Challenge
What was the challenge?
Critical pharmacy data flows — eligibility feeds, claims transfers, intake files from partners — ran on aging integration tooling: proprietary legacy transformation maps, rigid schedules, and pipelines that polled for new files every one to two minutes around the clock whether any work existed or not. That constant polling alone put real load on the orchestration platform.
The pipelines were also brittle in quieter ways. Common logic was copy-pasted across jobs, so a fix in one place had to be repeated everywhere. Queued runs executed in arrival order rather than business priority, so an urgent eligibility load could sit behind routine housekeeping. Memory pressure built up silently until it became an incident. And gathering configuration details for eligibility and claims-transfer analysis was a slow, manual chore.
Approach
How did we approach it?
Rather than lift-and-shift the old jobs, we rebuilt the platform's foundations: legacy transformation maps were migrated to Spark, orchestration was rebuilt on Airflow, and the DAGs themselves were redesigned to be dynamic and modular — generated programmatically so parallelism comes from structure, not hand-tuning, with all shared functionality consolidated into common, globalized modules.
We then attacked the operational waste. Constant polling was replaced with event-driven triggers: a lightweight watcher fires the right pipeline the moment a file lands. Queue handling became priority-aware so the most business-critical runs always go first. And the platform learned to look after itself — scheduled purging keeps memory in check, and dedicated alerting pipelines flag problems before they become outages.
Delivered
What did we deliver?
- Migration of legacy transformation maps to Spark with stronger data validation and materially faster processing across the modernized flows.
- Intake and processing DAGs for the major eligibility and claims-transfer data domains, rebuilt on Airflow.
- Dynamic DAG construction with shared logic globalized into common modules — one fix propagates everywhere, and parallelism is built into how pipelines are generated.
- Priority-aware scheduling so queued runs execute in business-importance order, not arrival order.
- Event-driven file triggers monitoring inbound and NAS locations, replacing every-minute polling and cutting orchestration load.
- Operational hardening: purging DAGs that control memory usage, alerting DAGs that catch capacity issues early, and automation scripts that cut the manual effort of configuration analysis for eligibility and claims transfers.
- A Spark-based REST API over the claims adjudication data source, so downstream teams fetch data through a clean interface instead of direct server access.
- New intake applications — including a medication-workflow application and a custom validation tool — developed, deployed, and delivered on time with zero defects in both UAT and production.
Mechanism
How does it work under the hood?
- 01Airflow DAGs are generated dynamically from configuration rather than hand-written per feed, with common functionality consolidated in shared modules — the codebase got smaller as the platform got bigger.
- 02A lightweight file-watcher script monitors inbound and NAS drop locations and triggers the matching DAG on arrival; pipelines run when there is work, not on a timer.
- 03Queue prioritization logic inspects waiting runs and promotes the most business-critical ones to execute first.
- 04Purging DAGs reclaim storage and memory on schedule; alerting DAGs watch platform health and page before thresholds are breached.
- 05The Spark REST API wraps the claims data source, giving other teams governed, self-serve access to the data they previously had to request manually.
Outcome
What was the outcome?
The modernization delivered approximately $5M in annual cost reduction through faster processing, less platform load, and the removal of manual overhead — while making the pipeline estate easier to extend, since new feeds now inherit the shared modules and dynamic DAG framework instead of starting from scratch.
Delivery quality held throughout: the new applications shipped with zero defects in both UAT and production, on schedule.
Learnings
Key learnings
Event-driven beats scheduled: replacing every-minute polling with file triggers cut platform load without touching a single business rule.
Consolidating shared logic into global modules pays compound interest — every later pipeline shipped faster because the common work was already done.
Priority-aware queues are a cheap, high-leverage fix: the same infrastructure feels dramatically faster when the most important work always runs first.
Related stories
Healthcare · Pharmacy Benefits
Digitizing complex pharmacy contracts with generative AI
An end-to-end GenAI pipeline that reads dense pricing contracts — tables, legal language, and billing rules — and turns them into audit-ready structured data, unlocking roughly $64M in annual savings.
Healthcare · Specialty Pharmacy
Preventing misdelivered medicines with NLP
An NLP system that extracts delivery addresses from call transcripts at ~95% accuracy and validates them against member records — saving roughly $2M a year and protecting specialty medicine deliveries.
Work with us