Data teams are falling apart. Data scientists stay 1.7 years on average. 70% of data engineers are actively job hunting. 85% of data science projects never make it to production. Jesse Anderson's 2024 survey found the obvious culprit: only half of struggling teams have data operations support, versus 100% of high-performing ones.
The problem isn't compensation or tech stack. It's organizational dysfunction—teams solving wrong problems, data engineering and data science working in isolation, and nobody owning the full path from raw data to business value.
The telephone game
Every piece of data travels a path: user action → frontend logging → API transformation → pipeline processing → warehouse storage → analysis → model training → business decision. Most teams lose the thread around step three.
I've watched this fail repeatedly. A growth team I worked with ran experiments for months before discovering less than 1% of sessions had device-specific bugs causing non-random assignment. The user ID hashing worked perfectly in tests, but returned zero on certain mobile builds. Months of "conclusive" A/B tests, subtly poisoned. The mobile team knew about the edge case. Data engineers who built the pipeline didn't. Data scientists analyzing experiments had no idea.
Unity Technologies lost $110 million when corrupted customer data poisoned their ML models. Target's pregnancy prediction worked technically but failed ethically—the team understood purchase patterns but not privacy implications. These failures happen when teams don't understand their data generating process.
After years building data products and teams, mostly independently or in small groups (but sometimes in large organizations), the pattern is clear. We structure data teams like siloed departments when we should be thinking about integrated systems. Half the team doesn't understand what the other half builds. Predictable disasters follow.
Different tools for different problems
Homogeneous teams create homogeneous blind spots. A team of CS grads will build technically correct solutions to the wrong problems. They'll optimize what's measurable, not what matters.
The healthcare algorithm that recommended fewer Black patients for care programs is the perfect example. It optimized healthcare costs as a proxy for healthcare needs. Technically flawless. Practically discriminatory. The model accurately predicted costs, but missed that systemic inequities mean Black patients receive less care for the same conditions, generating lower costs in training data.
Diverse technical backgrounds catch these failures. Your psychologist understands survey response bias your CS grad never studied. Your economist brings instrumental variables and regression discontinuity designs your physicist never learned. Your physicist spots measurement errors your MBA might miss. Your anthropologist asks whether engagement metrics measure anything users actually care about.
Teams with ethnic diversity outperform by 35%. Gender-diverse teams by 50%. This isn't corporate wellness. It's practical advantage. When everyone uses the same analytical hammer, everything looks like a nail. Mixed backgrounds mean actual toolboxes.
The antipatterns
Pardis Noorzad studied 30+ AI companies and found the same failing patterns everywhere. Her framework shaped how I think about team structure.
The Support Desk: Data teams become SQL factories. Quick pulls eat all their time. Data engineers maintain pipelines for reports nobody reads while data scientists build models for problems nobody has. Neither group knows what the other does. Noorzad calls this model "highly problematic"—teams excluded from product decisions stop caring.
Dashboard Theater: Beautiful charts for board meetings. Vanity metrics for investor decks. Real problems go unsolved. One data scientist's summary: "I have a PhD in statistics and I make PowerPoints."
Permanent Prototype: Jupyter notebooks everywhere, production nowhere. That revolutionary model from six months ago? Still on someone's laptop. Infrastructure can't support it anyway. Anderson's data confirms it—struggling teams lack operations support.
Premature Hiring: Companies hire data scientists before building data infrastructure. DataKitchen found over half of data engineers spend their time fixing errors instead of building. They become plumbers, not architects.
Noorzad identified the core issue: separation from impact. Her "center of excellence" model creates ivory towers where teams "focus on pie in the sky ideas while business suffers." When data engineering and data science operate separately, both fail.
The exodus
The numbers paint the full picture:
- 1.7 years average tenure for data scientists (LinkedIn)
- 70% of data engineers planning exits within 12 months (DataKitchen)
- 79% considering leaving the industry entirely (Wakefield)
- 42% cite unrealistic expectations as primary frustration
- 85% project failure rate (Gartner)
- 50% of struggling teams lack data operations (Anderson)
They're not leaving for money—tech pays well everywhere. They're leaving because they build infrastructure for models that never deploy. Because they optimize metrics that don't matter. Because they're isolated from impact.
The unicorn hunt makes it worse. Job posts demanding expertise in "Spark, Hadoop, Hive, Pig, SQL, Neo4J, MySQL, Python, R, Scala, Tensorflow, A/B Testing, NLP." Intel's Bob Rogers put it well: "Michael Jordan couldn't have scored so many points without Scottie Pippen." But companies keep searching for someone who can play all five positions simultaneously.
What works
Shopify, Airbnb, and Spotify converged on similar solutions. Noorzad calls it the "Product Data Science model"—not complicated, but requires discipline.
Equal standing: Data leaders need the same organizational weight as engineering and product leads. Noorzad's key insight: "for success on cross-functional teams, all functional leads should have similar amounts of negotiating power." Without it, data becomes a service desk.
Integrated teams from the start: Spotify runs 20,000+ daily pipelines with 300+ teams because data engineers and scientists work in the same squads. They share ownership. DE understands the models. DS understands the constraints.
Break down walls simply: Peak's kitchen office hours—data scientists available for questions at regular times. No meetings added. Data engineering joins. Product joins. Everyone starts understanding the full pipeline.
Real career paths: Netflix progression from Data Scientist I through Distinguished Data Scientist parallels management track to Chief Data Officer. Don't force ICs into management. Companies investing in actual learning—conferences, rotations, exploration time—see 326% better retention.
Infrastructure first: 100% of successful teams have dedicated data operations (Anderson). Build monitoring, testing, and data contracts before hiring more scientists.
Start here:
- Build data infrastructure and operations
- Create integrated teams with shared ownership
- Establish feedback loops—one-on-ones, peer review, impact measurement
- Fund real learning—conferences, rotations, research time
- Then scale
The opportunity
Everyone knows these problems. The research is public. Solutions are documented. Companies still burn through talent every 1.7 years, accepting 85% failure rates as normal.
While others treat this as inevitable, teams that fix structures build lasting advantage. They're not smarter. They just stopped separating engineering from science, stopped hunting unicorns, stopped treating data teams like service desks.
The choice is straightforward. Keep writing impossible job descriptions and wondering why nothing ships. Or build teams that understand data's complete journey, where diverse backgrounds catch blind spots, where infrastructure and science evolve together.
I've built teams across different organizations and contexts. The difference was never about the technology or the math. It was about whether anyone understood how user actions became business decisions—the whole path, not just their piece.
Your data team is probably updating their LinkedIn right now. The question is whether you'll fix the structure or just hire their replacements into the same broken system.