EcZachly

EcZachly

@Zachly

@Zachly
@eczachly
@eczachly
eczachly
Data with Zach

Recent Shorts

Fact data modeling explained day 1I became fluent in Yapanese on LinkedIn with WisprFlowData engineering roadmaps are broken in 2026
@eczachly

Recent Videos

34% of my feed is ads on LinkedIn today from a 250 piece of content sample. This is up from 29% in June 2026. Of that 34%, 60% of the ads are for LinkedIn. LinkedIn is currently taking 20% of the feed inventory for ads that market LinkedIn. All of these ads I immediately scroll by or mark “not interested” How is wasting 20% of the feed to market your own platform a good use money? Maybe they could give some of those slots back to organic creators who have been creating every single day for six years on this platform? Meta doesn’t use 20% of its feed marketing Meta products. It uses less than 1%. How is LinkedIn leadership so asleep at the wheel? Daniel Shapero, marketing LinkedIn on LinkedIn is pretty stupid. It doesn’t make you guys money. All it does is steal organic reach from creators that are actively contributing to your platform. Thanks for listening to my TED talk!Stop identifying with your job title! It’s a trap that will keep your poor You aren’t a data engineer or a product manager. You’re a human capable of adapting and learning so much. When I was working at Airbnb, they were like, you’re one of the best data engineers, Zach. We want you to become a principal data engineer! The glory of being a principal engineer in big tech. Wow! And I drank the Koolaid fully from 2016 to 2023. But so much sugar ultimately made my stomach sick and I realized my life was more than SQL, Scala, Spark and Airflow. I irrationally left a $600k data engineering job for a $0 content creator business. I was destined to be an internet clown and educator. I traded SQL for shitposting. I traded Spark for sales. I traded Scala for scaling my business. I traded Airflow for art. And now this absolutely absurd decision in 2023 is looking better and better as technical skills continue to get undermined by AI. Give yourself to reinvent yourself! Give yourself permission to change jobs! You’re a multi potential human Intelligence is the ability to adapt to change!Turning a vibe-coded data pipeline into production-grade software requires thinking about 100s of different engineering concepts! AI can generate an ETL pipeline in minutes. But getting that pipeline to process billions of records reliably, maintain data quality, and recover from failures? That’s a completely different animal! Here’s what vibe coders building data pipelines need to learn: 1. Data Ingestion & Extraction * Batch vs. streaming ingestion * Change Data Capture (CDC) * Full vs. incremental loads * API rate limiting * Schema inference * Data serialization * Idempotent ingestion * Late-arriving data 2. Data Modeling & Storage * Star schemas * Snowflake schemas * Slowly Changing Dimensions (SCDs) * Fact and dimension tables * Data normalization * Partitioning * Clustering * File compaction * Columnar storage * Data lakehouse architectures 3. Distributed Data Processing * Data shuffling * Data skew * Broadcast joins * Sort-merge joins * Predicate pushdown * Partition pruning * Parallelism * Memory management * Adaptive query execution 4. Orchestration & Reliability * DAG orchestration * Task dependencies * Retries * Exponential backoff * Idempotency * Backfills * SLAs * Data freshness * Failure recovery * Workflow scheduling 5. Streaming & Real-Time Data * Event time vs. processing time * Watermarks * Windowing * Exactly-once vs. at-least-once processing * Event deduplication * Out-of-order events * Checkpointing * State management * Kafka partitions * Backpressure 6. Data Quality & Governance * Data contracts * Schema evolution * Schema enforcement * Null handling * Duplicate detection * Referential integrity * Data validation * Data lineage * PII handling * Access controls * GDPR compliance 7. Performance & Cost Optimization * Warehouse sizing * Autoscaling * Query caching * Materialized views * Incremental materialization * Storage optimization * Compute optimization * Cost attribution * Concurrency management * Query profiling 8. Deployment & Observability * CI/CD for data pipelines * Infrastructure as Code * Environment isolation * Unit testing * Integration testing * Data reconciliation * Pipeline monitoring * Data quality alerts * Freshness monitoring AI has reduced the cost of writing data transformations! But it hasn’t eliminated the complexity of running data pipelines at scale. You can easily vibe code your way to a pipeline that works on 10,000 rows. But when that pipeline needs to process 10 billion rows, handle schema changes, recover from partial failures, or guarantee downstream data correctness. You’re basically cooked！ The future isn’t data engineers vs. vibe coders. It’s data engineers who understand distributed systems, data architecture, and production reliability AND know how to leverage AI. Writing SQL and Python is getting cheaper. Data engineering judgment is becoming more valuable！ What would you add to the list? Inspired by Natalie L.
Powered by Ferryman