We are seeking a highly skilled Lead Data Engineer to drive the design, development, and optimization of large-scale data platforms. This role involves leading complex, enterprise-wide initiatives, building modern data pipelines, and shaping best practices for data engineering across the organization.
Key Skills
Data Engineering, Python, SQL, Apache Spark, Hadoop, Airflow, AWS, Azure, GCP, REST APIs, CI/CD, Docker, Kubernetes, Data Lakehouse, ETL/ELT, Agile Key Responsibilities
Technical Leadership & Architecture
- Lead large-scale, high-impact technology initiatives across teams
- Define and implement best practices for data engineering and platform architecture
- Review and evaluate complex system designs aligned with business and enterprise goals
- Mentor team members and provide technical leadership
Data Engineering & Pipeline Development
- Design, build, and maintain scalable data pipelines (ETL/ELT) for structured and unstructured data
- Develop metadata-driven ingestion frameworks, validation layers, and reusable components
- Ensure high performance, reliability, and scalability of data systems
Distributed Computing & Lakehouse Engineering
- Build and optimize Apache Spark pipelines for batch and streaming workloads
- Work with modern data lakehouse technologies (Iceberg, Delta Lake, Hudi)
- Implement Medallion architecture (Bronze/Silver/Gold layers)
Data Quality & Observability
- Implement data quality frameworks (e.g., Great Expectations, Deequ)
- Build monitoring systems with SLAs/SLOs, anomaly detection, and lineage tracking
- Ensure robust validation during migrations and onboarding processes
API & Microservices Development
- Develop RESTful APIs using Python frameworks (FastAPI, Flask)
- Enable secure and governed data access across platforms
Cloud & DevOps
- Design and deploy pipelines on cloud platforms (AWS, Azure, GCP)
- Build CI/CD pipelines using tools like Jenkins, GitHub Actions, or Azure DevOps
- Implement infrastructure as code (Terraform, Helm) and secure engineering practices
Orchestration & Workflow Management
- Build and manage workflows using tools like Airflow or Autosys
- Design resilient pipelines with retries, alerts, and dependency handling
Collaboration & Delivery
- Work with cross-functional Agile teams including Product, Architecture, and Business stakeholders
- Analyze requirements, propose solutions, and contribute to technical roadmaps
- Independently deliver complex engineering solutions
Required Qualifications
- Bachelor’s degree in Engineering, Computer Science, or related field
- 5+ years of experience in software/data engineering or equivalent practical experience
Technical Skills & Experience
Core Skills
- Strong hands-on experience with Python, SQL, and Bash scripting
- Experience with big data technologies: Apache Spark, Hadoop, Hive
- Expertise in building scalable data pipelines and distributed systems
Data Platforms & Storage
- Experience with data lakehouse architectures and storage formats (Parquet, ORC)
- Knowledge of optimization techniques (partitioning, clustering, compaction, Z-ordering)
Streaming & Advanced Processing
- Experience with streaming frameworks such as Spark Structured Streaming or Apache Flink
APIs & Integration
- Working knowledge of REST APIs, object storage, and data access layers
Cloud & DevOps
- Hands-on experience with AWS, Azure, or GCP
- Familiarity with CI/CD tools, containerization (Docker, Kubernetes), and automation
Data Governance & Quality
- Experience with governance tools (Collibra, Alation, Purview)
- Understanding of compliance standards (SOX, PCI) and data validation practices
Additional (Good to Have)
- Experience with GenAI applications in data engineering (metadata extraction, anomaly detection, automation)
- Domain exposure to financial services, treasury, or risk management
Key Competencies
- Strong problem-solving and analytical thinking
- Leadership and mentoring capabilities
- Excellent communication and stakeholder management skills
- Ability to work in fast-paced Agile environments
Education
- UG: B.Tech / B.E. or equivalent in any specialization
- PG: Any Postgraduate (preferred)
