Never pay for any CBT, test or assessment as part of any recruitment process. When in doubt, contact us
Job Title: Senior Data Engineer
Reports To: Functionally – Data Engineering Manager
Data Engineer Job Responsibilities:
- Serve as the subject matter expert for data and systems.
- Develops and maintains scalable data pipelines and builds out new API integrations to support continuing increases in data volume and complexity.
- Collaborates with analytics and business teams to improve data models that feed business intelligence tools, increasing data accessibility and fostering data-driven decision making across the organization.
- Implements processes and systems to monitor data quality, ensuring production data is always accurate and available for key stakeholders and business processes that depend on it.
- Develop end-to-end ML pipelines encompassing the ML lifecycle from data ingestion, data transformation, model training, model validation, model serving, and model evaluation over time.
- Collaborate closely with AI scientists to accelerate deployments of ML algorithms to production.
- Setup CI/CD/CT pipelines, model repository for ML algorithms
- Deploy models as a service
- Contributes to engineering wiki, and documents work.
- Performs data analysis required to troubleshoot data related issues and assist in the resolution of data issues.
- Works closely with cross-functional teams of frontend and backend engineers, product managers, and analyst to enhance data models and support advanced BI and analytics.
- Defines company data assets (data models), ETL jobs to populate data models.
- Designs data integrations and data quality framework.
- Designs and evaluates open source and vendor tools for data lineage.
- Works closely with all business units and engineering teams to develop strategy for long term data platform architecture.
- Mentor junior data engineers, lead code reviews, and promote best practices and skill development.
Data Engineer Qualifications / Skills:
- Knowledge of best practices and IT operations in an always-up, always-available service
- Establish and promote best practices for data pipeline and model development.
- Experience with or knowledge of Agile Software Development methodologies
- Excellent problem solving, creativity, attention to detail and troubleshooting skills
- Process oriented with great documentation skills
- Excellent oral and written communication skills with a keen sense of customer service
- Eagerness to learn and upskill to stay at the forefront of technology offerings in the market.
- Understanding of ML Algorithms, experience creating & executing efficient MLOps pipelines, and tuning ML models
- Team player mindset with an enthusiasm for collaboration.
Education, Experience, and Licensing Requirements:
Must Have:
- BSc or MSc degree in Computer Science or a related technical field
- 5+ years of Python or R development experience
- 5+ years of MS SQL experience (PostgreSQL experience is a plus)
- 5+ years of experience with Warehouse Architecture, schema design and dimensional data modeling.
- 5+ years of experience in Data Analytics and Business Intelligence tools such as Power BI
- Ability in managing and communicating data warehouse plans to internal clients
- Experience designing, building, and maintaining data processing systems on multiple platforms both Cloud (Azure, AWS, MS Fabric is a plus) and On-Premises (MS SQL Server, SSIS)
- Experience in ML Model deployment, ML frameworks and libraries
- Good experience in Apache Spark.
- Experience debugging and reasoning about production issues is desirable.
- Experience presenting demos and training of technical, non-technical and analytical resources
Advantages to have:
- Experience in data streaming is advantageous i.e. Kafka and/or AWS Kinesis
- IoT device and systems integration