Data Engineer

Job Summary:

The Senior Data Engineer will design, implement, and optimize the company’s data infrastructure with a focus on building a scalable data warehouse, enabling AI-driven analytics, and supporting machine learning initiatives. This role will lead the development of efficient data pipelines, ensure data quality, and collaborate with data scientists to drive business insights and automation.

  Key Responsibilities:

• Data Warehouse Architecture: Design and maintain a cloud-based data warehouse that consolidates data from various systems (ERP, CRM, e-commerce, inventory) and enables seamless access for analytics and AI applications.
• Data Pipeline and ETL Management: Build and optimize ETL pipelines that transform raw data from multiple sources into structured, high-quality data, ensuring data integrity and availability for analysis and modeling.
• AI and Machine Learning Collaboration: Work closely with data scientists to enable machine learning capabilities by preparing, transforming, and feeding data to AI models for applications such as predictive analytics, personalization, and customer segmentation.
• Data Quality and Governance: Implement data validation, cleaning, and monitoring processes to ensure the accuracy and consistency of data across the data warehouse. Enforce data governance practices, including access control, compliance with data privacy standards (GDPR, PDPA), and metadata management.
• Scalable Data Architecture: Develop scalable data solutions that accommodate growing data volumes, ensuring that data infrastructure can handle real-time and batch processing for advanced analytics.
• Performance Optimization and Troubleshooting: Continuously monitor data pipelines and warehouse performance, optimizing workflows and resolving issues to improve data processing speed and efficiency.
• Collaboration with BI and Analytics Teams: Partner with BI developers and data analysts to build dashboards, reports, and data visualizations, supporting data-driven decision-making across departments.
• Mentorship and Leadership: Mentor junior data engineers, provide technical guidance, and promote best practices in data engineering, including ETL design, data modeling, and data warehouse optimization.

Required Skills and Qualifications:

• Bachelor’s degree in Computer Science, Data Engineering, Information Systems, or a related field; a master’s degree is a plus.
• 5+ years of experience in data engineering, with expertise in data warehousing, ETL processes, and data integration.
• Proficiency in data warehousing technologies (e.g., Snowflake, BigQuery, AWS Redshift) and ETL tools (e.g., Apache Airflow, Talend).
• Strong knowledge of programming languages (e.g., Python, SQL) for data processing and data pipeline automation.
• Experience working with AI and machine learning teams to develop data workflows that support model training, validation, and deployment.
• Familiarity with data governance and compliance standards, ensuring secure and compliant data practices.
• Excellent problem-solving skills, with a focus on performance optimization and troubleshooting in data systems.