Data Platform System Engineer

You will be responsible for operating and evolving SP Group's enterprise Data & AI platform to support mission-critical analytics, regulatory compliance, and AI-driven innovation. This role focuses on ensuring the platform is scalable, reliable, secure, and performant, while enabling data engineers, scientist and analysts to deliver business value efficiently.

What You'll Do

• Ensure high availability and reliability of the enterprise data platform (Cloudera and Talend Big Data).
• Deliver scalable and performant data infrastructure supporting batch and near real-time workloads.
• Enable secure and compliant data operations aligned with enterprise and regulatory requirements.
• Improve platform automation and operational efficiency through DevOps and Infrastructure-as-Code.
• Provide strong platform support and partnership to data engineering, analytics, and business teams.

Data Platform Engineering & Operations

• Design, deploy, and manage large-scale data platform environments (primarily Cloudera and Talend Big Data).
• Operate and optimize Hadoop ecosystem components (HDFS, Hive, Spark, HBase, Impala, Iceberg, Ozone).
• Manage cluster lifecycle including provisioning, scaling, patching, and upgrades.
• Ensure platform uptime, stability, and service reliability through proactive monitoring and maintenance.

Performance, Scalability & Capacity Management

• Monitor workload performance and optimize data processing efficiency.
• Perform capacity planning to support increasing data volumes and usage.
• Troubleshoot performance bottlenecks across compute, storage, and query layers.

Platform Delivery & Execution Planning

• Plan and coordinate execution of platform upgrades, patching, and scaling activities.
• Manage deployment schedules and ensure minimal service disruption.
• Coordinate cross-team dependencies (infrastructure, network, security).
• Track and report execution progress against platform roadmap.

DevOps, Automation & Platform Engineering

• Implement Infrastructure-as-Code using tools such as Terraform and Ansible.
• Design and maintain CI/CD pipelines for platform configuration.
• Automate platform provisioning, deployment, and operational workflows.
• Improve observability through monitoring, alerting, and logging practices.

Security, Governance & Compliance

• Implement and manage platform security controls including authentication (Kerberos), authorization (e.g. Ranger), and encryption.
• Ensure compliance with enterprise data governance, security policies, and regulatory requirements.
• Support auditability, access control, and data protection mechanisms.

Collaboration & Platform Enablement

• Work closely with data engineers, data scientists, and business teams to enable data solution delivery.
• Partner with infrastructure, network, and security teams to ensure platform integration and stability.
• Provide guidance, troubleshooting, and best practices to users of the platform.
• Document platform architecture, processes, and operational procedures.

What You'll Need

• 5- 10 years of experience in data platform engineering, big data operations, or infrastructure engineering.
• Strong hands-on experience managing big data platforms (Cloudera CDP, Hadoop ecosystem).
• Deep understanding of Hadoop components: HDFS, Hive, Spark, HBase, and related services.
• Experience with cluster operations, performance tuning, and troubleshooting.
• Proficiency in automation and scripting (Python, Bash, Terraform, Ansible).
• Experience with CI/CD pipelines and DevOps practices.
• Solid understanding of data platform security and access control concepts.
• Strong analytical and problem-solving skills with a proactive mindset.
• Ability to work in a fast-paced, cross-functional environment.
• Good communication skills with ability to translate technical topics to stakeholders.
• Ownership mentality with focus on platform reliability and continuous improvement.

Good to have

• Experience with Databricks, Talend, or hybrid data platforms.
• Knowledge of containerization and orchestration.
• Experience with streaming platforms (e.g. Kafka) or real-time processing.
• Familiarity with modern data lake-house concepts (e.g. Iceberg, Delta).
• Exposure to cloud or hybrid environments (Azure preferred).

What We'll Provide

• Join us to empower the future of energy.
• You will work in teams that thrive on collaboration for outstanding achievements.
• We have regular wellness programmes to promote healthy living