Databricks Infrastructure and DevOps Engineer

Job Description

Position Overview

We are seeking an experienced Databricks Infrastructure and DevOps Engineer The successful candidate will be responsible for managing and optimising the Databricks infrastructure, implementing robust CI/CD pipelines using government-approved tools, and ensuring compliance with stringent security and governance requirements.

Key Responsibilities

Infrastructure Management:

- Design, deploy, and maintain Databricks clusters and workspaces on AWS GCC environment

- Manage Infrastructure-as-Code (IaC) using Terraform OR Databricks components using Databricks Asset Bundle (DAB)

- Configure and optimise AWS services including S3, IAM, VPC, and security groups specifically for Databricks

- Ensure compliance with Confidential Cloud Eligible classification and Sensitive High data requirements

CI/CD Pipeline Management:

- Develop and maintain CI/CD pipelines using SHIP-HATS tools ecosystem

- Configure GitLab for version control and automated deployment workflows

- Implement code quality gates using SonarQube for static code analysis

- Manage security scanning using Fortify on Demand (FOD) for vulnerability assessment

- Oversee dependency scanning and management using Nexus IQ

- Coordinate with CloudSCAPE for security compliance scanning and remediation

Security and Compliance:

- Implement and maintain Automated Baseline Log Review (ABLR) systems

- Coordinate with GovTech Cyber Security Operations Centre (GCSOC) for security monitoring

- Manage Central Accounts Management (CAM) integration for user access control

- Coordinate and resolve Vulnerability Assessment and Penetration Testing (VAPT) findings.

- Ensure compliance with Personal Data Protection Impact Assessment (PDPIA) requirements

- Implement and maintain Vulnerability Management System (VMS)

Platform Operations:

- Monitor system performance and implement optimization strategies

- Manage secrets and encryption key rotation for data protection

- Configure email notifications for data processing status and error alerts

- Implement and maintain data retention policies

- Support SaaS whitelisting processes and approvals

- Coordinate System Security Acceptance Testing (SSAT) activities

Collaboration and Support:

- Work closely with Data Engineers, Data Analysts and security technical team.

- Support multiple datamarts

- Provide technical guidance for Raw-to-Tech and Tech-to-Business data pipelines

- Collaborate with various government divisions and stakeholders

- Support training environment setup for user onboarding and skill development

Qualifications

Technical Skills:

- 3+ years of experience with Databricks platform administration and optimization

- Strong expertise in AWS services, particularly in GCC environment

- Proficiency in Infrastructure-as-Code tools, preferably Terraform

- Experience with GitLab CI/CD pipeline development and management

- Knowledge of security scanning tools (SonarQube, Fortify, Nexus IQ)

- Understanding of government security frameworks and compliance requirements

- Experience with containerization and orchestration technologies

- Strong scripting skills in Python, Bash, or PowerShell

Security and Compliance:

- Experience with government security clearance processes

- Knowledge of data classification and handling procedures (Sensitive High)

- Understanding of encryption mechanisms and key management

- Familiarity with vulnerability management and remediation processes

- Experience with security monitoring and incident response procedures

Certifications (Preferred):

- Databricks Certified Associate or higher

- AWS Certified DevOps Engineer

Required Experience

- Minimum 3 years of DevOps/Infrastructure engineering experience

- Experience working in government environments, esp in GCC.

- Experience in agile development methodologies

Personal Attributes

- Good communication and collaboration abilities

- Attention to detail and commitment to security best practices

- Strong documentation and knowledge sharing practices

- Proactive approach to system monitoring and maintenance