ML Evaluation Engineer
5000 - 10000 SGDTemporary
Allegis Group Pune
Summary
We are looking for an ML Evaluation Engineer/Data Scientist to bring statistical rigor and measurement excellence to the evaluation of AI and LLM systems. This role will focus on ensuring that evaluation results are reliable, reproducible, and scientifically valid.
The successful candidate will partner closely with ML Evaluation Engineers and Applied Scientists to answer critical questions such as: "Is this performance improvement real?", "Are our evaluators aligned?", and "Can we trust these results?"
Key Responsibilities:
- Design and execute statistical analyses for AI and LLM evaluation programs.
- Measure and improve inter-annotator agreement between human evaluators.
- Develop sampling methodologies and evaluation datasets to ensure representative results.
- Perform significance testing and statistical validation of model performance changes.
- Analyze golden sets and benchmark datasets to assess model quality and consistency.
- Build dashboards and reporting frameworks to monitor evaluation outcomes and trends.
- Identify sources of evaluation noise, bias, and inconsistency.
- Partner with ML Evaluation Engineers to validate automated judges against human raters.
- Provide data-driven recommendations regarding evaluation quality and model readiness.
Must Have Skills:
- Bachelor's degree in Statistics, Data Science, Computer Science, Mathematics, Machine Learning, or a related field.
- Minimum 2+ years of professional experience in Data Science, Evaluation Science, Applied Statistics, or related disciplines.
- Strong foundation in statistical analysis, hypothesis testing, experimental design, and measurement methodologies.
- Experience working with structured and unstructured data using Python, SQL, and data analysis tools.
- Ability to communicate statistical findings clearly to technical and non-technical stakeholders.
- Experience building analytical dashboards and reporting solutions.
Nice to haves:
- Master's degree or PhD in Statistics, Data Science, Computer Science, Quantitative Social Sciences, or a related discipline.
- Direct experience supporting LLM, AI, NLP, or ML evaluation initiatives.
- Experience measuring agreement between human raters and automated evaluation systems.
- Knowledge of annotation quality frameworks, golden-set construction, sampling theory, and evaluation benchmarking.
- Familiarity with LLM-as-Judge frameworks and AI evaluation platforms.
We regret to inform that only shortlisted candidates will be notified / contacted.
EA registration number : YAP JIA YI, R25157934
Allegis Group Singapore Pte Ltd, Company Reg No. 200909448N, EA Licence No. 10C4544
Vacancy posted 3 days ago
Similar jobs that could be interesting for youBased on the ML Evaluation Engineer in Singapore vacancy
5500 - 8500 SGD
...Scope of Work/ Duties and Responsibilities Train, fine-tune and evaluate Computer Vision models. Prepare and maintain datasets, experiments, evaluation results and model artefacts. Investigate false detections, missed events, tracking errors and model degradation....6000 - 12000 SGD
...Summary We are seeking an AI Evaluation Engineer to help understand, analyse, and improve the performance of advanced AI systems. This role combines... ...fine-tuning. Latent Pattern Recognition: Apply advanced ML techniques (e.g., embedding-based clustering, representation...35 SGD per hour
...leading tech companies, focused on testing, evaluating, and improving AI systems. Participation... ...Not in scope Data labeling, prompt engineering. Production code to ship — you design... ...written. Not a fit Data Science, ML, or Computer Vision engineers without backend...6500 - 11500 SGD
...infrastructure that empowers Machine Learning Engineers to develop the next generation of self-... ...build, and deploy core components of our ML infrastructure platform on Kubernetes.... ...ingestion and processing to model training and evaluation. Write high-quality, maintainable code...6000 - 8200 SGD
...Minimum Required Experience and Qualifications: Practical software engineering experience with services, APIs or system integration. Strong... ...deployment is useful but not mandatory. Experience with ML inference, ONNX, TensorRT or Triton is useful but not mandatory....4500 - 5500 SGD
...highly motivated Fresh Graduates to join our AI & Machine Learning Engineering team. This role offers an exciting opportunity to work on... ...deliver AI-driven products and services. Participate in AI model evaluation, performance optimization, testing, and continuous improvement...50 SGD per hour
...opportunities for leading tech companies, focused on testing, evaluating, and improving AI systems. Participation is project-based, not... ...robust What this is NOT Not data labeling Not prompt engineering Not writing code from scratch - the agent writes most of the...6000 - 8500 SGD
...Overview We are seeking a Applied AI Engineer to embed directly with our business units and... ...AI workloads Maintain & enhance ML/DL models • Own, maintain, and improve production ML/DL models • Retrain, evaluate, and tune models as data and operating conditions...7000 - 9000 SGD
...Learning & Predictive Solutions Develop, evaluate and deploy machine learning and... ...Management & Reliability Apply reliability engineering knowledge to support asset performance and... ...requirements and translate them into appropriate ML and digital solution approaches....13000 - 16000 SGD
...About the Role We are seeking a skilled Machine Learning Engineer to join our AI team and build cutting-edge recommendation systems for... ...Operations teams to translate business requirements into practical AI/ML solutions that enhance overall user experience. Support in-...5500 - 6500 SGD
...seeking an experienced Senior Data Scientist / Machine Learning Engineer to join our dynamic team. As a Senior Data Scientist / Machine... ...deploy models into production environments. Model training and evaluation: Train and fine-tune machine learning models using appropriate...130 SGD
...and managing model lifecycle at scale. The role works closely with ML, backend, and platform teams, contributes to automation... ...experimentation to production. Requires 5+ years of MLOps or DevOps engineering experience, with a track record of operating robust ML infrastructure...7600 - 9200 SGD
...DevSecOps / MLOps Engineer Employment Type: Contract (1 year) Seniority: Senior Consultant (Level 3) Start Date: September 2026... ...We are seeking a senior DevSecOps Engineer to operationalize ML models into production, manage platform security and compliance,...6500 - 11500 SGD
...About the Role We are looking for a Software Engineer to join our ML Infrastructure: Dev Enablement Team . Our mission is to build a frictionless development environment that empowers our researchers and engineers to rapidly innovate on deep learning models for autonomous...5000 - 8000 SGD
...As a Machine Learning Engineer, you will build core ML components. You will work on real production systems from day one, learning how large-scale... ...Build and improve ML components across data, training, evaluation, and inference. Fine-tune and adapt models as part of...- ...Role This is a high-ownership applied ML role focused on speech in real... ...be comfortable where: Requirements and evaluation criteria evolve. Data is messy, multi-... ...switching scenarios. Collaborate with engineering to integrate models into production pipelines...
6000 - 10000 SGD
...and managingmodel lifecycle at scale. The role works closely with ML, backend, and platformteams, contributes to automation frameworks... ...to production.Requires 5+ years of MLOps or DevOps engineering experience, with a trackrecord of operating robust ML infrastructure...11000 - 14000 SGD
...building AI-native research systems for professional users. We are looking for a Lead AI Research Engineer to establish the scientific and technical foundations for evaluating our LLM and agent systems. This role operates at the intersection of applied AI research, experimental...5750 - 8500 SGD
...& Climate) to help deliver IPUR’s research programme on risk perceptions and communications. Specifically, the RF will work on the evaluation of risk communication and resilience interventions. Evaluations will be conducted in experimental and real-world settings employing...- ...The Opportunity We are looking for sensible and inquisitive engineers and scientists with a strong background in the fields of artificial... ...opportunities to develop your technical expertise in many subfields of AI, ML, HPC, geoscience, engineering, etc. Have your work...
- ...Job Summary We are looking for an experienced AI Engineer with strong expertise in Generative AI , AI/ML , or Microsoft Copilot/Agentic AI development.... ...Analytics, or Recommendation Systems. Model training, evaluation, deployment, and MLOps. Option 3 – Microsoft...
- ...effective use and proper maintenance. Adapt and identify gaps to improve and enhance our fragrances. Active participation in the Evaluation Sessions (actual projects, fragrance stability in actual product matrix, new product launches, etc) on categories assigned. Good...
8000 - 18000 SGD
...Role Overview｜岗位概述 We are looking for a Senior AI Systems Engineer with deep experience in multi-model systems, model routing, AI evaluation, and production AI infrastructure. The role will focus on designing systems that can compare, select, and operate multiple foundation...11000 - 18000 SGD
...customer teams. Mentor and guide junior consultants and customer engineers. Engage in pre-sales activities, including scoping, technical... ..., governance, and lifecycle management. ~ Experience with AI/ML frameworks and libraries such as PyTorch, TensorFlow, Scikit-learn...16000 - 25000 SGD
...About the Role We are looking for exceptional AI Engineers to build the next generation of AI infrastructure and Machine Learning Systems(MLSys). This role focuses on large-scale system infrastructure rather than model research. You will work on the core foundations...6000 - 12000 SGD
...currently looking to hire Assistant Director/Senior Manager, AI ML Operations. This is an exciting opportunity to expand your skill... ...serve asa four-eyes deployment approver. Lead a small team of MLOps engineers across HQand offshore. How You willMake An Impact: Own AI/...6000 - 9500 SGD
...covering both manpower and financial resources. This includes planning, managing, and allocating budget and manpower resources as well as evaluating budget requests for various programmes; • Funding Policy Formulation: Formulate corporate-level funding policies and funding...5000 - 20000 SGD
...professionals across multiple seniority levels, from Fragrance Evaluators and Fragrance Development Managers (FDM / CFM) up to Associate... ...Bachelor's or Master's degree in Chemistry, Perfumery, Chemical Engineering, Cosmetic Science, or a related scientific/creative field. Direct...- ...degrade quality, latency, or cost. Work with runtime engineers to identify bottlenecks and verify improvements;... .... How you will stand out: ~5+ years in ML infrastructure, performance engineering, model evaluation, QA automation, or backend testing for production systems...
150 - 220 SGD per hour
...comfortable having your judgment reviewed and calibrated against peers. Preferred ~ Prior experience with AI training , evaluation, or human-data projects. Application Process (Takes 20–30 mins to complete) Submit your resume or relevant...Remote job
Do you want to receive more vacancies?
Subscribe and receive similar vacancies to ML Evaluation Engineer. Be the first to apply!
