AI Data Annotation Quality Reviewer
Gramian Consulting
Leads quality review for AI training data at Gramian Consulting. Oversees reviewers, audits, feedback, training, and process improvements across remote project teams.
खोलें
mpathic मानव-एआई और मानव-मानव संवादों को समझने और बेहतर बनाने के लिए एआई सुरक्षा और विश्लेषण सॉफ़्टवेयर विकसित करती है। इसके SaaS उत्पाद और API भाषण-से-पाठ प्रतिलेखन, व्यक्तिगत पहचान योग्य जानकारी को हटाने, व्यवहार और चेकलिस्ट विश्लेषण, बेंचमार्किंग डेटासेट, रेड-टीमिंग और इसके Observing Agent API के माध्यम से डोमेन-आधारित सुरक्षा उपायों का समर्थन करते हैं। कंपनी स्वास्थ्य सेवा, जीवन विज्ञान, क्लिनिकल अनुसंधान, बीमा और एंटरप्राइज़ टीमों को सेवा देती है, जो सहानुभूति बढ़ाने, पक्षपात कम करने और अधिक सुरक्षित व अनुपालन-सम्मत संवादों को समर्थन देने के लिए काम करती हैं।
Healthcare AI/ML Evaluation Reviewer
Review evidence for healthcare AI applications in mpathic’s regulatory sandbox evaluations. Assess model validation, reliability, safeguards, and monitoring for clinical use.
विवरण
- Assess technical evidence for healthcare AI applications, covering performance, validation, generalizability, reliability, monitoring, and safeguards.
- Review validation methods, reference standards, sample sizes, statistical analyses, metrics, operating thresholds, and uncertainty.
- Examine data quality and representativeness, including leakage, subgroup differences, and generalizability across patient groups and care settings.
- Test technical reliability, robustness, reproducibility, failure modes, and behavior on atypical or out-of-distribution inputs.
- Review evaluation plans, benchmarks, and success criteria for pilot deployments.
- Assess technical safeguards, escalation paths, and human oversight processes.
- Review monitoring and change-management plans for detecting performance decline, tracking model behavior, managing updates and interventions, and setting rollback criteria.
- Work with clinicians and technical specialists to address cross-disciplinary concerns.
- Record strengths, limitations, evidence gaps, and technical concerns using the standard review framework.
- Give clear, evidence-based recommendations and specify needed evidence or changes.
- Take part in reviewer calibration and quality assurance.
- Maintain independence and confidentiality, following conflict-of-interest and data-handling requirements.
आवश्यकताएँ
- At least three years of professional or applied research experience in machine learning, data science, biostatistics, clinical informatics, or a related field, including direct work evaluating or validating predictive models or AI systems.
- Understanding of model evaluation methods, including suitable performance metrics, validation design, statistical uncertainty, threshold selection, and common sources of bias or methodological error.
- Experience evaluating model performance on independent or held-out data and judging whether results generalize beyond the evaluation dataset.
- Practical experience evaluating at least one relevant AI system type, such as clinical NLP, large language models, conversational AI, medical imaging, computer vision, or tabular predictive models.
- Ability to critically review technical research, evaluation reports, or validation studies and identify methodological weaknesses or unsupported conclusions.
- Strong analytical writing skills and the ability to explain complex findings clearly to nontechnical audiences.
- Ability to work independently and contribute to multidisciplinary expert review and calibration.
- Willingness to complete confidentiality agreements, conflict-of-interest disclosures, and reviewer training.
- Experience evaluating AI or ML systems in healthcare, clinical decision support, digital health, or another high-stakes setting.
- Familiarity with clinical AI evaluation and reporting frameworks such as TRIPOD+AI, STARD-AI, CONSORT-AI, DECIDE-AI, or model cards.
- Experience evaluating generative AI, LLM-based systems, or autonomous agents, including multi-turn behavior, reliability, hallucinations, and safeguards.
- Experience with robustness testing, adversarial evaluation, red teaming, or systematic failure-mode analysis.
- Knowledge of production monitoring, distribution shift, model drift, version control, and change management.
- Experience designing or evaluating human-in-the-loop systems or comparing model performance with human expert judgments.
- Familiarity with FDA guidance on AI-enabled medical devices, clinical decision support, or software as a medical device.
- Experience conducting independent technical reviews, scientific peer reviews, grant reviews, or formal technology evaluations.
- Experience working with clinicians, researchers, or regulatory stakeholders on AI evaluation or deployment.
लाभ
- Contract work with payment per review.
- Reviewer orientation covering the evaluation framework and project requirements before the first assignment.
संबंधित नौकरियाँ
English AI Safety Specialist
Mercor51 – 200 कर्मचारी
Review English-language conversations and assess AI responses for Mercor. Apply project guidelines and provide structured feedback to support training frontier AI systems.
खोलें
Remote AI Assistant User Feedback Research Participant
Mercor51 – 200 कर्मचारी
Use Meta Muse or Instinct and rate its proactive messages as part of Mercor’s AI training study. Submit weekly conversations after removing personal details.
खोलें
Cantonese Chinese Linguistic Annotator
Sigma AIWork remotely as a freelance Cantonese-English annotator on Sigma AI data collection projects. Tasks include annotation, transcription, validation, and voice recording.
खोलें
Healthcare Subject Matter Expert, AI Model Evaluation
Volga Partners
ऑस्ट्रेलिया
कनाडा
यूनाइटेड किंगडम
भारत
न्यूज़ीलैंड
फ़िलिपींस
संयुक्त राज्य$35 – $150 / घंटा
रिमोटअनुबंधमध्य स्तरवरिष्ठआवश्यक भाषा: अंग्रेज़ीकृत्रिम बुद्धिमत्ताVolga Partners seeks physicians, nurse practitioners, and physician assistants to assess the clinical accuracy, reasoning, and safety of AI-generated medical content. Remote, project-based contract assignments vary according to project needs.
खोलें