Localization Academy

NLP Annotation Judge

⚠️ This job was posted more than 2 weeks ago and may no longer be active

Job Description

The NLP Annotation Judge will play a crucial role in fine-tuning and calibrating AI/ML processes that handle verbal interactions, between home buyers and real estate agents for example. Will work directly with experienced Operations Managers, ML Engineers, Data Scientist, Software

Developers and Natural Language experts. Will be responsible for ensuring the quality of a number of possible elements related to projects. From the accuracy and humanlike traits of automated chatbots answers, to the classification of elements of a conversation annotated automatically or the review of search results based on understanding of complex users queries.

Additionally, will play an essential role in defining clear guidelines which may be used by non-native English speakers. Attention to detail and natural curiosity for language subtleties will be essential for the long term scaling of solutions.

This is a 12 month contract position (with possibility to extend) that will be remote (United States). The pay range for this role is $18-19.60/HR.


● Review and/or evaluate annotated data to ensure it meets quality standards and project requirements.

● Provide feedback and guidance to ML/NLP engineers to improve accuracy.

● Collaborate with NLP engineers and data scientists to optimize model performance based on annotated data.

● Develop and maintain annotation guidelines and best practices.

● Provide training data to ML/NLP engineers to establish Ground Truth data.

● Provide input/examples on Training content.

● Identify gaps in training content and suggest improvements.


● Experience working on NLP projects with a focus on transcriptions, localization, translation.

● Ability to work effectively in a collaborative team environment.

● Experience with tools and software used for NLP annotation is a plus.

● Experience with copywriting, UX authoring, content development around voice and brand guidelines.

● Experience with Machine-generated Speech evaluation, Search Results Evaluation, Search Query Categorization

The pay for this W2 role is $18-19.60/hr. This position may be eligible for PTO, health and dental insurance, and/or 401(k) benefits upon meeting certain length of service and hours requirements.

Mathys+Potestio values applicants of all backgrounds and experiences. We do not discriminate based on race, color, national or ethnic origin, ancestry, age, religion or religious creed, disability or handicap, sex or gender, gender identity and/or expression, sexual orientation, military or veteran status, genetic information, or any other characteristic protected under applicable federal, state, or local law.