Our Customer is a tech real-estate marketplace company with a portfolio of the largest and most vibrant real estate and home-related brands on the web and mobile. The company’s brands focus on all stages of the home lifecycle: renting, buying, selling, financing and home improvement. They are committed to empowering consumers with unparalleled data, inspiration and knowledge around homes, and connecting them with the right local professionals to help.
We are seeking an NLP Annotation Judge on a contract basis to support our Customer’s business needs. This role is 100% remote.
This role will join Human In The Loop Operations team and contribute to growing set of Natural Language Processing (NLP) projects. The Annotation Judge will play a crucial role in fine-tuning and calibrating AI/ML processes that handle verbal interactions, between home buyers and real estate agents for example.
Will work directly with experienced Operations Managers and Natural Language experts. You will be responsible for ensuring the quality of a number of possible elements related to the projects, from the accuracy and humanlike traits of automated chatbots answers, to the classification of elements of a conversation annotated automatically or the review of search results based on understanding of complex users queries.
You will play an essential role in defining clear guidelines which may be used by non-native English speakers. Your attention to detail and natural curiosity for language subtleties will be essential for the long term scaling of solutions.
Responsibilities
- Review and/or evaluate annotated data to ensure it meets quality standards and project requirements, and lead teams doing the same.
- Provide feedback and guidance to ML/NLP engineers to improve accuracy.
- Collaborate with NLP engineers and data scientists to optimize model performance based on annotated data.
- Develop and maintain annotation guidelines and best practices.
- Provide training data to ML/NLP engineers to establish Ground Truth data.
- Provide input/examples on Training content, and lead training sessions with Annotation Judges.
- Identify gaps in training content and suggest improvements.
MINIMUM QUALIFICATIONS:
- A Bachelor’s degree in English, Linguistics or a related field.
- 1-3 years of relevant experience.
- Experience working on NLP projects with a focus on transcriptions, localization, translation.
- Experience leading teams, coordinating and leading meetings, conducting trainings, and delivering feedbacks.
- Excellent attention to detail and analytical skills.
- Interpersonal leadership skills and the ability to work effectively in a collaborative team environment.
- Experience following or managing complex processes, and working both independently and with a team to achieve a goal.
- Is a responsible and reliable self-starter who can be comfortable with ambiguity.
PREFERRED QUALIFICATIONS:
- Experience or demonstrated interest in linguistics.
- Experience working on NLP projects with a focus on chatbots, text classification, and named entity recognition.
- Experience with copywriting, UX authoring, content development around voice and brand guidelines.
- Experience with Machine-generated Speech evaluation, Prompt Engineering, Search Results Evaluation, Search Query Categorization.
- Experience following complex processes.
- Experience with tools and software used for NLP annotation.
Perks are available through our 3rd Party Employer of Record (Available upon completion of waiting period for eligible engagements)
Health Benefits: Medical, Dental, Vision, Life (including spouse & child), 401k, STD/LTD, AD&D, and Commuter Benefits program.
An Equal Opportunity Employer. All qualified applicants will receive consideration for employment without regard to race, color, religion, sex, sexual orientation, gender identity, national origin, or protected veteran status and will not be discriminated against on the basis of disability.