-
Worldwide
-
Ongoing
-
Fixed rate per approved asset
-
Annotation
LLM
Evaluation
Description:
In this task, participants are required to rank prompts in natural language based on different criteria provided.
Their participation includes:
- Analyzing user requests. Identifying the user’s intent. Determining what the response needs to include to fulfil the user’s requirements in the best possible way. Identifying if there are any valid reasons to skip the task.
- Answering additional questions about the user request. Answering questions that will provide more insights about the user request, such as the number of instructions it contains and its clarity.
- Analyzing & grading the first response. Rating the first response in regard to various evaluation criteria such as truthfulness, concision, satisfaction and other.
- Rating the other responses. Comparing and choosing a response to be ranked.
- Writing a comment/reason for the chosen response.
- Submitting the task.
Purpose:
Participants will act as language leaders, meticulously analyzing each request to identify its core intent. Their mission is then to evaluate the perfect response – a response that not only satisfies the user’s needs but also leaves them feeling understood and empowered.
