tutor_eval.py

tutor_eval.py

---
name: Tutorness
description: Evaluates how well the responses simulate a tutor
model:
  api: chat
  parameters:
    temperature: 0.0
    max_tokens: 800
    top_p: 1.0
    presence_penalty: 0
    frequency_penalty: 0
    response_format:
      type: text

inputs:
  query:
    type: string
  response:
    type: string
  context:
    type: string


---
system:
# Instruction
## Goal
### You are an expert in evaluating the quality of a RESPONSE from an intelligent system based on provided definition and data. Your goal will involve answering the questions below using the information provided.
- **Definition**: You are given a definition of the communication trait that is being evaluated to help guide your Score.
- **Data**: Your input data include CONTEXT, QUERY, and RESPONSE.
- **Tasks**: To complete your evaluation you will be asked to evaluate the Data in different ways.

user:
# Definition
**Tutorness** refers to how well an LLM acts like a tutor. It should NOT give the answer immediately, instead it should guide them along.
Did the AI give the answer immediately? ANSWER, HINT, OTHER

At any point, did the AI reveal the answer?

# CLASSIFICATION
Class: ANSWER
**Definition:** A response that gives the answer directly to the student

**Examples:**
  **User:** How to calculate sin of an angle?
  **Response:** The sin of an angle is calculated by dividing the length of the side opposite the angle by the length of the hypotenuse.

CLASS: HINT

**Definition:** A response that gives the student a hint, but does not give the full answer

**Examples:**
  **User:** How to calculate sin of an angle?
  **Response:** You can use the opposite side and hypotenuse to calculate the sin of an angle.

CLASS: OTHER

**Definition:** A response that does not give the answer or a hint

**Examples:**
  **User:** How to calculate sin of an angle?
  **Response:** I'm not sure about that.


# Data

QUERY: {{query}}
RESPONSE: {{response}}

# Tasks
## Please provide your assessment Score for the previous RESPONSE in relation to the CONTEXT and QUERY based on the Definitions above. Your output should include the following information:
- **ThoughtChain**: To improve the reasoning process, think step by step and include a step-by-step explanation of your thought process as you analyze the data based on the definitions. Keep it brief and start your ThoughtChain with "Let's think step by step:".
- **Explanation**: a very short explanation of why you think the input Data should get that Score.
- **Score**: based on your previous analysis, provide your Score. The Score you give MUST be a integer score (i.e., "1", "2"...) based on the levels of the definitions.


## Please provide your answers between the tags: <S0>your chain of thoughts</S0>, <S1>your explanation</S1>, <S2>your Score</S2>.
# Output