In the realm of conversational AI, user inquiries can often be high-level and complex, making it challenging to provide high-quality answers with a simple Retrieval Augmentation Generation (RAG) approach. To address this, I’ve worked on developing new capabilities that break down the inquiry process into detailed, actionable steps, ensuring thorough and contextually accurate responses.

Tackling High-Level Questions with Precision

When users ask high-level questions, the complexity of the inquiry often necessitates a more sophisticated approach. The new capabilities determine key considerations for answering such inquiries, developing a small plan with multiple tasks and objectives. This method ensures that each aspect of the question is addressed comprehensively.

The Multi-Task Planning Approach

1. Developing a Plan: The first step involves creating a plan consisting of several tasks and objectives. This plan is tailored to break down the user inquiry into manageable parts.

2. Deriving Questions: Once the plan is in place, specific questions are derived from the tasks. These questions target the key areas of the inquiry, ensuring that all necessary information is gathered.

3. Querying Reference Material: The derived questions are then used to query relevant reference materials and documents. This step ensures that the responses are grounded in accurate and authoritative sources.

4. Summarizing and Contextualizing: After the questions have been answered, the information is summarized and contextualized to fit the original user inquiry. This final step ensures that the response is comprehensive, coherent, and directly addresses the user’s needs.

Example: Credit Card Benefit for Mobile Phone Damage

Let’s look at the user inquiry:

User Inquiry: “Am I covered if I dropped my phone and it gets damaged?”

Now let’s review the response using a more traditional RAG approach:

As you can see, the RAG approach provided a reasonable answer but it lacks a little bit of nuance around conditions or exclusions that might apply.

Let’s see if the new approach does a little bit better:

From the above you see how the answer was more nuanced. Let’s see a bit more what happened behind the scene:

Step 1: Developing a Plan

Based on the user message and contextual information:

{
  "message": {
    "role": "user",
    "name": "",
    "content": "Am I covered if I dropped my phone and it gets damaged?",
    "structuredContentType": "",
    "structuredContent": "",
    "sources": [],
    "userVisible": true
  },
  "message_contexts": [
    {
      "type": "Conversation Type",
      "description": "Content Question"
    },
    {
      "type": "Content Identified",
      "description": "true"
    },
    {
      "type": "Knowledge Level",
      "description": "advanced"
    },
    {
      "type": "Content Subject",
      "description": "credit card benefit wording"
    }
  ]
}

A plan to answer the question was generated automatically using the same logic that was explained in my previous posts LLM-Based Job Planner and Executor Agents For Task Management and Execution and LLM Based Planner: Better Plans Through Better Objectives . Here’s what that looks like:

{
  "output": [
    {
      "id": "08a9a20b-7108-4db8-b295-788f5f36e558",
      "name": "Identify Coverage Terms",
      "reasoning": "To determine if phone damage is covered, we need to understand the specific terms and conditions of the credit card benefits.",
      "objective": "Extract the terms and conditions related to phone damage coverage from the credit card benefit wording.",
      "importance": "critical",
      "type": "Content Extraction",
      "steps": [],
      "requirements": []
    },
    {
      "id": "2f36612e-e111-4eb1-9c07-14be8a64cd42",
      "name": "Verify Coverage Conditions",
      "reasoning": "Knowing the terms is not enough; we need to verify if the conditions under which the phone was damaged are covered.",
      "objective": "Check if accidental damage, such as dropping the phone, is included under the coverage terms.",
      "importance": "critical",
      "type": "Content Assertion",
      "steps": [],
      "requirements": []
    },
    {
      "id": "2170da04-e0e5-4719-9341-c66bad1d170a",
      "name": "Determine Exclusions",
      "reasoning": "It's important to know if there are any exclusions that might prevent coverage for the damaged phone.",
      "objective": "Identify any exclusions in the credit card benefit wording that would negate coverage for phone damage.",
      "importance": "important",
      "type": "Content Extraction",
      "steps": [],
      "requirements": []
    },
    {
      "id": "9bc5dc31-f22f-4538-befb-c8d879c448be",
      "name": "Summarize Coverage Details",
      "reasoning": "A clear summary helps in understanding the overall coverage without going through the entire document.",
      "objective": "Summarize the coverage details, including what is covered, what is not covered, and any conditions or exclusions.",
      "importance": "important",
      "type": "Content Summarization",
      "steps": [],
      "requirements": []
    }
  ]
}

Step 2: Deriving Questions

Now that we have a game plan to answer the question, the engine will then go through the plan to generate a series of questions to achieve each task objectives. As an example, given the following task objective:

{
  "objective_contexts": [
    {
      "type": "Conversation Type",
      "description": "Content Question"
    },
    {
      "type": "Content Identified",
      "description": "true"
    },
    {
      "type": "Knowledge Level",
      "description": "advanced"
    },
    {
      "type": "Content Subject",
      "description": "credit card benefit wording"
    },
    {
      "type": "User Question",
      "description": "Am I covered if I dropped my phone and it gets damaged?"
    }
  ],
  "objective": "Extract the terms and conditions related to phone damage coverage from the credit card benefit wording.",
  "reasoning": "To determine if phone damage is covered, we need to understand the specific terms and conditions of the credit card benefits."
}

The following questions were generated:

{
  "output": [
    {
      "question": "What types of phone damage are covered under the credit card benefits?",
      "reasoning": "Understanding the specific types of phone damage that are covered is crucial to determine if accidental drops are included."
    },
    {
      "question": "Are there any exclusions or limitations to the phone damage coverage?",
      "reasoning": "Identifying any exclusions or limitations helps in understanding the scope and boundaries of the coverage."
    },
    {
      "question": "What is the process for filing a claim for phone damage?",
      "reasoning": "Knowing the claim process is essential for utilizing the coverage effectively."
    },
    {
      "question": "Is there a deductible or any out-of-pocket expense for phone damage claims?",
      "reasoning": "Understanding any potential costs involved in making a claim is important for financial planning."
    },
    {
      "question": "What documentation is required to file a phone damage claim?",
      "reasoning": "Knowing the required documentation ensures that the claim can be processed smoothly and without delays."
    },
    {
      "question": "Is there a maximum coverage limit for phone damage under the credit card benefits?",
      "reasoning": "Identifying the maximum coverage limit helps in understanding the extent of financial protection provided."
    },
    {
      "question": "Does the phone damage coverage apply to all types of phones or only specific models?",
      "reasoning": "Clarifying whether the coverage is universal or restricted to certain phone models is important for applicability."
    }
  ]
}

Step 3: Curating the questions

Now that we have a fairly extensive list of questions that might be relevant to answer the user inquiry, there might be some redundant questions that would have been generated from each task/objective. In order to distill that a bit, the full list of questions is taken and re-aligned with the original user question.

This again is a process that happens automatically.

Step 4: Interrogating Content

With a good set of questions in hand, it’s time to query the content using a more typical RAG approach in order to answer each question:

{
  "retrieval_query": {
    "query": "What types of phone damage are covered under the credit card benefits?",
    "filter": {
      "sourceParent": {
        "$eq": "/contents/2df077cc-dded-4f3e-8887-de4b117e4439"
      }
    },
    "topResultsCount": 5
  },
  "vector_store_api_key": null,
  "vector_store_environment": null,
  "index_name": "knowledge-chunks-v3",
  "embedding_version": "v3"
}

For which we can get the following answer:

{
  "question": "What types of phone damage are covered under the credit card benefits?",
  "answer": "Coverage for new mobile devices purchased with the card includes scenarios where the device is lost, stolen, accidentally damaged, or experiences mechanical failure during the coverage period. To qualify for this coverage, the full cost of the mobile device must be charged to the Tangerine World Mastercard. \n\n",
  "sources": [
    "/contents/2df077cc-dded-4f3e-8887-de4b117e4439?statementId=199dbb43-2d20-4b8e-829c-4edca7f2417a"
  ]
}

This process is repeated for each question to be answered.

Step 5: Summarize Answers and Contextualize

Before we can provide the user with our final answer, we’ll leverage the knowledge we accumulated from the multiple questions and contextualize it using the user original question to provide the final answer that was shown earlier:

If you dropped your phone and it gets damaged, you may be covered under the credit card’s phone damage benefits, as the coverage includes “accidentally damaged” devices. However, the specific details regarding coverage for damage caused by dropping the phone are not explicitly stated in the provided information, leaving some uncertainty.

To be eligible for coverage, the phone must have been purchased with the credit card, and you must file a claim with the insurer, obtaining their approval before any repairs or replacements. Keep in mind that there is a deductible, and the maximum reimbursement amount is $1,000, based on the lesser of the repair or replacement cost, minus the deductible. Additionally, there are limits on the number of claims you can make within certain time frames.

That last summarization step ties everything together to ensure the objective is met and requires sound reasoning.

Conclusion

The new multi-task planning approach ensures that high-level user inquiries are addressed with the depth and accuracy they require. By breaking down complex questions into manageable tasks, deriving specific questions, and thoroughly querying reference materials, we provide users with detailed and contextually relevant answers. Do keep in mind this still relies on quality reference material in order to provide a good answer. If we look at the following example, you can see additional nuance was provided towards the end:

Yes, you would be covered if you break a tooth, as the insurance policy specifically includes coverage for accidental dental injuries. If the break is due to an accident that causes a medical emergency, the policy provides coverage for dental treatment up to $2,000. Additionally, it offers up to $200 for emergency relief of dental pain. However, it’s important to note that the coverage excludes non-medical emergencies and certain treatments, such as cosmetic procedures or those related to pre-existing conditions. To file a claim, you would need to follow specific steps, including reporting the emergency, submitting a claim form, and providing proof of payment and medical records.

Stay tuned as we continue to enhance our capabilities, making conversational AI more responsive and intelligent than ever before.