As I continue to explore what’s needed to build a robust LLM based planning agent, one thing I’ve been focusing on lately was how to improve the quality of the plan itself. One dimension I’ve been exploring is how refining the objective of the job and tasks themselves to provide a clearer and more adjusted plan.

I’ll go over some of the things that provided improvements in that area.

Improving Contextual Information

What I was noticing in certain situations, was the plan generated lacked contextual information that would help provide the planner with a better understanding of the job to accomplish.

Here’s an example of how would tap into the objective refinement capability specifically through an API call:

{
    "contexts":[
        {
            "type":"Domain","description":"Application Development"
        },
        {
            "type":"Analysis Type","description":"Content Review"
        },
        {
            "type":"Subject","description":"Application User Feedback"
        },
        {
            "type":"Knowledge Level","description":"beginner"
        }
    ],
    "objective": "Capture and prioritize improvements for an application. We need to identify which dimension of the application we need to focus on. i.e. UX, performance, etc. We're working on a claim intake application.",
    "validateOutput":true,
    "provideReasoning":true
}

As you can see in the example above, additional context is provided with the original objective. Now let’s review how the refiner did in improving the original objective:

{
    "objective": "Capture and prioritize improvements for a claim intake application based on user feedback. First, gather user feedback through surveys, interviews, and usability testing to identify pain points and areas for improvement. Focus on specific dimensions of the application such as User Experience (UX), performance, and functionality. Clearly define what each dimension entails: UX includes aspects like ease of navigation and visual design; performance covers load times and responsiveness; functionality involves the accuracy and completeness of features. Once feedback is collected, categorize the issues under these dimensions. Prioritize the improvements by considering factors such as the frequency of the issue, the impact on user satisfaction, and the feasibility of the solution. Create a detailed action plan outlining the steps needed to implement the prioritized improvements, including timelines and responsible team members.",
    "reasoning": "The refined objective is superior because it provides clear, step-by-step instructions suitable for a beginner. It explains the process of capturing and prioritizing improvements in detail, including how to gather user feedback and what specific dimensions to focus on. It also defines potentially unfamiliar terms like UX and performance, ensuring there is no ambiguity. Additionally, it outlines how to categorize and prioritize the feedback, making the objective actionable and comprehensive.",
    "outputValidations": [
        {
            "reasoning": "Let's evaluate the submission based on the given criterion:\n\nClarity: Is the objective sufficiently clear and detailed given the knowledge level of the executor?\n\nStep 1: Identify the main objective\nThe submission clearly states the main objective: to capture and prioritize improvements for a claim intake application based on user feedback.\n\nStep 2: Assess the level of detail\nThe submission provides a detailed step-by-step process:\n1. Gather user feedback through various methods\n2. Focus on specific dimensions (UX, performance, functionality)\n3. Define what each dimension entails\n4. Categorize issues under these dimensions\n5. Prioritize improvements based on specific factors\n6. Create a detailed action plan\n\nStep 3: Evaluate the clarity of instructions\nThe submission offers clear instructions on how to proceed with each step. For example, it specifies methods for gathering feedback (surveys, interviews, usability testing) and provides examples of what each dimension includes.\n\nStep 4: Consider the knowledge level of the executor\nThe submission assumes a basic understanding of application development and improvement processes. It doesn't delve into highly technical details, making it accessible to a wide range of team members.\n\nStep 5: Assess the actionability of the plan\nThe submission concludes with a clear instruction to create an action plan with specific elements (steps, timelines, responsible team members), making it actionable for the executor.\n\nConclusion:\nBased on this analysis, the submission meets the criterion for clarity. It provides a clear, detailed, and actionable plan that should be understandable to the intended executor.\n\nY",
            "score": 1,
            "value": "Y"
        }
    ]
}

From the example above, you can see a few different things. The first thing is the refined objective itself. Not how much more detailed it is versus the original objective. The second thing is the reasoning as to why the refined objective is better than the original. This can provide some guidance to understand why the objective was changed. Lastly, you can see how the objective was assessed. This was done through multiple validation steps that were dynamically generated based on the context and the objective itself.

Now, let’s adjust our skill level.

Adjusting Tasks and Steps to Skill Level

In the example above, we provided the refiner with a context that stated the executor of the task had a beginner level. Let’s reissue the same request but this time we’ll specify the expert knowledge level. Here’s the response:

{
    "objective": "Capture and prioritize improvements for an application. We need to identify which dimension of the application we need to focus on. i.e. UX, performance, etc. We're working on a claim intake application.",
    "reasoning": "",
    "outputValidations": [
        {
            "reasoning": "Let's analyze the submission against the given criterion:\n\nClarity: Is the objective sufficiently clear and detailed given the knowledge level of the executor?\n\nStep 1: Consider the knowledge level of the executor.\nThe context indicates that the knowledge level is \"expert\". This means the submission should be clear and detailed enough for someone with extensive experience in the field.\n\nStep 2: Examine the clarity of the objective.\nThe submission states the main objective: \"Capture and prioritize improvements for an application.\" This provides a clear overall goal.\n\nStep 3: Assess the level of detail provided.\nThe submission goes further by specifying that they need to \"identify which dimension of the application we need to focus on\" and provides examples like \"UX, performance, etc.\" This adds more specificity to the task.\n\nStep 4: Evaluate the context provided.\nThe submission mentions that they're working on a \"claim intake application.\" This provides important context for the task at hand.\n\nStep 5: Consider if an expert would have enough information to proceed.\nGiven that the executor is an expert in application development and user feedback analysis, they would likely understand the implications of focusing on different dimensions of the application and the importance of prioritizing improvements based on user feedback for a claim intake application.\n\nConclusion: The submission provides a clear objective with sufficient detail for an expert to understand and begin the task. It outlines the main goal, specifies the need to identify focus areas, and provides context about the type of application. An expert in the field would likely have enough information to proceed with the task effectively.\n\nY",
            "score": 1,
            "value": "Y"
        }
    ]
}

As you can see above, the original objective remains intact and you can see why it hasn’t changed based on the explanation provided in the outputValidations. As the executor has an expert level of expertise, the refiner deemed reasonable the original objective as it was stated.

As a final example, let’s sit right in the middle with an intermediate level:

{
    "objective": "Capture and prioritize improvements for a claim intake application by conducting a thorough content review and analysis of user feedback. Focus on identifying specific dimensions such as User Experience (UX) and performance. Utilize user feedback, application metrics, and industry best practices to determine the most critical areas for improvement. Document each identified improvement with detailed descriptions, potential impact, and prioritization rationale. Ensure the process includes clear steps for capturing feedback, analyzing data, and making informed decisions on prioritization.",
    "reasoning": "The refined objective provides a clear and detailed approach suitable for an intermediate-level executor. It specifies the type of application, the dimensions to focus on, and the methods to use for capturing and prioritizing improvements. By including steps for capturing feedback, analyzing data, and documenting improvements, the objective ensures a comprehensive and structured process. This eliminates ambiguity and provides a clear path for execution.",
    "outputValidations": [
        {
            "reasoning": "Let's evaluate the submission against the given criterion:\n\nClarity: Is the objective sufficiently clear and detailed given the knowledge level of the executor?\n\nStep 1: Identify the main objective in the submission.\nThe main objective is to capture and prioritize improvements for a claim intake application.\n\nStep 2: Assess the clarity of the process outlined.\nThe submission provides a clear process:\n- Conduct a thorough content review\n- Analyze user feedback\n- Focus on specific dimensions like UX and performance\n- Utilize user feedback, application metrics, and industry best practices\n- Document improvements with descriptions, impact, and prioritization rationale\n- Include steps for capturing feedback, analyzing data, and making informed decisions\n\nStep 3: Evaluate the level of detail provided.\nThe submission offers a good level of detail, explaining what needs to be done (capture and prioritize improvements), how to do it (through content review, user feedback analysis, etc.), and what to focus on (UX, performance).\n\nStep 4: Consider the knowledge level of the executor.\nThe submission assumes some familiarity with application development and improvement processes, which is reasonable for someone tasked with this kind of project.\n\nStep 5: Determine if any crucial information is missing.\nThe submission covers the main aspects of the task without leaving out any critical information. It provides a clear framework for approaching the improvement process.\n\nConclusion: The submission meets the clarity criterion. It provides a clear, detailed, and well-structured approach to capturing and prioritizing improvements for the claim intake application, suitable for an executor with relevant knowledge in the field.\n\nY",
            "score": 1,
            "value": "Y"
        }
    ]
}

Other Use Cases

One thing I’ve also explored that leverages this capability, is how the refiner can be used during a user conversation to enhance the quality of the answers provided by the chatbot. For example, sometimes the question submitted by the user are somewhat ambiguous or in other cases, they involve multiple independent questions. By combining the refiner with the job planner during a conversion, you can handle better certain situations that may come up during a conversation.

It’s still very early stage in that area as there are still some things to solve. For example, controlling the level of depth in the reasoning and clarification is still something to be solved. Let’s just say the chatbot dials it to 11 by providing a bit more than what the user might have expected.

Conclusion

That’s a wrap for this week’s post! Using this approach iteratively has impacted positively the quality of the plans generated by expanding better on what needs to be done but in way that’s more adjusted to task at hand. This approach is by no mean the end all be all but it’s a decent improvement over the first version.