While working on research and analysis capabilities for my personal project, I started to dive into the possibility of automating further the generation of what needs to be done in order to reach a particular objective.

In the current iteration of the Content Analyzer (see Content Analyzer posts), you articulate the objective in terms of what you want to accomplish with the content but I wanted to see what would happen if you took a step back and used an LLM to generate a game plan.

Let’s delve into the key highlights of how that works in the current experiment I’m working on.

Job Planner Agent

Objective-Driven Job Planning

The core functionality of the Job Planner Agent is to generate a comprehensive job plan given a specific objective. This ensures that every task is aligned with the overarching goal, providing a structured approach to achieving desired outcomes.

Not only do you need to generate a plan a human can understand but you also need to do it in a way where you can hand that off to a subsequent step that can execute it (the executor agent); so it needs to be logically sound and well structured.

Structured Task and Step Hierarchy

A job plan consists of a series of tasks, each of which can be broken down into a series of steps. This hierarchical structure allows for meticulous planning and execution, ensuring that every aspect of the job is covered.

Each task will be further divided into actionable steps, such as gathering data, analyzing content, extracting data points, asserting data, etc.

Every task and step comes with a detailed description, reasoning, objective, importance level, type (e.g., Content Research, Content Extraction, Content Assertion), and a set of requirements. This comprehensive detailing ensures clarity and facilitates smooth execution.

Objective Validation Using LLM

The Job Planner Agent will validate each task and step to ensure that the objectives are clear and complete. This validation process leverages another LLM to minimize potential biases, enhancing the accuracy and reliability of the plan.

Seamless Transition to Plan Execution

Once the plan is generated, it can be seamlessly passed to an executor agent, which will interpret the plan structured data and execute the plan as defined. This transition from planning to execution ensures a cohesive workflow, reducing the risk of errors and improving efficiency.

Using the Planner Agent

Now that we covered the basics, let’s see how we can get started with the planner agent. First, you need to provide the overall objective of what you’re trying to achieve.

For this example we’ll use the following:

POST /job/generate-plan
{
    "objective":"I want to increase brand awareness for my company who designs portable power stations."
}

Triggering this API call will start the planning process, which will iteratively build a detailed plan to achieve the objective.

Let’s look at the result (shortened to keep the post a bit lighter):

As you can see above, the planner created a fairly comprehensive plan involving multiple tasks and steps to achieve the marketing goal.

Using the Executor Agent: An Early Look

As I’m still iterating on the plan generation to improve quality/relevance, I’ve started to look into the implications of automating the execution of the generated plan. This is a fairly complex problem to tackle but like in everything, I like to crawl first. For my first stab at this, I chose the execution of tasks of type Content Querying. This involves the following:

• Based on the step objective, generate a series of questions
• Using the generated questions, use retrieval augmented generation (RAG) to query content knowledge chunks in order to formulate an answer to each question (at least try to)

To test that, I simply pass the original job plan and pass which task/step (of type Content Querying) I’d like to execute to control things better while building the capability.

Here’s an example of how would call the API method to do that which show an shortened extract of the plan/task/step to illustrate:

POST /job/execute-plan-task-step?task-id=1&step-id=1

{
    "name": "",
    "description": "",
    "objective": "<objective>",
    "plan": {
        "tasks": [
            {
                "rank": 1,
                "name": "Review Credit Card  Benefits Wording",
                "description": "Examine the benefit wording related to the extended warranty.",
                "reasoning": "Understanding the benefits terms and conditions is essential to determine if the claim is covered.",
                "objective": "Identify the relevant clauses and conditions that apply to the inquiry.",
                "importance": "critical",
                "type": "review",
                "steps": [
                    {
                        "rank": 1,
                        "name": "Extract Relevant Clauses",
                        "description": "Extract the specific clauses from the policy wording that are relevant to extended warranty claims.",
                        "reasoning": "Extracting the relevant clauses will help in directly addressing the conditions that apply to the claim.",
                        "objective": "Have a clear list of clauses (limits, conditions, exclusions,elgibility) that pertain to extended warranty.",
                        "importance": "critical",
                        "type": "Content Query",
                        "requirements": [
                            {
                                "type":"Content",
                                "referenceUri":"/contents/2df077cc-dded-4f3e-8887-de4b117e4439"
                            }
                        ]
                    }
                ]
            }
        ]
    }
}

The executor agent will then go ahead and generate relevant questions to achieve the objective and then return the answers to those questions:

[
    {
        "question": "What are the specific limits mentioned in the policy wording for extended warranty claims?",
        "answer": "The specific limits mentioned in the policy wording for extended warranty claims are a lifetime liability limit of $60,000 for combined Purchase Assurance and Extended Warranty benefits and a limit of 1 claim in any 12 consecutive month period for mobile device insurance.\n",
        "sources": [...]
    },
    {
        "question": "What conditions must be met to be eligible for an extended warranty claim according to the policy wording?",
        "answer": "To be eligible for an extended warranty claim, the insured items must require repair or replacement during the coverage period, be lost, stolen, or damaged within 90 days of purchase, and the extended warranty adds up to 1 year to the original manufacturer's warranty. Exclusions include aircraft, motorized vehicles, business or commercial items, and repairs/replacements not covered by the original manufacturer's warranty.\n",
        "sources": [...]
    },
    {
        "question": "What exclusions are specified in the policy wording for extended warranty claims?",
        "answer": "The exclusions specified in the policy wording for extended warranty claims include coverage for aircraft, motorized vehicles, business or commercial items, repairs/replacements not covered by the original manufacturer's warranty, accessories, batteries, mobile devices purchased for resale, professional, or commercial use, mail order items not delivered or damaged upon delivery, traveller's cheques, tickets, trucks, vans, vehicles over $65,000, wear and tear, mechanical breakdown, and a limit of 1 claim in any 12 consecutive month period for mobile device insurance.\n",
        "sources": [...]
    },
    {
        "question": "What are the eligibility criteria for making an extended warranty claim as per the policy wording?",
        "answer": "The eligibility criteria for making an extended warranty claim include adding up to 1 year to the original manufacturer's warranty, requiring repair or replacement of insured items during the coverage period, losing, stealing, or damaging insured items within 90 days of purchase, and experiencing specific scenarios with a mobile device during the coverage period. Additionally, the policy limits mobile device insurance to 1 claim in any 12 consecutive month period.\n",
        "sources": [...]
    }
]

The next logical step in that process would be to build on top of those answers to reach the conclusion defined by the objective.

Conclusion

The development of the LLM-based Job Planner Agent signifies a step forward in steering an LLM agent planning and execution activities. By generating objective-driven plans with structured tasks and steps, validating objectives through advanced LLM capabilities, and ensuring seamless execution, these capabilities aim to enhance efficiency, reduce errors, and drive successful outcomes across various domains.

Stay tuned for further updates as I continue to refine and expand this capability. There’s still a lot of areas to explore and improve. This is very much early research for me at the moment and I’m learning a lot about all the different quirks and challenges to consider while tackling agentic behaviors.