I seem to be stuck in a loop with the LLM setting up the plan. I have tried a number of ways to break out of this or let it know I want to keep the plan simpler and not recurse/loop through anything. I ended up recreating the Thunk with simpler instructions, which worked well. More curious about what it is trying to get from me here, or if there is a way to actually fix this when it occurs. Since the Thunk is still in setup mode ,I can’t manually edit the property bindings, yet that seems to be part of the error:
<toolResult>
<output>
Error: Playbook node "Run: for each CRM data" has no input binding.
</output>
</toolResult>
We started out wanting to support all kinds of possible workflows and automation. But through a couple of years of learning, we’ve realized that most of the common workflows have this pattern ..
for each X,
do A
do B
…
So we’ve focused the system a lot more on taking this kind of workflow and doing it well (reliably, scalably, etc) with AI. The planning stage also reflects that. It really expects that the workflow you want to build fits into that pattern.
Starting with today’s deployment actually, there’'s a change in the “build a new thunk” initial dialogs to steer you more down this path.
So that’s the general answer.
Overall, we need to add more flexible control in the planning phase, so that’s an area for us to improve. Anytime you get stuck in planning, one escape hatch is “Restart” — that starts again from the initial goal. That’s useful only in case the errors were transient in some way.
If you can share the original instructions that got to this bad state, we’d love to see. It will help us debug the problem.
Plan that didnt work:
- Determine if the question is valid for CRM
- Determine the keywords from the question
- Send keywords to Make.com for Netsuite records
- Determine what records are relevant for this question
- Send record IDs to Make.com for record details
- Create response to the question based on record and details
Plan that did work:
- Determine if valid CRM question
- Determine keywords from question
- Send keywords and receive CRM records
- Determine relevant CRM records
- Send record IDs and receive record details
- Answer question based on records and details
Also, there’s a minor bug where the start of the process is getting wrapped in a fenced code block.
Thanks for sending those two plan descriptions. Will investigate.
That messy rendering problem is fixed. By way of explanation, we support markdown format for the content being rendered and whitespaces at the start of lines results in that wierd assumption that it is a code block.
In any case, it is fixed now and thanks for reporting it. It doesn’t fix thunks created already (the bug only existed for about 24 hours) but it will render properly for any new thunk.
Hi @Logan_T , we’ve figured out the problem you ran into planning that thunk. The fix will be deployed sometime on Monday.
Internally, there are a few stages of planning that happen before you can intervene – and there was an error happening inside there that should not happen unless there is an LLM error + a code bug on our side. There was a bug here that got surfaced when the LLM makes a certain kind of error. Anyway, it should not surface again.
I realized one thing looking at your example. We need to is to provide more guidance on how to provide step instructions to the AI agent. Your example looks like this (because you think like a programmer)..
- Determine if the question is valid for CRM
- Determine the keywords from the question
- Send keywords to Make.com for Netsuite records
- Determine what records are relevant for this question
- Send record IDs to Make.com for record details
- Create response to the question based on record and details
Instead, you want to think more in terms of logical steps, where perhaps each of them might have been givien to a different person if you had actual people processing the workflow. In this case, I might say:
- Analyse the question to determine validity and keywords
- Fetch matching records from Make.com
- Create response to the question based on the records
Inside any one of those workflow steps, you can define further detailed instructions. For example in step#2, you might say something like this .. “First send keywords to Make.com to fetch relevant record ids, then fetch the record details”. The platform’s workflow orchestration ensure the top-level 3 steps get executed in order. Within any one step, the AI agent will plan and execute the instructions more dynamically. In the case of step#2, if the agent is given a tool to access the Make.com API for fetching records based on keywords, it will invoke it appropriately.
I liked having each step separated so I can diagnose when there was an error. My general rule has been that if I am updating fields, they should be very similar to be grouped in a single step; otherwise, they should be separated so it’s easier to change instructions and restart the step.
Since I got this working with slightly different instructions, is there anything inherently wrong with how I planned this, or more of a best practice?
Nothing was wrong with how you wrote it and no best practice issue. Minor differences in how you write things cause variances in the LLM response. Once in a while (actually, maybe upto 5% of the time), there are errors in the plan that the LLM responds with. It is the job of our platform to detect those and either fix them up automatically or ask the LLM to fix them — and do it silently so that it doesn’t surface up to the user. We had a bug in one part of the code that fixes things up.
Re separating into separate steps for diagnosis, adding @m33na for her comments. She’s been building a lot of thunks with customers. She may have other suggestions also for this diagnosis need. I know I tend to break the flow into logical steps corresponding to one conceptual activity and then experiment within each step. Our system as currently designed doesnt make it very comvenient to change the top-level workflow whereas changing the instructions and tools within each step is very easy
