Comment by joshstrange

I love the use of reusable non-LLM code for the "happy path" and only using the LLM to create a workflow or repair it.

One area for improve seems to be the checkboxes/radio buttons (possibility other input types?) given the demo didn't pick up on that (it did make the same selections but it didn't recognize this was a multiple-choice input). It might be useful to have a step before creating the JSON where it asks the user some follow up questions like, "Here are the inputs I found and my understanding of their data type". And then go through each input asking for a default value and maybe even clarification on "Should we even prompt for this?" (Example, always select country X).

I wonder if, for workflow repair purposes, it would be helpful to, at recording time, save more contextual information about the fields you are filling/clicking on. "This a country selector", "This is the birthdate field", etc. So that if the xpath/css/etc fails you can give the LLM doing the repair work a description of what it's looking for.

I'm excited to see more efforts in QA testing with things like this. Brittle e2e tests are the bane of my (limited) automated experience and the ability to auto-heal and/or deal with minor deviations would be wonderful.