Comment by sprobertson

Comment by sprobertson 3 days ago

1 reply

View on Hacker News

Great demo, straight to the point. It might be nice to have some kind of feedback mechanism if it can't find the element on the page, or if it's partially cut off. For example, I changed the GitHub profile to my own (https://github.com/spro) in the example and it doesn't scroll down far enough for the whole image. I imagine in general it would be nice to scroll to an element ID (or even element description using the vision models) instead of a hardcoded value.

Side-note: The comment for the frequency graph is wrong, it mentions stars instead.

marcon680 3 days ago

RE: feedback mechanism -- yep, a feedback mechanism is definitely something we're thinking of adding. Since we use VLMs that are trained to always output coordinates (i.e., they don't have a way to say "not on the page"), we're probably going to try fine tuning with some negative examples to see if we can build that feedback mechanism in.

One way to hack the scrolling to an element is to first run extract_bbox on a natural language description (in your case for GitHub it might be "follow button") then take the Y coordinate of that element and scroll that number of pixels. I just wrote this bit of code that I tested and it brings the contribution graph into full view:

    simplex.goto("github.com/spro")
    coords = simplex.extract_bbox("follow button")
    simplex.scroll(coords[1])
    simplex.wait(2000)
    image = simplex.extract_image("green tile graph")
But then it incorrectly picks the code review/submissions/etc. graph as the green tile graph -- we'll look into it!

re: frequency graph typo -- just pushed a fix, thanks!

Reply View 0 replies