Hacker Remix

Show HN: Simplex: Automate browser workflows using code and natural language

141 points by marcon680 3 days ago | 37 comments

Simplex is an SDK that provides low-level functions you can use to create reliable web automations using code and natural language.

Here's a quick video to show you how it works: https://www.loom.com/share/8d27d0f9e0f34c77a81b0ac770151c12

A couple weeks ago, we needed a way to automatically find 3D assets from the internet for one of our work contracts. We didn’t need an AI to choose all the steps for us autonomously — we knew generally what the flow to find items should be and just needed some intelligence to handle different webpage formats.

Playwright couldn’t easily generalize to the different sites we wanted to search over and Claude computer use was tough to use — it was slow, expensive, struggled to click on the correct web elements, and there was no way to correct its actions if it failed at step 20 of our workflow.

So we built a vision-based solution using vision language models that sits on top of Playwright. In our SDK, building a function that can universally search websites is as easy as this:

    from simplex import Simplex
    simplex = Simplex(api_key="api_key")
    
    def search(website, search_query):
        simplex.goto(website)
        simplex.click("the search bar")
        simplex.type(search_query)
        simplex.press_enter()
        
    search("mit.edu", "alyssa p hacker")

You can play around with what we’ve built and see a few more examples at https://simplex.sh/playground.

We'd love feedback!

jackienotchan 3 days ago

Would you mind sharing the story behind your pivot from on-demand photorealistic vision datasets[0] to browser automation?

[0] https://www.ycombinator.com/launches/Lbx-simplex-on-demand-p...

marcon680 3 days ago

Yeah sure! We mentioned a little bit in the post that we stumbled upon this problem when working on a synthetic data contract. Internally, we were building software at the time to fully automate the process of creating synthetic data, all the way from purchasing assets to building scene layouts to rendering images. We realized after 2 months of building that there wasn't a great need for this problem nor could we build something minimal but feature-complete that we could give to people and iterate off of. We also saw the writing on the wall re: building 3D world models (see WorldLabs and NVIDIA's new 3D world models). We're excited to see where this goes!

TypingOutBugs 3 days ago

It's quite a crowded market, browser workflow automation. What are you guys trying to do differently? For me, as someone who does a bunch of browser automation tasks, the real issue is stability. So many tools fall over on non-trivial workflows like complex forms with tabs. Fix that and you may open up a large testing automation market.

marcon680 2 days ago

Some of the things we're trying to differently are 1) making flows deterministic by using vision models that have a consistent output given an image and an input query and 2) breaking up flows into these smaller atomic actions in order to improve consistency as well.

RE: workflows w/ complex forms and tabs -- do you have some sites that are good examples of this? We'd love to see how Simplex does.

Mandatum 3 days ago

VC funding.

globnomulous 3 days ago

Apparently a simplex is a concept in geometry. I'm guessing this is the intended meaning of the term in the name. When most decently well informed, native speakers of English hear "simplex," though, they are likely to hear "herpes simplex" and so are likely to wonder, as I did, whether, or why, the program is named after a sexually transmitted disease.

gashmol 3 days ago

I thought it's related to the simplex method in linear programming

TeMPOraL 3 days ago

Which IIRC is named for that geometry thingie GP mentioned, that this method uses to represent the problem, with the solutions being vertices/edges of the thingie.

globnomulous 3 days ago

Interesting, TIL herpes is top of mind for me, but not for most of HN.

TypingOutBugs 3 days ago

If you gave me the chance to come up with 500 things related to simplex, herpes would not be there.

IanCal 3 days ago

I would never have made that link.

internet_points 3 days ago

I for one did not make that connection. Only thing I thought of was https://simplex.chat/ (which is fortunately not marketed as a dating app)

sprobertson 3 days ago

Great demo, straight to the point. It might be nice to have some kind of feedback mechanism if it can't find the element on the page, or if it's partially cut off. For example, I changed the GitHub profile to my own (https://github.com/spro) in the example and it doesn't scroll down far enough for the whole image. I imagine in general it would be nice to scroll to an element ID (or even element description using the vision models) instead of a hardcoded value.

Side-note: The comment for the frequency graph is wrong, it mentions stars instead.

marcon680 3 days ago

RE: feedback mechanism -- yep, a feedback mechanism is definitely something we're thinking of adding. Since we use VLMs that are trained to always output coordinates (i.e., they don't have a way to say "not on the page"), we're probably going to try fine tuning with some negative examples to see if we can build that feedback mechanism in.

One way to hack the scrolling to an element is to first run extract_bbox on a natural language description (in your case for GitHub it might be "follow button") then take the Y coordinate of that element and scroll that number of pixels. I just wrote this bit of code that I tested and it brings the contribution graph into full view:

    simplex.goto("github.com/spro")
    coords = simplex.extract_bbox("follow button")
    simplex.scroll(coords[1])
    simplex.wait(2000)
    image = simplex.extract_image("green tile graph")

But then it incorrectly picks the code review/submissions/etc. graph as the green tile graph -- we'll look into it!

re: frequency graph typo -- just pushed a fix, thanks!

thasaleni 3 days ago

tried:

  simplex.goto("www.amazon.com")
  simplex.wait(2000)
  simplex.click('the search bar')
  simplex.type("bicycle")
  simplex.press_enter()
  simplex.wait(1000)
  image = simplex.extract_image("first 3 rows of results")

get error:

  simplex error processing error

marcon680 3 days ago

So sorry about this, it looks like our servers went down overnight. Just fixed, should be good now!