Skyvern

We’ve been working hard, cooking up something new to share with you all!

Skyvern 2.0 scored state-of-the-art 85.85% on the WebVoyager Eval.

This is the best-in-class performance of all WebAgents, giving advanced closed-source web agents like Google Mariner a run for their money.

TL;DR

• Real-World Tests: We ran all of the tests in Skyvern Cloud to get a better representation of autonomous browser operations (ie, they didn’t run on any local machines)
• Open-Sourced Results: All of the runs can be seen here through our UI.
• We’re just getting started. Try Skyvern Cloud or Skyvern Open Source out for yourself and see Skyvern in action!

Agent Architecture

Achieving this SOTA result required expanding Skyvern’s original architecture. Skyvern 1.0 involved a single prompt operating in a loop both making decisions and taking actions on a website. This approach was a good starting point but scored ~45% on the WebVoyager benchmark because it had insufficient memory of previous actions and could not do complex reasoning.

To solve this problem, we created a self-reflection feedback loop within Skyvern. This resulted in 2 main changes:

1. We added a “Planner” phase, which could decompose very complex objectives down into smaller achievable goals
2. This allowed Skyvern to have a working memory of things it had completed and things that were still waiting to be finished
   • This allows Skyvern to work with long, complex prompts without increasing the hallucination rate
3. We added a “Validator” phase, which confirmed whether or not the original goals the “Planner” generates are successfully completed
4. This acts as a supervisor function to confirm that the Task executor is achieving its objectives as expected and report any errors/tweaks back to the Planner so it can make adjustments in real-time as needed

Test Setup

All tests were run in Skyvern Cloud with an async cloud browser and used a combination of GPT-4o and GPT-4o-mini as the primary decision-making LLMs. The goal of this test is to assert real-world quality — the quality represented by this benchmark is the same as what you would experience with Skyvern’s browsers running asynchronously.

💡 Why is this important? Most benchmarks are run on local browsers with a relatively safe IP address and an impressive browser fingerprint. This is not representative of how Autonomous agents will run in the cloud, and we wanted our benchmark to represent how agents would behave in production

In addition to the above, we’ve made a few minor tweaks to the dataset to bring it up to date:

1. We’ve removed 8 tasks from the dataset because the results are no longer valid. For example, one of the tasks asked to go to apple.com and check when the Apple Vision Pro will be released — in 2025, it’s already been released and forgotten
2. Many of the flight/hotel booking tasks referenced old dates. We updated both the prompt and the answer to more modern dates for this evaluation

🔍 For the curious:

The full dataset can be seen here: https://github.com/Skyvern-AI/skyvern/tree/main/evaluation/datasets

The full list of modifications can be seen here: https://github.com/Skyvern-AI/skyvern/pull/1576/commits/60dc48f4cf3b113ff1850e5267a197c84254edf1

Test Results

We’re doing something out of the ordinary. In addition to the results, we’re making our entire benchmark run public.

💡 Why is this important? Most benchmarks are run behind closed doors, with impressive results being published without any accompanying material to verify the results. This makes it hard to understand how things like hallucinations or website drift over time play an effect on agent performance

We believe this isn’t aligned with our open source mission, and have decided to publish the entire eval results to the public.

📊 All individual run results can be seen here: https://eval.skyvern.com

🔍 The entire Eval dataset can be seen here: https://github.com/Skyvern-AI/skyvern/tree/main/evaluation/datasets

Limitations of the WebVoyager benchmark

The WebVoyager benchmark is a comprehensive benchmark testing a variety of prompts on 15 different websites. While this acts as a good first step in testing Web agents, this only captures 15 hand-picked websites of the millions of active websites on the internet.

We think there is tremendous opportunity here to better evaluate web agents against one another with a more comprehensive benchmark similar to SWE-Bench.

What’s on the horizon

Browser automation is still a nascent space with tons of room for improvement. While we’ve achieved a major milestone in agent performance, a few important issues are next to be solved:

1. Can we improve Skyvern’s reasoning to operate efficiently in situations with more uncertainty? Examples include vague prompts, ambiguous or highly complex websites/tools, websites with extremely poor UX (legacy portals)
2. Can we give Skyvern access to more tools so it can effectively log into websites, make purchases, and behave more like a human?
3. Can we have Skyvern better memorize things it has already done in the past so it can do them again at a lower price point?

References

• Google Mariner Report
• Claude Computer Use Report
• WebVoyager Report
• WILBUR Report
• AgentE Report
• HCompany Report