Foundry
RL Gym for Browser Agents
Founding Engineer: Large-Scale Web Scraping & Crawling
About the role
About Us
We’re building the first end-to-end testing platform for web agents, including a Browser Gym for RL-driven optimization. Our platform helps teams evaluate, benchmark, and improve web agents before they go live, ensuring they can handle real-world, dynamic environments.
With synthetic user simulations, automated evaluations, and large-scale benchmarking, we’re setting a new standard for web agent testing.
We’re a YC-backed team, and this is a founding engineering role—you’ll be one of the first hires defining how we crawl, structure, and analyze the open web at scale.
The Role
We need a Founding Web Scraping Engineer to build internet-scale web crawling infrastructure—not just scraping a single site, but handling millions of domains and evolving anti-bot defenses.
You’ll be responsible for designing robust, distributed crawling systems that adapt dynamically to web changes, optimize for efficiency, and ensure reliable data extraction.
What You’ll Do
- Build large-scale, distributed crawlers that intelligently prioritize, schedule, and optimize requests across millions of domains.
- Develop adaptive web scraping systems that handle DOM changes, WebSockets, AJAX-heavy sites, and dynamically loaded content.
- Optimize scraping performance and resilience, ensuring high-throughput data extraction with proxy/network optimizations and behavior-driven stealth tactics.
- Solve captchas at scale, integrating third-party solvers, heuristic-based workarounds, and behavior-driven bypass techniques.
- Manage proxy and identity rotation, implementing session-aware scraping, JA3/TLS fingerprint spoofing, and request signature control.
- Structure and clean extracted data for downstream analytics, AI training, and benchmarking applications.
What We’re Looking For
- Expert-level experience in large-scale web scraping & crawling (Selenium, Puppeteer, Playwright, Scrapy, undetected-chromedriver).
- Deep knowledge of anti-bot detection strategies (TLS fingerprinting, JA3 signatures, request header anomalies, and bot behavior tracking).
- Hands-on expertise with captcha-solving strategies, including leveraging APIs, OCR-based approaches, and behavior-driven evasion.
- Proven experience building efficient proxy management systems, including rotating IPs across residential, datacenter, and mobile networks.
- Proficiency in Python, Go, or JavaScript, with experience in high-performance, parallelized scraping frameworks.
- Understanding of HTTP/2, HTTP/3, WebSockets, GraphQL, and browser-based fingerprinting.
- Experience designing scalable, fault-tolerant scraping infrastructure that adapts to changes in real time.
Bonus Points
- Experience with search engine-scale crawling.
- Background in LLM-driven web extraction or RL-enhanced adaptive crawling.
- Contributions to open-source scraping tools or web automation projects.
Why Join?
- Founding role—you’ll define and own our web crawling infrastructure from day one.
- Work at internet scale—building a system that dynamically adapts and scales across millions of domains.
- YC-backed—we’re building something that doesn’t exist yet, and you’ll be part of the core team making it happen.
About Foundry
At Foundry, we're building the first end-to-end testing platform for web agents, including a Browser Gym for RL-driven optimization. Our platform helps teams run pre-production evaluations, benchmarking, and reinforcement learning to make web agents more reliable. With synthetic user simulations, automated evaluations, and large-scale benchmarking, we make sure web agents can handle dynamic environments with confidence.
