Build a web scraper with Spring Boot and Jsoup

Australian software teams frequently need to harvest structured information from third-party pages, whether for price comparison tools in Melbourne or travel aggregators in Sydney. A Spring Boot service paired with the Jsoup HTML parser gives Java developers a lightweight stack for this work without dragging in a heavy browser. The combination keeps code idiomatic to the Spring ecosystem while delegating tag traversal and CSS selection to a mature library.

Spring Boot supplies the application context, dependency wiring, and HTTP layer, so the scraper itself stays focused on extraction. Jsoup handles malformed markup gracefully, exposes a jQuery-like selector API, and can sanitise content before persistence. With sensible defaults around throttling, retries, and user-agent strings, the resulting service behaves politely and recovers from network hiccups.

The same building blocks scale from a developer experimenting on a laptop in Adelaide to a production deployment on AWS Sydney. The patterns below cover project setup, extraction, transport, scheduling, and the legal landscape relevant to Australian operators.

Project setup and Maven dependencies

Generate a fresh Spring Boot project through Spring Initializr, selecting the Web and Validation starters alongside a recent stable Java release. Add Jsoup as a single dependency, since it bundles the parser, selector engine, and HTML cleaning helpers. Configure an application.yml file with server port, logging levels, and any proxy settings required by your network.

Externalise configuration through @ConfigurationProperties so the user-agent, timeouts, and target URLs live in environment-specific files. Production profiles typically point to the ap-southeast-2 region for low latency across the eastern seaboard. A clean package layout, with scraper, dto, and domain packages, pays off once scheduled tasks and REST controllers begin sharing the same service bean.

Extracting data with Jsoup selectors

A service annotated with @Service accepts a URL and returns a domain object. Inside, call Jsoup.connect(url).userAgent(...).timeout(...).get() to fetch the document, then chain CSS selectors such as doc.select("div.product-card") to navigate relevant nodes. Jsoup tolerates mixed casing, unclosed tags, and inline scripts, which is why it outperforms regex-based approaches.

Map each card to a plain Java record or POJO, keeping parsing tolerant of missing fields. Australian retailers in Brisbane and Perth often serve slightly different HTML to mobile user-agents, so send a desktop-style identifier unless a mobile layout is targeted. Normalise currency strings and convert AUD prices to BigDecimal so downstream calculations stay accurate. Wrap the parsing call in a try-catch block that translates IOException into a custom ScrapingException.

Exposing results through a REST endpoint

A thin @RestController bridges the scraping service and external consumers. Accept query parameters such as the source identifier and search term, then return a list of DTOs serialised to JSON. Validation annotations reject empty strings before the scraper runs, sparing the upstream site pointless requests and keeping the code path easy to reason about.

When the scraper powers an internal dashboard for a Sydney analytics team, response caching with Spring's @Cacheable annotation reduces upstream load. Cache keys should include URL plus a normalised query, and TTLs should respect the target site's update frequency. Adding an actuator health indicator that pings the upstream domain tells operators in Canberra that the pipeline is failing before users see stale data.

Scheduling and respecting crawling etiquette

Long-running scraping workloads belong in background jobs, and Spring's @Scheduled annotation handles that without external infrastructure. Wire a scheduler that runs during off-peak hours, which in Australia means early morning AEST before Sydney commuters start their day. Configure cron expressions relative to Australia/Sydney and document the rationale in comments so on-call engineers in other time zones are not caught off guard.

Robots.txt compliance deserves more than lip service. Parse the file with Jsoup at startup, cache its directives, and refuse disallowed paths. Identify the crawler with a descriptive user-agent containing a contact URL, mirroring conventions used by Googlebot and well-behaved bots in the Australian SEO community. Add jitter between requests and cap concurrency with a Semaphore so a small team does not hammer a single origin server.

Legal and ethical considerations for Australian deployments

The Privacy Act 1988 and the Australian Privacy Principles govern how personal information is collected, used, and stored. Scraped data often contains names, emails, or public reviews, so build redaction or pseudonymisation into the persistence layer before anything is written to disk. For organisations above the current turnover threshold, the Notifiable Data Breaches scheme applies, meaning sloppy practices can escalate into regulatory incidents.

The Spam Act 2003 restricts unsolicited commercial email, limiting what can be done with harvested contact details even when technically permissible. The Copyright Act 1968 protects original expression, so storing full article text without permission carries risk; linking and short excerpts are safer defaults. Australian courts have examined scraping in cases involving real estate listings and SEO, so maintain a written policy covering purpose, retention, and deletion. Teams that keep audit logs find it easier to respond to the Office of the Australian Information Commissioner when questions arise.

Practical recommendations for a robust scraper