How to Build a Log Aggregator with Spring Boot and ELK Stack Integration
Centralised logging turns scattered console output into searchable operational evidence. A Spring Boot service can emit structured events, Logstash can receive and enrich them, Elasticsearch can index them, and Kibana can turn them into dashboards and alerts. Together, these tools create an ELK log aggregation layer that supports troubleshooting across microservices without granting engineers SSH access to every host.
For Australian engineering teams, the design must work across cities and clocks. Developers in Sydney might support a platform used in Melbourne, operations may run from Brisbane, and field systems around Perth add another network boundary. Australian workplaces often favour direct, low-drama communication, but an informal “no worries” is too vague for incident status. Shared dashboards and runbooks need precise wording so a Melbourne engineer can follow a Sydney outage at 3 a.m.
Emit structured logs from Spring Boot
Spring Boot’s default console output is useful locally, but it is inefficient for aggregation. Configure Logback with net.logstash.logback.encoder.LogstashEncoder so every event is written as a JSON object to a mounted log directory. Include stable fields such as service, environment, level, message, timestamp and trace_id; an Elastic Agent or Filebeat can then tail the file and buffer events when the network is unavailable.
Use SLF4J’s MDC to attach a request ID, trace ID and safe tenant reference to each request. Redact credentials, payment-card data, access tokens and raw request bodies before logging. Keep application logs separate from access logs, then use a consistent timestamp in UTC so events from different Australian regions remain easy to order.
Ingest and enrich events with Logstash
Have the shipping agent send records to Logstash’s Beats input on port 5044. In the pipeline, parse JSON only when necessary, normalise field names to the Elastic Common Schema, and add metadata that the application cannot know, such as host, Kubernetes pod or deployment region. Avoid repeated Grok patterns for known JSON fields because a failed match can silently discard useful context.
Make ingestion resilient with an agent disk buffer, Logstash persistent queue and a dead-letter queue for malformed events. Monitor the event timestamp against the ingestion timestamp to expose lag. For high-volume services, batch Elasticsearch writes, but retain the original correlation identifiers so one request can be followed across API, payment and authentication components.
Index, search and visualise events
Create an index template that maps timestamp as a date and keyword fields for service, environment, level and trace_id. Full-text fields can use text with a keyword subfield. Apply index lifecycle management to roll old data, delete it according to the organisation’s retention policy, and avoid mapping every dynamic string as full text, which can inflate storage costs.
A practical separation of responsibilities keeps each component focused:
| Component | Main responsibility | Data handled | Operational signal |
|---|---|---|---|
| Logstash | Receive, parse and enrich events | JSON records and metadata | Queue depth and processing errors |
| Elasticsearch | Store and search indexed events | Mapped documents and aggregations | Cluster health, rejected writes and latency |
| Kibana | Explore data and present dashboards | Saved queries, visualisations and alerts | Stale dashboards and alert delivery failures |
Build Kibana dashboards around service health, error rates, slow requests and ingestion lag. A trace-based view lets an engineer filter one request across services, while data panels reveal which host produced the highest error volume. Restrict dashboard links to relevant teams and encode service ownership in filters so alerts reach the right on-call rotation.
Secure access and meet Australian obligations
Terminate TLS between every component, encrypt data at rest, and use role-based access for Logstash, Elasticsearch and Kibana. Rotate credentials through a secrets manager, disable anonymous privileges, and review audit events. In Elastic Cloud, choose an Australian region such as Sydney where available, but confirm contractual data-residency and incident-response requirements before moving regulated logs there.
Australian privacy obligations shape what should be collected. The Privacy Act 1988 and the Notifiable Data Breaches scheme can apply to personal information, while APRA obligations affect many financial services; neither makes detailed logging of customer records acceptable. The Australian Signals Directorate’s Essential Eight also places a premium on controlled administrative access, logging and incident response. Redact sensitive values at source and keep only the fields needed for investigation.
Document retention periods, deletion jobs and privileged-access reviews as operational controls rather than legal footnotes. For a Melbourne ecommerce workload, a Sydney fintech platform or a Perth resources system, the same baseline can be adapted to sector rules. Use unambiguous statuses—“investigating”, “mitigated” and “resolved”—so Australian teams’ preference for concise communication never obscures an active incident.
Verify deployment and operational behaviour
Run Elastic Stack components in Docker Compose for development, then pin compatible versions for production. On Kubernetes, use Elastic Agent or the Elastic Operator, define memory requests and heap limits, and keep Elasticsearch storage separate from application pods. For Elastic Cloud, verify network paths from each Australian office and choose an endpoint with predictable latency to the hosting cluster.
Test the path from a controlled log event to a Kibana panel, including redaction, timestamp ordering, retries and agent reconnection. Inject malformed JSON deliberately, confirm that it reaches the dead-letter queue, and verify that alerts contain the service name, environment and correlation ID. JavaWhizz provides IT consultancy services for teams that want an external review of this architecture and its support model. The finished aggregator should provide traceable events, measurable ingestion lag and controlled access, giving Australian on-call engineers one dependable source during regional incidents.