14 KiB
14 KiB
Distributed Tracing with Spring Boot Actuator
Spring Boot Actuator provides dependency management and auto-configuration for Micrometer Tracing, a facade for popular tracer libraries.
Tip
To learn more about Micrometer Tracing capabilities, see its reference documentation.
Supported Tracers
Spring Boot ships auto-configuration for the following tracers:
- OpenTelemetry with Zipkin, Wavefront, or OTLP
- OpenZipkin Brave with Zipkin or Wavefront
Getting Started with OpenTelemetry and Zipkin
Dependencies
Add the following dependencies to your project:
Maven:
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>io.micrometer</groupId>
<artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
<groupId>io.opentelemetry</groupId>
<artifactId>opentelemetry-exporter-zipkin</artifactId>
</dependency>
</dependencies>
Gradle:
dependencies {
implementation 'org.springframework.boot:spring-boot-starter-actuator'
implementation 'io.micrometer:micrometer-tracing-bridge-otel'
implementation 'io.opentelemetry:opentelemetry-exporter-zipkin'
}
Configuration
Add the following application properties:
management:
tracing:
sampling:
probability: 1.0 # Sample 100% of requests in development
zipkin:
tracing:
endpoint: "http://localhost:9411/api/v2/spans"
logging:
pattern:
level: "%5p [%X{traceId:-},%X{spanId:-}]"
Basic Application Example
@SpringBootApplication
@RestController
public class MyApplication {
private static final Logger logger = LoggerFactory.getLogger(MyApplication.class);
@GetMapping("/")
public String home() {
logger.info("Handling home request");
return "Hello World!";
}
public static void main(String[] args) {
SpringApplication.run(MyApplication.class, args);
}
}
Configuration Options
Sampling Configuration
Control which traces are collected:
management:
tracing:
sampling:
probability: 0.1 # Sample 10% of requests in production
rate: 100 # Maximum 100 traces per second
Zipkin Configuration
management:
zipkin:
tracing:
endpoint: "http://zipkin:9411/api/v2/spans"
timeout: 1s
connect-timeout: 1s
read-timeout: 10s
OpenTelemetry OTLP Configuration
management:
otlp:
tracing:
endpoint: "http://otlp-collector:4318/v1/traces"
timeout: 1s
compression: gzip
headers:
Authorization: "Bearer your-token"
Wavefront Configuration
management:
wavefront:
tracing:
application-name: "my-application"
service-name: "my-service"
api-token: "${WAVEFRONT_API_TOKEN}"
uri: "https://your-instance.wavefront.com"
Custom Spans
Using @Observed Annotation
@Service
public class UserService {
@Observed(name = "user.service.find-by-id")
public User findById(Long id) {
// Service logic
return userRepository.findById(id);
}
@Observed(
name = "user.service.create",
contextualName = "creating-user",
lowCardinalityKeyValues = {"operation", "create"}
)
public User createUser(CreateUserRequest request) {
// Creation logic
return save(request.toUser());
}
}
Programmatic Span Creation
@Service
public class OrderService {
private final ObservationRegistry observationRegistry;
public OrderService(ObservationRegistry observationRegistry) {
this.observationRegistry = observationRegistry;
}
public Order processOrder(OrderRequest request) {
return Observation.createNotStarted("order.processing", observationRegistry)
.lowCardinalityKeyValue("order.type", request.getType())
.observe(() -> {
// Add custom tags
Observation.Scope scope = Observation.start("order.validation", observationRegistry);
try {
validateOrder(request);
} finally {
scope.close();
}
// Process order
return saveOrder(request);
});
}
private void validateOrder(OrderRequest request) {
// Validation logic
}
private Order saveOrder(OrderRequest request) {
// Save logic
return new Order();
}
}
Using Micrometer's Tracer API
@Service
public class PaymentService {
private final Tracer tracer;
public PaymentService(Tracer tracer) {
this.tracer = tracer;
}
public PaymentResult processPayment(PaymentRequest request) {
Span span = tracer.nextSpan()
.name("payment.processing")
.tag("payment.method", request.getMethod())
.tag("payment.amount", String.valueOf(request.getAmount()))
.start();
try (Tracer.SpanInScope ws = tracer.withSpanInScope(span)) {
// Add events
span.event("payment.validation.started");
validatePayment(request);
span.event("payment.validation.completed");
span.event("payment.processing.started");
PaymentResult result = processPaymentInternal(request);
span.event("payment.processing.completed");
// Add result information
span.tag("payment.status", result.getStatus());
return result;
} catch (Exception ex) {
span.tag("error", ex.getMessage());
throw ex;
} finally {
span.end();
}
}
private void validatePayment(PaymentRequest request) {
// Validation logic
}
private PaymentResult processPaymentInternal(PaymentRequest request) {
// Processing logic
return new PaymentResult();
}
}
Baggage
Baggage allows you to pass context information across service boundaries:
@Service
public class UserService {
private final BaggageManager baggageManager;
public UserService(BaggageManager baggageManager) {
this.baggageManager = baggageManager;
}
public User getCurrentUser(String userId) {
// Set baggage that will be propagated to downstream services
try (BaggageInScope baggageInScope =
baggageManager.createBaggage("user.id", userId).makeCurrent()) {
return fetchUserFromDatabase(userId);
}
}
private User fetchUserFromDatabase(String userId) {
// This method and any downstream calls will have access to the baggage
String currentUserId = baggageManager.getBaggage("user.id").get();
// Use the user ID for security context, logging, etc.
return userRepository.findById(userId);
}
}
HTTP Client Tracing
WebClient Tracing
Spring Boot automatically configures tracing for WebClient:
@Service
public class ExternalApiService {
private final WebClient webClient;
public ExternalApiService(WebClient.Builder webClientBuilder) {
this.webClient = webClientBuilder
.baseUrl("https://api.example.com")
.build();
}
public ApiResponse callExternalApi(String data) {
return webClient
.post()
.uri("/process")
.bodyValue(data)
.retrieve()
.bodyToMono(ApiResponse.class)
.block();
}
}
RestTemplate Tracing
For RestTemplate, add the interceptor manually:
@Configuration
public class RestTemplateConfig {
@Bean
public RestTemplate restTemplate(RestTemplateBuilder builder) {
return builder
.interceptors(new TraceRestTemplateInterceptor())
.build();
}
}
Database Tracing
JPA/Hibernate Tracing
Enable SQL tracing with additional configuration:
spring:
jpa:
properties:
hibernate:
generate_statistics: true
session:
events:
log:
LOG_QUERIES_SLOWER_THAN_MS: 25
management:
tracing:
enabled: true
metrics:
distribution:
percentiles-histogram:
http.server.requests: true
Custom Database Observation
@Repository
public class UserRepository {
private final JdbcTemplate jdbcTemplate;
private final ObservationRegistry observationRegistry;
public UserRepository(JdbcTemplate jdbcTemplate,
ObservationRegistry observationRegistry) {
this.jdbcTemplate = jdbcTemplate;
this.observationRegistry = observationRegistry;
}
public User findById(Long id) {
return Observation.createNotStarted("db.user.find-by-id", observationRegistry)
.lowCardinalityKeyValue("db.operation", "select")
.lowCardinalityKeyValue("db.table", "users")
.observe(() -> {
String sql = "SELECT * FROM users WHERE id = ?";
return jdbcTemplate.queryForObject(sql,
new UserRowMapper(), id);
});
}
}
Async Processing Tracing
@Async Methods
@Service
public class NotificationService {
@Async
@Observed(name = "notification.send")
public CompletableFuture<Void> sendNotificationAsync(String recipient, String message) {
// Async notification logic
return CompletableFuture.completedFuture(null);
}
}
Manual Trace Propagation
@Service
public class EmailService {
private final Tracer tracer;
private final ExecutorService executorService;
public EmailService(Tracer tracer) {
this.tracer = tracer;
this.executorService = Executors.newFixedThreadPool(5);
}
public void sendEmailAsync(String recipient, String subject, String body) {
TraceContext traceContext = tracer.currentSpan().context();
executorService.submit(() -> {
try (Tracer.SpanInScope ws = tracer.withSpanInScope(
tracer.toSpan(traceContext))) {
Span span = tracer.nextSpan()
.name("email.send")
.tag("email.recipient", recipient)
.start();
try (Tracer.SpanInScope emailScope = tracer.withSpanInScope(span)) {
// Send email logic
sendEmailInternal(recipient, subject, body);
} finally {
span.end();
}
}
});
}
private void sendEmailInternal(String recipient, String subject, String body) {
// Email sending implementation
}
}
Production Configuration
Performance Optimizations
management:
tracing:
sampling:
probability: 0.01 # Sample 1% in production
rate: 1000 # Max 1000 traces per second
baggage:
enabled: false # Disable if not needed
remote-fields: []
zipkin:
tracing:
endpoint: "${ZIPKIN_ENDPOINT:http://zipkin:9411/api/v2/spans}"
timeout: 1s
connect-timeout: 1s
# Optimize logging for performance
logging:
pattern:
level: "%5p [%X{traceId:-},%X{spanId:-}]"
level:
io.micrometer.tracing: WARN
org.springframework.web.servlet.mvc.method.annotation: WARN
Security Considerations
management:
endpoints:
web:
exposure:
include: "health,info,metrics"
exclude: "trace" # Don't expose trace endpoint
endpoint:
trace:
enabled: false
tracing:
baggage:
correlation:
enabled: false # Disable MDC correlation if sensitive data
remote-fields: [] # Don't propagate sensitive fields
Troubleshooting
Common Issues
- No traces appearing: Check sampling probability and endpoint configuration
- High overhead: Reduce sampling probability or disable baggage
- Missing spans: Ensure proper dependency injection of ObservationRegistry
- Broken trace context: Check async processing and thread boundaries
Debug Configuration
logging:
level:
io.micrometer.tracing: DEBUG
io.opentelemetry: DEBUG
brave: DEBUG
zipkin2: DEBUG
management:
tracing:
sampling:
probability: 1.0 # Sample everything for debugging
Health Check for Tracing
@Component
public class TracingHealthIndicator implements HealthIndicator {
private final Tracer tracer;
public TracingHealthIndicator(Tracer tracer) {
this.tracer = tracer;
}
@Override
public Health health() {
try {
Span span = tracer.nextSpan().name("health.check.tracing").start();
span.end();
return Health.up()
.withDetail("tracer", tracer.getClass().getSimpleName())
.build();
} catch (Exception ex) {
return Health.down()
.withDetail("error", ex.getMessage())
.build();
}
}
}
Best Practices
- Sampling Strategy: Use lower sampling rates in production (1-10%)
- Span Naming: Use consistent, meaningful span names with low cardinality
- Tag Strategy: Add meaningful tags but avoid high-cardinality values
- Error Handling: Always properly handle and tag errors in spans
- Performance: Monitor the overhead of tracing in production
- Security: Be careful not to include sensitive data in span tags or baggage
- Correlation: Use correlation IDs to link traces across service boundaries
- Testing: Include tracing in your testing strategy with TestObservationRegistry