Learn how to monitor, optimize, and scale Apache Flink in production. This expert guide covers key metrics, checkpointing, SLOs, observability tools, and configuration best practices for reliable, high-performance streaming applications.
Running Apache Flink in production is more than a one-time deployment; it requires continuous monitoring and fine-tuning to ensure optimal performance, stability, and efficiency.
This guide summarizes the key insights from our webinar delivered by Lior, “Apache Flink Production Monitoring & Optimizations Tips,” providing a practical framework for keeping your streaming applications healthy and production-ready.
This content is essential for DevOps/SRE teams, data engineers building streaming pipelines, and platform engineers managing Flink workloads on Kubernetes, managed services, or standalone clusters.
Let’s begin with some basics.
Understanding Flink’s Core Architecture
To effectively monitor Flink, it’s crucial to understand its fundamental components. A standard Flink deployment consists of two primary types of processes:
- Job Manager: This is the coordinator of your Flink cluster. It’s responsible for scheduling jobs, coordinating checkpoints, managing recovery, and overseeing the overall execution of the application. While you have a single leading Job Manager, you can configure standby managers for high availability.
- Task Managers: These are the worker nodes. They execute the actual tasks of your Flink job, which are divided into parallel units called subtasks. Task Managers are dynamic resources that can be scaled up or down based on workload demands. Each Task Manager has a number of “task slots” where the parallel instances of your job’s operators run.
A common deployment pattern is the Application Mode, where a single Flink job is submitted per deployment. In this mode, the Job Manager’s main method starts the job, requests resources (Task Manager pods in Kubernetes, for example), and then allocates the job’s tasks to the available task slots.
The Flink Job Graph: From Logic to Execution
Every Flink application is represented as a directed acyclic graph (DAG), often called the job graph. Understanding its structure is key to diagnosing performance issues.
- Operators: These are the logical building blocks of your application, representing sources (e.g., reading from Kafka), transformations (map, filter, join), and sinks (e.g., writing to a database).
- Operator Chaining: Flink optimizes performance by “chaining” consecutive operators together into a single task where possible. This avoids the overhead of shuffling data over the network or through local buffers. Chaining is not possible when an operation requires data shuffling, such as a
keyBy, join, or aggregation. - Tasks & Subtasks: The logical operator is translated into a physical execution unit called a Task. To achieve horizontal scaling, a task can have multiple parallel instances, each known as a Subtask. Each subtask processes a partition or shard of the data and runs in a dedicated task slot on a Task Manager.
The Lifeline of Durability: Checkpointing
Checkpointing is Flink’s mechanism for fault tolerance and durability. It periodically saves the state of your application—such as in-flight data, window aggregations, or join states—to durable storage like Amazon S3 or HDFS.
The checkpoint flow is a two-phase process:
- Synchronous Phase: This phase can briefly pause data processing to flush the current state. The duration of this phase is critical to monitor.
- Asynchronous Phase: The state data is uploaded to durable storage in the background.
Monitoring checkpoints is non-negotiable. Key concerns include ensuring checkpoints succeed, keeping their size and duration under control, and preventing them from infinitely growing, which could stall your application.
Defining Success: SLOs and SLIs for Flink
Before diving into specific metrics, define what success looks like for your application using Service Level Objectives (SLOs) and track them with Service Level Indicators (SLIs). This focuses your monitoring efforts on what truly matters.
Example SLOs:
- Uptime: The application should be up 99.9% of the time.
- Processing Latency: The time from a record entering the source to exiting the sink must be under a specific threshold (e.g., 5 seconds).
- Data Freshness: The event-time lag (how far behind the application is from the latest event in the source) should not exceed a few seconds.
- Recovery Time: The application must recover from failure in under a specified time.
Key SLIs to Track:
- Uptime & Stability: Job restarts, compute utilization (CPU/Memory). High resource utilization is a risk to uptime.
- Throughput & Progress: Event consumption/production rates, event-time lag. It’s crucial to ensure the job is actually processing data, not just reporting zero lag because the source is empty.
- Checkpointing Health: Checkpoint success/failure rates, duration, size, and recovery time.
Flink provides several built-in mechanisms for collecting the data needed to track your SLIs.
1. Metrics
Metrics are the primary tool for automated monitoring and alerting. Flink exposes a vast number of metrics that can be published to an observability platform.
-
Metric Reporters: The recommended approach is to configure a metric reporter to push metrics to a centralized system. Prometheus is a popular choice.
Example
flink-conf.yamlfor Prometheus Reporter:metrics.reporters: prom metrics.reporter.prom.class: org.apache.flink.metrics.prometheus.PrometheusReporter metrics.reporter.prom.port: 9999-9999With this setup, each Job Manager and Task Manager pod will expose its metrics on the specified port, which can then be scraped by a Prometheus server.
-
Custom Metrics: You can add business-specific metrics to your Flink jobs (DataStream API) using the
RuntimeContextto create custom counters, gauges, etc.
2. Logs
Logs are invaluable for debugging specific failures and understanding event-level errors.
- Configuration: Flink uses Log4j. By default, logs are written to files and the console.
- Best Practice: In production, especially in containerized environments, it’s highly recommended to use a JSON logging layout. This makes logs structured and significantly easier to parse, search, and analyze in a centralized logging system like Loki, OpenSearch, or Datadog.
3. Tracing
Tracing in Flink is still evolving but can be useful for identifying bottlenecks in specific, critical flows. Currently, its primary use is for tracing the checkpointing and job restore processes, helping to debug slow startup or recovery times.
What to Monitor: A Checklist of Key Metrics
With hundreds of metrics available, it’s easy to get lost. Focus on these key areas.
Job Stability & Resource Utilization
- JVM CPU Usage: Should consistently be under 80-85% to avoid resource starvation and long wait times.
- JVM Memory: Monitor on-heap memory usage and garbage collection (GC) time.
- Managed Memory (Off-heap): If using RocksDB, this memory is expected to be fully utilized. Monitor RocksDB-specific metrics for deeper insights.
- Network Buffers: Ensure network buffers are not exhausted, especially in high-throughput jobs with significant data shuffling.
- Job Restarts: A high number of restarts is a clear indicator of instability.
Checkpointing Performance
numberOfCompletedCheckpointsvs.numberOfFailedCheckpoints: The failure count should be zero, and the completed count should be steadily increasing.lastCheckpointSize: Watch for uncontrolled growth, which could indicate a state leak.lastCheckpointDuration: Spikes can indicate pressure on the system.lastCheckpointAlignmentTime: The time spent in the synchronous phase. High alignment time means your job is paused for longer, impacting processing latency.
Throughput and Lag
- Source Lag (
records-lag-max): The number of pending records in the source (e.g., Kafka). This is a key indicator for autoscaling. - Watermark Lag (
currentInputWatermarkvs.currentOutputWatermark): Measures the event-time delay. A growing watermark lag indicates the job cannot keep up with the data rate. - Throughput (
numRecordsInPerSecond,numRecordsOutPerSecond): Monitor the rate of data processing to understand the current load and detect stalls. - Backpressure (
isBackPressured): This boolean metric on a subtask indicates if it is being slowed down by a downstream operator.
Production-Ready Checklist
Before going live, review these critical configuration points:
-
Topology & Parallelism:
- Define your default and max parallelism.
maxParallelismis hard to change later, as it affects how state is sharded. Plan for future growth.
.
- Define your default and max parallelism.
-
Memory Configuration:
- Correctly size your on-heap, off-heap, and managed memory based on your state backend (HashMap vs. RocksDB) and workload.
-
Restart Strategy:
- Define a sane restart strategy (e.g., fixed-delay, failure-rate). An aggressive restart loop on a failing job can cause downstream issues, like producing thousands of duplicate messages into Kafka.
-
State Management & Evolution:
- Assign Operator UIDs: Manually set unique IDs for all stateful operators. This is critical for allowing state to be restored after a code change.
- Plan for Schema Evolution: Understand how changes to your state objects will impact job restarts. You may need a state migration strategy.
-
High Availability (HA):
- Configure Job Manager HA for production clusters to avoid a single point of failure.
- Treat your state as a database. Regularly take Savepoints as manual backups of your state, in addition to the automated checkpoints. This provides a clean restore point for disaster recovery or planned migrations.
By implementing this structured approach to monitoring and configuration, you can move from a reactive to a proactive operational model, ensuring your Apache Flink applications run efficiently, reliably, and at scale.
