Skip to content

[FLINK-38777][history] HistoryServer supports application archives #27430

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
zhuzhurk merged 1 commit into from
Jan 23, 2026
+1,113 −218
Merged

[FLINK-38777][history] HistoryServer supports application archives #27430

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
18 changes: 15 additions & 3 deletions docs/layouts/shortcodes/generated/history_server_configuration.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,11 +8,17 @@
</tr>
</thead>
<tbody>
<tr>
<td><h5>historyserver.archive.clean-expired-applications</h5></td>
<td style="word-wrap: break-word;">false</td>
<td>Boolean</td>
<td>Whether HistoryServer should cleanup applications that are no longer present in the archive directory defined by <code class="highlighter-rouge">historyserver.archive.fs.dir</code>. </td>
</tr>
<tr>
<td><h5>historyserver.archive.clean-expired-jobs</h5></td>
<td style="word-wrap: break-word;">false</td>
<td>Boolean</td>
<td>Whether HistoryServer should cleanup jobs that are no longer present `historyserver.archive.fs.dir`.</td>
<td>Whether HistoryServer should cleanup jobs that are no longer present in the archive directory defined by <code class="highlighter-rouge">historyserver.archive.fs.dir</code>. <br />Note: This option applies only to legacy job archives created before the introduction of application archiving (FLINK-38761).</td>
</tr>
<tr>
<td><h5>historyserver.archive.fs.dir</h5></td>
Expand All @@ -26,17 +32,23 @@
<td>Duration</td>
<td>Interval for refreshing the archived job directories.</td>
</tr>
<tr>
<td><h5>historyserver.archive.retained-applications</h5></td>
<td style="word-wrap: break-word;">-1</td>
<td>Integer</td>
<td>The maximum number of applications to retain in each archive directory defined by <code class="highlighter-rouge">historyserver.archive.fs.dir</code>. This option works together with the TTL (see <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>). Archived entities will be removed if their TTL has expired or the retention count limit has been reached. <br />If set to <code class="highlighter-rouge">-1</code> (default), there is no limit to the number of archives. If set to <code class="highlighter-rouge">0</code> or less than <code class="highlighter-rouge">-1</code>, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consistent across them. </li></ul></td>
</tr>
<tr>
<td><h5>historyserver.archive.retained-jobs</h5></td>
<td style="word-wrap: break-word;">-1</td>
<td>Integer</td>
<td>The maximum number of jobs to retain in each archive directory defined by <code class="highlighter-rouge">historyserver.archive.fs.dir</code>. <ul><li>If the option is not specified as a positive number without specifying <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>, all of the jobs archives will be retained. </li><li>If the option is specified as a positive number without specifying a value of <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>, the jobs archive whose order index based modification time is equals to or less than the value will be retained. </li><li>If this option is specified as a positive number together with the specified <code class="highlighter-rouge">historyserver.archive.retained-ttl</code> option, the job archive will be removed if its TTL has expired or the retained job count has been reached. </li></ul>If set to <code class="highlighter-rouge">0</code> or less than <code class="highlighter-rouge">-1</code>, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consistent across them. </li></ul></td>
<td>The maximum number of jobs to retain in each archive directory defined by <code class="highlighter-rouge">historyserver.archive.fs.dir</code>. <ul><li>If the option is not specified as a positive number without specifying <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>, all of the jobs archives will be retained. </li><li>If the option is specified as a positive number without specifying a value of <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>, the jobs archive whose order index based modification time is equals to or less than the value will be retained. </li><li>If this option is specified as a positive number together with the specified <code class="highlighter-rouge">historyserver.archive.retained-ttl</code> option, the job archive will be removed if its TTL has expired or the retained job count has been reached. </li></ul>If set to <code class="highlighter-rouge">0</code> or less than <code class="highlighter-rouge">-1</code>, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consistent across them. </li></ul><br />Note: This option applies only to legacy job archives created before the introduction of application archiving (FLINK-38761).</td>
</tr>
<tr>
<td><h5>historyserver.archive.retained-ttl</h5></td>
<td style="word-wrap: break-word;">(none)</td>
<td>Duration</td>
<td>The time-to-live duration to retain the jobs archived in each archive directory defined by <code class="highlighter-rouge">historyserver.archive.fs.dir</code>. <ul><li>If the option is not specified without specifying <code class="highlighter-rouge">historyserver.archive.retained-jobs</code>, all of the jobs archives will be retained. </li><li>If the option is specified without specifying <code class="highlighter-rouge">historyserver.archive.retained-jobs</code>, the jobs archive whose modification time in the time-to-live duration will be retained. </li><li>If this option is specified as a positive time duration together with the <code class="highlighter-rouge">historyserver.archive.retained-jobs</code> option, the job archive will be removed if its TTL has expired or the retained job count has been reached. </li></ul>If set to equal to or less than <code class="highlighter-rouge">0</code> milliseconds, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consistent across them. </li></ul></td>
<td>The time-to-live duration to retain the archived entities (jobs and applications) in each archive directory defined by <code class="highlighter-rouge">historyserver.archive.fs.dir</code>. This option works together with the retention count limits (see <code class="highlighter-rouge">historyserver.archive.retained-applications</code> and <code class="highlighter-rouge">historyserver.archive.retained-jobs</code>). Archived entities will be removed if their TTL has expired or the retention count limit has been reached. <br />If set to equal to or less than <code class="highlighter-rouge">0</code> milliseconds, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consistent across them. </li></ul></td>
</tr>
<tr>
<td><h5>historyserver.log.jobmanager.url-pattern</h5></td>
Expand Down
93 changes: 71 additions & 22 deletions flink-core/src/main/java/org/apache/flink/configuration/HistoryServerOptions.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,17 +51,6 @@ public class HistoryServerOptions {
+ " monitor these directories for archived jobs. You can configure the JobManager to archive jobs to a"
+ " directory via `jobmanager.archive.fs.dir`.");

/** If this option is enabled then deleted job archives are also deleted from HistoryServer. */
public static final ConfigOption<Boolean> HISTORY_SERVER_CLEANUP_EXPIRED_JOBS =
key("historyserver.archive.clean-expired-jobs")
.booleanType()
.defaultValue(false)
.withDescription(
String.format(
"Whether HistoryServer should cleanup jobs"
+ " that are no longer present `%s`.",
HISTORY_SERVER_ARCHIVE_DIRS.key()));

/**
* Pattern of the log URL of TaskManager. The HistoryServer will generate actual URLs from it.
*/
Expand Down Expand Up @@ -137,6 +126,24 @@ public class HistoryServerOptions {
"Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, ";
private static final String CONFIGURE_CONSISTENT =
"Or you can keep the value of this configuration consistent across them. ";
private static final String LEGACY_NOTE_MESSAGE =
"Note: This option applies only to legacy job archives created before the introduction of application archiving (FLINK-38761).";
private static final String RETAINED_STRATEGY_MESSAGE =
"Archived entities will be removed if their TTL has expired or the retention count limit has been reached. ";

/** If this option is enabled then deleted job archives are also deleted from HistoryServer. */
public static final ConfigOption<Boolean> HISTORY_SERVER_CLEANUP_EXPIRED_JOBS =
key("historyserver.archive.clean-expired-jobs")
.booleanType()
.defaultValue(false)
.withDescription(
Description.builder()
.text(
"Whether HistoryServer should cleanup jobs that are no longer present in the archive directory defined by %s. ",
code(HISTORY_SERVER_ARCHIVE_DIRS.key()))
.linebreak()
.text(LEGACY_NOTE_MESSAGE)
Copy link
Contributor

@davidradl davidradl Jan 16, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should deprecate the options that only apply to the legacy case. I assume there is an intention to remove them in the next version change

Copy link
Contributor Author

@eemario eemario Jan 19, 2026 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @davidradl Thank you for raising this point. After some thought, I believe there are some nuances worth clarifying regarding the History Server.

These configuration options specifically apply to job archives generated prior to FLINK-38761 (most likely before Flink 2.3). Until that legacy format reaches its end-of-service, the History Server must remain capable of parsing such archives and providing users with the necessary configuration parameters to control their behavior. Therefore, these options are not deprecated and will not be removed in the next major version.

That said, if you feel this approach is problematic or have concerns about maintaining these options, I’m happy to discuss further!

Copy link
Contributor Author

@eemario eemario Jan 21, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @davidradl I'd appreciate a quick confirmation—are there any remaining concerns or questions from your side? If not, the PR should be ready to merge.

.build());

public static final ConfigOption<Integer> HISTORY_SERVER_RETAINED_JOBS =
key(HISTORY_SERVER_RETAINED_JOBS_KEY)
Expand Down Expand Up @@ -164,6 +171,52 @@ public class HistoryServerOptions {
code("IllegalConfigurationException"))
.linebreak()
.text(NOTE_MESSAGE)
.list(
text(CONFIGURE_SINGLE_INSTANCE),
text(CONFIGURE_CONSISTENT))
.linebreak()
.text(LEGACY_NOTE_MESSAGE)
.build());

/**
* If this option is enabled then deleted application archives are also deleted from
* HistoryServer.
*/
public static final ConfigOption<Boolean> HISTORY_SERVER_CLEANUP_EXPIRED_APPLICATIONS =
key("historyserver.archive.clean-expired-applications")
.booleanType()
.defaultValue(false)
.withDescription(
Description.builder()
.text(
"Whether HistoryServer should cleanup applications that are no longer present in the archive directory defined by %s. ",
code(HISTORY_SERVER_ARCHIVE_DIRS.key()))
.build());

public static final ConfigOption<Integer> HISTORY_SERVER_RETAINED_APPLICATIONS =
key("historyserver.archive.retained-applications")
.intType()
.defaultValue(-1)
.withDescription(
Description.builder()
.text(
"The maximum number of applications to retain in each archive directory defined by %s. ",
code(HISTORY_SERVER_ARCHIVE_DIRS.key()))
.text(
"This option works together with the TTL (see %s). ",
code(HISTORY_SERVER_RETAINED_TTL_KEY))
.text(RETAINED_STRATEGY_MESSAGE)
.linebreak()
.text(
"If set to %s (default), there is no limit to the number of archives. ",
code("-1"))
.text(
"If set to %s or less than %s, HistoryServer will throw an %s. ",
code("0"),
code("-1"),
code("IllegalConfigurationException"))
.linebreak()
.text(NOTE_MESSAGE)
.list(
text(CONFIGURE_SINGLE_INSTANCE),
text(CONFIGURE_CONSISTENT))
Expand All @@ -176,18 +229,14 @@ public class HistoryServerOptions {
.withDescription(
Description.builder()
.text(
"The time-to-live duration to retain the jobs archived in each archive directory defined by %s. ",
"The time-to-live duration to retain the archived entities (jobs and applications) in each archive directory defined by %s. ",
code(HISTORY_SERVER_ARCHIVE_DIRS.key()))
.list(
text(
"If the option is not specified without specifying %s, all of the jobs archives will be retained. ",
code(HISTORY_SERVER_RETAINED_JOBS_KEY)),
text(
"If the option is specified without specifying %s, the jobs archive whose modification time in the time-to-live duration will be retained. ",
code(HISTORY_SERVER_RETAINED_JOBS_KEY)),
text(
"If this option is specified as a positive time duration together with the %s option, the job archive will be removed if its TTL has expired or the retained job count has been reached. ",
code(HISTORY_SERVER_RETAINED_JOBS_KEY)))
.text(
"This option works together with the retention count limits (see %s and %s). ",
code(HISTORY_SERVER_RETAINED_APPLICATIONS.key()),
code(HISTORY_SERVER_RETAINED_JOBS_KEY))
.text(RETAINED_STRATEGY_MESSAGE)
.linebreak()
.text(
"If set to equal to or less than %s milliseconds, HistoryServer will throw an %s. ",
code("0"), code("IllegalConfigurationException"))
Expand Down
Loading