[FLINK-38777][history] HistoryServer supports application archives #27430
Conversation
| <td><h5>historyserver.archive.retained-applications</h5></td> | ||
| <td style="word-wrap: break-word;">-1</td> | ||
| <td>Integer</td> | ||
| <td>The maximum number of applications to retain in each archive directory defined by org.apache.flink.configuration.description.TextElement@ae3540e. This option works together with the TTL (see <code class="highlighter-rouge">historyserver.archive.retained-ttl</code>). Archived entities will be removed if their TTL has expired or the retention count limit has been reached. <br />If set to `-1`(default), there is no limit to the number of archives. If set to <code class="highlighter-rouge">0</code> or less than <code class="highlighter-rouge">-1</code>, HistoryServer will throw an <code class="highlighter-rouge">IllegalConfigurationException</code>. <br />Note, when there are multiple history server instances, two recommended approaches when using this option are: <ul><li>Specify the option in only one HistoryServer instance to avoid errors caused by multiple instances simultaneously cleaning up remote files, </li><li>Or you can keep the value of this configuration consistent across them. </li></ul></td> |
There was a problem hiding this comment.
What is org.apache.flink.configuration.description.TextElement@ae3540e - is this supposed to be a link?
what is <br />. This seems to be an end br tag with a space in. what does this mean?
There was a problem hiding this comment.
This was caused by a mistake in the HistoryServerOptions description. I will fix it and regenerate the docs. Thanks for pointing it out!
| "Whether HistoryServer should cleanup jobs that are no longer present in the archive directory defined by %s. ", | ||
| code(HISTORY_SERVER_ARCHIVE_DIRS.key())) | ||
| .linebreak() | ||
| .text(LEGACY_NOTE_MESSAGE) |
There was a problem hiding this comment.
we should deprecate the options that only apply to the legacy case. I assume there is an intention to remove them in the next version change
There was a problem hiding this comment.
Hi @davidradl Thank you for raising this point. After some thought, I believe there are some nuances worth clarifying regarding the History Server.
These configuration options specifically apply to job archives generated prior to FLINK-38761 (most likely before Flink 2.3). Until that legacy format reaches its end-of-service, the History Server must remain capable of parsing such archives and providing users with the necessary configuration parameters to control their behavior. Therefore, these options are not deprecated and will not be removed in the next major version.
That said, if you feel this approach is problematic or have concerns about maintaining these options, I’m happy to discuss further!
There was a problem hiding this comment.
Hi @davidradl I'd appreciate a quick confirmation—are there any remaining concerns or questions from your side? If not, the PR should be ready to merge.
github-actions
bot
added
the
community-reviewed
flink-core/src/main/java/org/apache/flink/configuration/HistoryServerOptions.java
Outdated
Show resolved
Hide resolved
...runtime-web/src/test/java/org/apache/flink/runtime/webmonitor/history/HistoryServerTest.java
Outdated
Show resolved
Hide resolved
...java/org/apache/flink/runtime/webmonitor/history/HistoryServerApplicationArchiveFetcher.java
Outdated
Show resolved
Hide resolved
4 participants
What is the purpose of the change
This pull request make HistoryServer support application archives while ensuring that legacy job archives are handled properly.
Brief change log
Verifying this change
This change added tests and can be verified as follows:
Does this pull request potentially affect one of the following parts:
@Public(Evolving): (no)Documentation