#1638 - Upgrade tree tagger#1639
Merged
Merged
Conversation
- Update TreeTagger binaries in build.xml: Linux 3.2.2 -> 3.2.5, macOS Intel 3.2.2 -> 3.2.3, Windows 3.2.2 -> 3.2.3a - Add native binaries for new platforms: macOS Apple Silicon (osx-aarch64, M1 3.2.3) and Linux ARM64 (linux-aarch64 3.2) - Retire 32-bit Windows binary (windows-x86_32) — no longer offered upstream (404) - Bump treetagger-bin artifact version to 20260614.0 and refresh executable MD5 checksums - Add aarch64 architecture support to PlatformDetector so M1/ARM64 binaries are selected at runtime - Register updated TreeTagger models (new upstream MD5s and dates, suffix reset to 0): German 2020-06-22, French 2023-03-13, Polish 2021-06-02 - Refresh German POS test reference data: add ADJNE to the STTS tagset - Refresh Polish POS test reference data: NKJP-style sentence tags (nom case, ncol, interp:sent) and full 1620-tag tagset array - Expand pl-ncp-pos.map with 745 additive entries to cover the 2021 Polish model's NKJP tagset (collectivity :col/:ncol, interp:* subtypes, agglutinated _aglt compounds) - Fix JUnit 5 assertEquals argument order in SegmenterCompatibilityTest.checkLemma - Reconcile Mongolian model version mismatch (pom pinned a non-existent version matching no build.xml output) - Normalize build.xml line endings to LF - treetagger-bin 20190408.0 -> 20260614.0 - treetagger-model-tagger-de-le 20190409.1 -> 20200622.0 - treetagger-model-tagger-fr-le 20190404.1 -> 20230313.0 - treetagger-model-tagger-mn-le 20141011.1 -> 20120925.1 - treetagger-model-tagger-pl-le 20150506.1 -> 20210602.0
- Update Spanish, Estonian, and Portuguese TreeTagger models to their latest upstream versions (new MD5s, dates, suffix reset to 0) - Add es/et/pt model dependencies to the use-proprietary-resources test profile - Complete the Spanish crater tagset map: add 58 entries (common nouns, verbs, adverbs, prepositions, pronouns, etc.), classified from the official Spanish TreeTagger tagset - Complete the Estonian tartu tagset map: add 29 entries (A.pos.sg/pl.* -> POS_ADJ, Z.Sent -> POS_PUNCT) - Populate the Portuguese gamallo tagset map (previously an empty stub) with 35 entries; 6 undocumented tags (B, E, F, G, IL, L) mapped to POS_X - Add testSpanish, testEstonian, and testPortuguese with model-derived reference data - Update URL for sharoff chinese model - Normalize pt-gamallo-pos.map line endings to LF - treetagger-model-tagger-es-le 20161222.1 -> 20210927.0 - treetagger-model-tagger-et-le 20110124.1 -> 20190904.0 - treetagger-model-tagger-pt-le 20101115.2 -> 20260615.0
- Add Catalan, Swedish and Danish TreeTagger POS tagger models - build.xml: register new ca, da and sv tagger targets and add them to the separate-jars list - ca: catalan.par (md5 e4a47f6..., upstream 2018-10-15), tagset "tt" - da: danish.par (md5 75ccb4a..., upstream 2018-10-15), tagset "parole-tt" - sv: swedish.par (md5 fb81823..., upstream 2021-10-01), tagset "suc-tt" - pom.xml: add model dependencies treetagger-model-tagger-ca-le 20181015.0, treetagger-model-tagger-da-le 20181015.0 and treetagger-model-tagger-sv-le 20211001.0 (dependencyManagement + use-proprietary-resources test profile) - Add coarse POS mapping ca-tt-pos.map (Catalan, 255 tags, UD-style heads) - Add coarse POS mapping sv-suc-tt-pos.map (Swedish, 143 fine-grained SUC tags) - Add coarse POS mapping da-parole-tt-pos.map (Danish, 174 PAROLE/ePOS tags; Z* punctuation determined empirically) - Tagsets extracted from the models via TT4J; non-ASCII keys escaped as \uXXXX for ISO-8859-1 properties loading - TreeTaggerPosTaggerTest: add testCatalan, testSwedish and testDanish (lemmas, tags, mapped POS and full tagset assertions)
- Add Norwegian, Ukrainian and Hungarian TreeTagger POS tagger models - build.xml: register new no, uk and hu tagger targets and add them to the separate-jars list - no: norwegian.par (md5 0ed1085..., upstream 2019-05-21), tagset "tt" - uk: ukrainian.par (md5 02187da..., upstream 2023-01-30), tagset "tt" - hu: hungarian.par (md5 1067ffe..., upstream 2020-12-12), tagset "tt" - pom.xml: add model dependencies treetagger-model-tagger-no-le 20190521.0, treetagger-model-tagger-uk-le 20230130.0 and treetagger-model-tagger-hu-le 20201212.0 (dependencyManagement + use-proprietary-resources test profile) - Add coarse POS mapping no-tt-pos.map (Norwegian, 210 tags, UD universal-POS heads) - Add coarse POS mapping uk-tt-pos.map (Ukrainian, 19 tags, UD universal-POS heads; SENT->POS_PUNCT, _->POS_X) - Add coarse POS mapping hu-tt-pos.map (Hungarian, 924 tags, UD universal-POS heads) - Tagsets extracted from the models via TT4J - TreeTaggerPosTaggerTest: add testNorwegian, testUkrainian and testHungarian (lemmas, tags, mapped POS and full tagset assertions)
- Add Indonesian and Romanian TreeTagger POS tagger models - build.xml: register new id and ro tagger targets and add them to the separate-jars list - id: indonesian.par (md5 b8b4404..., upstream 2022-01-03), tagset "tt" - ro: romanian.par (md5 3bb6819..., upstream 2018-10-15), tagset "tt" - pom.xml: add model dependencies treetagger-model-tagger-id-le 20220103.0 and treetagger-model-tagger-ro-le 20181015.0 (dependencyManagement + use-proprietary-resources test profile) - Add coarse POS mapping id-tt-pos.map (Indonesian, 25 tags, Dinakaramani et al. 2014 tagset) - Add coarse POS mapping ro-tt-pos.map (Romanian, 649 tags, MULTEXT-East MSD mapped by leading letter) - Tagsets extracted from the models via TT4J - TreeTaggerPosTaggerTest: add testIndonesian and testRomanian (lemmas, tags, mapped POS and full tagset assertions)
- Add Ancient Greek, Albanian and Persian TreeTagger POS tagger models - build.xml: register new grc, sq and fa tagger targets and add them to the separate-jars list - grc: ancient-greek.par (md5 b4ce02b..., upstream 2018-10-15), tagset "tt" - sq: albanian.par (md5 f0ab0fc..., upstream 2024-02-23), tagset "tt" - fa: persian.par (md5 3e3714b..., upstream 2022-10-31), tagset "tt" - pom.xml: add model dependencies treetagger-model-tagger-grc-le 20181015.0, treetagger-model-tagger-sq-le 20240223.0 and treetagger-model-tagger-fa-le 20221031.0 (dependencyManagement + use-proprietary-resources test profile) - Add coarse POS mapping grc-tt-pos.map (Ancient Greek, 15 tags, self-documenting word-class names) - Add coarse POS mapping sq-tt-pos.map (Albanian, 220 tags, UD universal-POS heads; PART|PRON mapped by first component) - Add coarse POS mapping fa-tt-pos.map (Persian, 412 tags, UD heads + '+'-joined portmanteaus mapped by main component + punctuation literals; distinct from fa-upc) - Tagsets extracted from the models via TT4J; non-ASCII Persian punctuation keys escaped as \uXXXX - TreeTaggerPosTaggerTest: add testAncientGreek, testAlbanian and testPersian (lemmas, tags, mapped POS and full tagset assertions)
- Add Coptic (Sahidic) and Hausa TreeTagger POS tagger models - build.xml: register new cop and ha tagger targets and add them to the separate-jars list - cop: coptic_coarse.par from CopticScriptorium/tagger-part-of-speech (md5 4a79e6a..., last commit 2018-06-10), uncompressed .par, tagset "tt" - ha: hausa_model.par from amir-zeldes/hausa (md5 8e4dfa2..., last commit 2015-08-25), uncompressed .par, tagset "tt" - pom.xml: add model dependencies treetagger-model-tagger-cop-le 20180610.0 and treetagger-model-tagger-ha-le 20150825.0 (dependencyManagement + use-proprietary-resources test profile) - Add coarse POS mapping cop-tt-pos.map (Coptic SCRIPTORIUM coarse tagset, 31 tags; mapping derived from UD_Coptic-Scriptorium XPOS->UPOS, _PPER portmanteaus -> POS_PRON except V_PPER) - Add coarse POS mapping ha-tt-pos.map (Hausa, 32 tags per Amir Zeldes README; portmanteaus mapped by first component) - Tagsets extracted from the models via TT4J - TreeTaggerPosTaggerTest: add testCoptic and testHausa (lemmas, tags, mapped POS and full tagset assertions)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What's in the PR
This PR brings the DKPro Core TreeTagger module up to date with the current TreeTagger
website, adds ARM64 platform support, refreshes the registered models to their latest
upstream versions, and broadens language coverage from ~20 to 33 languages. Every new or
updated model has its coarse POS
.mapauthored from the model's actual tagset (dumped viaTT4J) and is covered by a unit test asserting lemmas, raw tags, mapped POS and the full
tagset. Tests run from local
~/.m2under theuse-proprietary-resourcesprofile (skippedon CI, as before).
Binaries
20190408.0 -> 20260614.0)osx-aarch64) and Linux ARM64 (linux-aarch64)PlatformDetector: addaarch64/arm64architecture resolutionUpdated existing models (to latest upstream)
german.par, latest upstreamfrench.par, latest upstreampolish.par, latest upstreamspanish.par, latest upstreamestonian.par, latest upstreampt.par, latest upstreamcorpus.leeds.ac.ukURL tossharoff.github.io(byte-identical)New languages (13)
fa-upc)Tagset mappings & tests
.mapfiles for all new languages (and complete the drifted es/et/pt mappings)TreeTaggerPosTaggerTestSegmenterCompatibilityTest; normalize line endings to LFHow to test manually
build.xmlant script.Automatic testing
Documentation