java.util.regex.PatternSyntaxException for HelmIgnorePathMatcher.addPatterns()

[gavinfish]

When I use sample codes to install helm charts, I got below exception:

java.util.regex.PatternSyntaxException: Unclosed character class near index 23
^templates/\.[^\]?[^\]*$
                       ^
	at java.util.regex.Pattern.error(Pattern.java:1957)
	at java.util.regex.Pattern.clazz(Pattern.java:2550)
	at java.util.regex.Pattern.clazz(Pattern.java:2506)
	at java.util.regex.Pattern.sequence(Pattern.java:2065)
	at java.util.regex.Pattern.expr(Pattern.java:1998)
	at java.util.regex.Pattern.compile(Pattern.java:1698)
	at java.util.regex.Pattern.<init>(Pattern.java:1351)
	at java.util.regex.Pattern.compile(Pattern.java:1028)
	at org.microbean.helm.chart.HelmIgnorePathMatcher.addPatterns(HelmIgnorePathMatcher.java:274)
	at org.microbean.helm.chart.HelmIgnorePathMatcher.addPattern(HelmIgnorePathMatcher.java:189)
	at org.microbean.helm.chart.HelmIgnorePathMatcher.<init>(HelmIgnorePathMatcher.java:98)
	at org.microbean.helm.chart.HelmIgnorePathMatcher.<init>(HelmIgnorePathMatcher.java:112)
	at org.microbean.helm.chart.HelmIgnorePathMatcher.<init>(HelmIgnorePathMatcher.java:163)
	at org.microbean.helm.chart.DirectoryChartLoader$PathWalker.<init>(DirectoryChartLoader.java:186)
	at org.microbean.helm.chart.DirectoryChartLoader$PathWalker.<init>(DirectoryChartLoader.java:156)
	at org.microbean.helm.chart.DirectoryChartLoader.toNamedInputStreamEntries(DirectoryChartLoader.java:145)
	at org.microbean.helm.chart.DirectoryChartLoader.toNamedInputStreamEntries(DirectoryChartLoader.java:51)
	at org.microbean.helm.chart.StreamOrientedChartLoader.load(StreamOrientedChartLoader.java:213)
	at org.microbean.helm.chart.AbstractChartLoader.load(AbstractChartLoader.java:87)
	at com.microsoft.jenkins.devspaces.commands.DeployHelmChartCommand.execute(DeployHelmChartCommand.java:40)
	at com.microsoft.jenkins.devspaces.commands.DeployHelmChartCommand.execute(DeployHelmChartCommand.java:27)
	at com.microsoft.jenkins.azurecommons.command.CommandService.runCommand(CommandService.java:88)
	at com.microsoft.jenkins.azurecommons.command.CommandService.execute(CommandService.java:96)
	at com.microsoft.jenkins.azurecommons.command.CommandService.executeCommands(CommandService.java:75)
	at com.microsoft.jenkins.azurecommons.command.BaseCommandContext.executeCommands(BaseCommandContext.java:77)
	at com.microsoft.jenkins.devspaces.DevSpacesBuilder.perform(DevSpacesBuilder.java:91)
	at hudson.tasks.BuildStepCompatibilityLayer.perform(BuildStepCompatibilityLayer.java:78)
	at hudson.tasks.BuildStepMonitor$1.perform(BuildStepMonitor.java:20)
	at hudson.model.AbstractBuild$AbstractBuildExecution.perform(AbstractBuild.java:779)
	at hudson.model.Build$BuildExecution.build(Build.java:205)
	at hudson.model.Build$BuildExecution.doRun(Build.java:162)
	at hudson.model.AbstractBuild$AbstractBuildExecution.run(AbstractBuild.java:534)
	at hudson.model.Run.execute(Run.java:1741)
	at hudson.model.FreeStyleBuild.run(FreeStyleBuild.java:43)
	at hudson.model.ResourceController.execute(ResourceController.java:98)
	at hudson.model.Executor.run(Executor.java:410)

It seems that the default constructor of HelmIgnorePathMatcher will try to call addPattern("templates/.?*") method and it will generate a regex pattern like ^templates/\.[^\]?[^\]*$ which is an invalid one. Any help here?

[ljnelson]

Something is odd here; note that this line in the relevant unit test works:

microbean-helm/src/test/java/org/microbean/helm/chart/TestHelmIgnorePathMatcher.java

Line 114 in 183f5f5

final HelmIgnorePathMatcher helmIgnorePathMatcher = new HelmIgnorePathMatcher();

It is called here:

microbean-helm/src/main/java/org/microbean/helm/chart/HelmIgnorePathMatcher.java

Line 98 in 183f5f5

this.addPattern("templates/.?*");

I wonder if it is a JDK version thing? Do you have more details about your situation?

[gavinfish]

My jdk version is

openjdk version "1.8.0_181"
OpenJDK Runtime Environment (build 1.8.0_181-8u181-b13-1ubuntu0.16.04.1-b13)
OpenJDK 64-Bit Server VM (build 25.181-b13, mixed mode)

If you want to test in this environment, you should specify the surefire plugin version as 3.0.0-M1, more information SUREFIRE-1588.

[ljnelson]

It's going to be a problem in your .helmignore file, or a problem that this class has reading it (perhaps still a legitimate bug). It is not the default constructor that is the problem. The error is really occurring from here:

microbean-helm/src/main/java/org/microbean/helm/chart/HelmIgnorePathMatcher.java

Line 163 in 183f5f5

this(helmIgnoreFile == null ? (Collection<? extends String>)null : Files.readAllLines(helmIgnoreFile, StandardCharsets.UTF_8));

That's going to read in a .helmignore file and pass the patterns to the addPattern method. Then each .helmignore pattern is translated into a regular expression.

Reverse engineering the error message, it looks like this regex:

^templates/\.[^\]?[^\]*$

…is resulting from the following .helmignore template:

templates/.?*

which doesn't make much sense to me. Could you send me the offending .helmignore file so I may write a test to reproduce the issue?

The fact that \ shows up also tells me this is Windows, not Ubuntu.

[gavinfish]

My .helmignore file:

# Patterns to ignore when building packages.
# This supports shell glob matching, relative path matching, and
# negation (prefixed with !). Only one pattern per line.
.DS_Store
# Common VCS dirs
.git/
.gitignore
.bzr/
.bzrignore
.hg/
.hgignore
.svn/
# Common backup files
*.swp
*.bak
*.tmp
*~
# Various IDEs
.project
.idea/
*.tmproj

My windows jdk version:

java version "1.8.0_191"
Java(TM) SE Runtime Environment (build 1.8.0_191-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.191-b12, mixed mode)

BTW, where does the .helmignore should be put? My file structure here is:

chart
  --templates
      --service.yaml
      ....
  --Chart.yaml
  --values.yaml
  --.helmignore

[ljnelson]

This gets weirder and weirder.

Note that the Travis build (Linux) passes:

https://travis-ci.org/microbean/microbean-helm/jobs/432367395#L821

It is running with JDK 8 build 151:

https://travis-ci.org/microbean/microbean-helm/jobs/432367395#L441

But I note that now on my (Mac) laptop with JDK 8 build 181 the test fails:

[INFO] --- maven-surefire-plugin:2.20.1:test (default-test) @ microbean-helm ---
[INFO] Toolchain in maven-surefire-plugin: JDK[/Library/Java/JavaVirtualMachines/jdk1.8.0_181.jdk/Contents/Home]
[INFO] 
[INFO] -------------------------------------------------------
[INFO]  T E S T S
[INFO] -------------------------------------------------------
[INFO] Running org.microbean.helm.chart.TestHelmIgnorePathMatcher
[ERROR] Tests run: 23, Failures: 2, Errors: 0, Skipped: 0, Time elapsed: 0.147 s <<< FAILURE! - in org.microbean.helm.chart.TestHelmIgnorePathMatcher
[ERROR] testStuff[cargo/, cargo, true](org.microbean.helm.chart.TestHelmIgnorePathMatcher)  Time elapsed: 0.009 s  <<< FAILURE!
java.lang.AssertionError: cargo/ did not match cargo expected:<true> but was:<false>
	at org.junit.Assert.fail(Assert.java:88)
	at org.junit.Assert.failNotEquals(Assert.java:834)
	at org.junit.Assert.assertEquals(Assert.java:118)
	at org.microbean.helm.chart.TestHelmIgnorePathMatcher.testStuff(TestHelmIgnorePathMatcher.java:128)

Investigating further but looks like maybe a regex regression in JDK 8.

[ljnelson]

Hmm, even weirder. I can now reproduce this error at will on my laptop using JDK 8 builds 144, 162 and 181. There is definitely a bug in here somewhere.

However, in order to actually release microbean-helm to Maven Central, I had to run mvn release:perform, which runs all tests and fails if any of them fail. So given that I've released many versions of microbean-helm, and given that master is (currently) what is released, and given that I haven't touched either the test case in question or HelmIgnorePathMatcher.java, what has changed to cause this test to fail? Still investigating.

[ljnelson]

Somewhat more level-headed refinement: I can't reproduce the error actually described here. And I can make all the tests pass if I run mvn test instead of mvn test -Dtest=TestHelmIgnorePathMatcher.

So the pattern compilation is working fine on environments that use / as their file separator. It is not a JVM version issue.

At issue now is what happens when the file separator is \ (your issue) and the also disturbing issue where mvn test -Dtest=TestHelmIgnorePathMatcher fails.

[daiqingliang]

I got this problem too.
But this is a little weirder, it work fine on Centos7 and get this error on Windows.
Just DirectoryChartLoader.load() have this error when I want read chart from directory, TapeArchiveChartLoader.load() work fine when I read a XXX.tgz on Windows.
As we know, I can't run k8s and helm Windows, so I just ignore this error...
Error code:

String directoryPath = "C:/Users/daiqingliang/Desktop/helm-kubeapps/my-hello-world";
DirectoryChartLoader loader = new DirectoryChartLoader();
ChartOuterClass.Chart.Builder chart = loader.load(Paths.get(directoryPath));
MetadataOuterClass.MetadataOrBuilder metadata = chart.getMetadataOrBuilder();

exception:

java.util.regex.PatternSyntaxException: Unclosed character class near index 23
^templates/\.[^\]?[^\]*$
                       ^
	at java.util.regex.Pattern.error(Pattern.java:1957)
	at java.util.regex.Pattern.clazz(Pattern.java:2550)
	at java.util.regex.Pattern.clazz(Pattern.java:2506)
	at java.util.regex.Pattern.sequence(Pattern.java:2065)
	at java.util.regex.Pattern.expr(Pattern.java:1998)
	at java.util.regex.Pattern.compile(Pattern.java:1698)
	at java.util.regex.Pattern.<init>(Pattern.java:1351)
	at java.util.regex.Pattern.compile(Pattern.java:1028)
	at org.microbean.helm.chart.HelmIgnorePathMatcher.addPatterns(HelmIgnorePathMatcher.java:274)
	at org.microbean.helm.chart.HelmIgnorePathMatcher.addPattern(HelmIgnorePathMatcher.java:189)
	at org.microbean.helm.chart.HelmIgnorePathMatcher.<init>(HelmIgnorePathMatcher.java:98)
	at org.microbean.helm.chart.HelmIgnorePathMatcher.<init>(HelmIgnorePathMatcher.java:112)
	at org.microbean.helm.chart.HelmIgnorePathMatcher.<init>(HelmIgnorePathMatcher.java:163)
	at org.microbean.helm.chart.DirectoryChartLoader$PathWalker.<init>(DirectoryChartLoader.java:186)
	at org.microbean.helm.chart.DirectoryChartLoader$PathWalker.<init>(DirectoryChartLoader.java:156)
	at org.microbean.helm.chart.DirectoryChartLoader.toNamedInputStreamEntries(DirectoryChartLoader.java:145)
	at org.microbean.helm.chart.DirectoryChartLoader.toNamedInputStreamEntries(DirectoryChartLoader.java:51)
	at org.microbean.helm.chart.StreamOrientedChartLoader.load(StreamOrientedChartLoader.java:213)
	at org.microbean.helm.chart.AbstractChartLoader.load(AbstractChartLoader.java:87)
	at com.hisense.kubeappshelm.mytest.TestReadChartByDireactory.TestLocalChart(TestReadChartByDireactory.java:40)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
	at org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
	at org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
	at org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
	at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
	at org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
	at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
	at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
	at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
	at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
	at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
	at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
	at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
	at com.intellij.junit4.JUnit4IdeaTestRunner.startRunnerWithArgs(JUnit4IdeaTestRunner.java:68)
	at com.intellij.rt.junit.IdeaTestRunner$Repeater.startRunnerWithArgs(IdeaTestRunner.java:33)
	at com.intellij.rt.junit.JUnitStarter.prepareStreamsAndStart(JUnitStarter.java:230)
	at com.intellij.rt.junit.JUnitStarter.main(JUnitStarter.java:58)

Process finished with exit code 0