Lack of compatibility with HCL

[saginadir]

Awesome work and great initiative, but this is kinda misleading because this is not compatible with HCL the output of both parsers is different.

I might take this on and create a parser for hcl for python, but meanwhile I'll use yaml for my projects.

Anyway, I've opened this issue because it's maybe worth to add a warning for future comers that this is not compatible with the original HCL.

But really, love the work ❤️

[scottbelden]

Do you have some examples of the differences? The code is not set in stone and can be updated if there are examples of what functionality is missing.

[virtuald]

When created in 2014, this project was a literal translation of the HCL source code at the time, so it's definitely compatible with the 'original' HCL. It's possible there are features they've added since then that we've missed, but as Scott said it should be easy enough to add them if you can produce testcases.

[saginadir]

Sure!

Input file:

version = 1

variable "one" {
    a = 1
    b = 2
}

variable "one" {
    a = 3
    b = 4
}

variable "two" {
    hk = 12
    bw = ["big", "array"]
}

output using json2hcl

{
  "variable": [
    {
      "one": [
        {
          "a": 1,
          "b": 2
        }
      ]
    },
    {
      "one": [
        {
          "a": 3,
          "b": 4
        }
      ]
    },
    {
      "two": [
        {
          "bw": [
            "big",
            "array"
          ],
          "hk": 12
        }
      ]
    }
  ],
  "version": 1
}

hcltool output (by pyhcl):

{
    "variable": {
        "one": {
            "a": 3,
            "b": 4
        },
        "two": {
            "bw": [
                "big",
                "array"
            ],
            "hk": 12
        }
    },
    "version": 1
}

beside the 2nd missing "one" variable which is an open issue.

for some reason hcl decided to wrap each layer in an array.

[virtuald]

Oh interesting, did they finally create an official hcl to json converter? Ok, that's fair, we definitely don't do it their way because they had been discussing a json converter for awhile and then I stopped paying attention.

If there is an official canonical json output now, I'm happy to have pyhcl switch to use it (along with a corresponding major version bump to indicate the incompatibilities)... though I have to say I don't really like the arrays.

Either way, I don't really use HCL anymore these days, so I'm not inclined to do the work myself, but happy to accept pull requests.

[saginadir]

@virtuald from a 2nd look at this issue.

Looks like HCL current state is in mess. I don't think it's worth doing anything until they'll release HCL2 (if they'll release it)

I'm saying that because according to their documentation:

But the actual result of the json is:

{
  "variable": [
    {
      "ami": [
        {
          "description": "the AMI to use"
        }
      ]
    }
  ]
}

So.. I'll stay away from HCL for a while until this mess is cleared.

[virtuald]

@plainsane just raised this issue yet again, so it seems like this is a popular problem.

I would say that we should ignore their documentation, and focus on the output of the json2hcl tool. I think the right approach is to do something like so:

• Add these breaking tests (include examples from Add failing structure_list2 decoder test #22 and allow multiple providers #44)
• Use the golang json2hcl tool and convert all of pyhcl's unit tests to use their new resulting output.
• Fix all of the pyhcl unit tests to pass
• Bump the major version of pyhcl and release the breaking changes to the world

This seems to me to be the best way to make pyhcl continue to be useful in the future.

.. unfortunately, I don't have the time or motivation to do it as I haven't used HCL in years at this point. Perhaps @saginadir or @plainsane would be interested in doing this?

[virtuald]

I was curious what the transformed HCL files would look like... they're quite different. I pushed the transformed files to the hcl2json branch.

[saginadir]

@virtuald Great work. Thanks for the updates.

I might take this as a pet project in the near future. I will let you know if it happens.

[retr0h]

@saginadir this might shed some light as to the rationale behind the output.

I'm actually in the process of switching over to ucl, since it is fairly well maintained, and I'm not tied to terraform or hcl.

[saginadir]

@retr0h ucl looks good for some cases although I'm quite concerned with the coverage libucl, and therefore will likely not use it in my projects (same for hcl atm..)

The entire ucl/hcl concept feels like it wasn't explored properly and has a potential for something greater in the long run.

[retr0h]

@saginadir sure, but UCL at least conforms to it's standard where this project does not adhere to the current HCL standard which is a larger concern.