to-directory-tree doesn't create subdirectories

[timbertson]

aside: thanks for for-directory-tree, I'm very excited to use it :D

$ echo '{ a = "hello" }' | dhall to-directory-tree --output issue
$ cat issue/a
hello

$ echo '{ `a/b` = "hello" }' | dhall to-directory-tree --output issue
dhall: issue/a/b: openFile: inappropriate type (Not a directory)

If I mkdir -p issue/a first it works, but that's obviously inconvenient.

While I'm at it, my initial assumption was that to-directory-tree would take a map, instead of a record literal. That seems like it could be more flexible (because the keys can be programmatically generated too), and would also remove the need to support Optional Text (you could just filter out the unwanted items from the map). Should I make a separate issue for that, or does it intentionally not support maps?

[sjakobi]

@timbertson Instead of

$ echo '{ `a/b` = "hello" }' | dhall to-directory-tree --output issue

you can use

$ echo '{ a = { b = "hello" } }' | dhall to-directory-tree --output issue

Does that work for you?

[Profpatsch]

We should disallow / in the keys for that and tell the user that they should use nested records instead.

[Gabriella439]

Yeah, I would be fine with changing the to-directory-tree command to disallow / in the keys

[timbertson]

Oh, I was kind of relying on that actually. I am combining a number of files from multiple places, and (for example) a couple of unrelated files both belong in `scripts/` (one for docker, one for Travis). Having to pluck those out from each place and put them into a record makes it hard to combine multiple file sets.

…

On Sat, 18 Jan 2020, 7:13 am Gabriel Gonzalez, ***@***.***> wrote: Yeah, I would be fine with changing the to-directory-tree command to disallow / in the keys — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1633?email_source=notifications&email_token=AAADOXHYSSVMXWQY7ZI5UCDQ6IGO7A5CNFSM4KIBRJM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJI22BI#issuecomment-575778053>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADOXE2VOPKTIFAQKB7JHDQ6IGO7ANCNFSM4KIBRJMQ> .

[sjakobi]

Hehe, that reminds me of https://xkcd.com/1172/. (No offense intended)

@timbertson Could you illustrate a bit what you're doing and how you want to manipulate these file sets in Dhall?

AFAIK, the idea of the to-directory-tree command was that you can use Dhall's record operators (/\, //, .{...}, etc.) to manipulate and combine directory trees / file sets.

[timbertson]

Sure. It may be that I can use /\ to do everything I need, it just may be fragile when multiple things come into play (recursive merges and overrides).

The basic idea is that I want to package up some common boilerplate, and reference it (via remote import) from many git repos. So (greatly) simplified I might have a Travis.dhall module which provides:

{
, `.travis.yaml` = "TODO"
, `script/lint.sh` = "TODO"
}

The script/lint.sh path would be referenced in .travis.yaml, so the user of this module shouldn't be responsible for choosing where that file goes.

Then say I've got a Docker.dhall module which provides:

{
, `Dockerfile` = "TODO"
, `script/compile.sh` = "TODO"
}

Again, the script path is used by the Dockerfile, so it should all be under control of this module.

To bring together the final set, I was doing ./Travis.dhall // ./Docker.dhall // { ... extra files as a record literal }

I believe I can use nested records and /\. But thinking ahead, I expect I might want some default provided files (for convenience) which some users may wish to override or opt out of. I could do that with, say:

{
, `Dockerfile` = "TODO"
, script = {
  , `compile.sh` = "TODO"
  , `optionalStuff.sh` = Some Text "TODO"
  }
}

Which then makes it quite difficult to combine and override:

let dockerStuff =
  let base = ./Docker.dhall
  in
  base // { script = base.script // { `optionalStuff.sh` = None Text } }

in
./Travis.dhall /\ dockerStuff /\ { ... extra files as a record literal }

So it all seems possible, but:

• Having to manually merge multiple levels with // to override is awkward
• Knowing when to use // vs /\ is not obvious (using /\ when you want to override will at least give you an error, but doing the opposite will result in silently omitting files in any overridden subdirectory (script/ in this case))

Flat records with directory-separated paths makes all this much simpler, because the data structure is flat. I can see how it's awkward to have two representations for the same thing though. So perhaps this is just another use case for dhall-lang/dhall-lang#448 or dhall-lang/dhall-lang#340

[sjakobi]

Thanks for the great explanation!

• Having to manually merge multiple levels with // to override is awkward

Indeed. Lenses should eventually make this easier but they're not here yet.

• Knowing when to use // vs /\ is not obvious (using /\ when you want to override will at least give you an error, but doing the opposite will result in silently omitting files in any overridden subdirectory (script/ in this case))

Fair point. IMHO it's generally good to start with /\, and to use // only when fields must be overridden. You could also give record completions a try.

I can see how it's awkward to have two representations for the same thing though.

Yes, I think that would make things more complicate down the road.

I think you should give nested records a try for now. If that turns out to be too inergonomic or cause other problems, please report back, and we'll try to find a solution.

[timbertson]

Yeah, I'll give that a try. I hopefully won't actually need overrides much or at all in the end anyway. Thanks for the discussion. Should I leave this open (and/or rename) to track "disallow keys with a slash in them"?

…

On Sat, 18 Jan 2020 at 20:35, Simon Jakobi ***@***.***> wrote: Thanks for the great explanation! - Having to manually merge multiple levels with // to override is awkward Indeed. Lenses <dhall-lang/dhall-lang#754 (comment)> should eventually make this easier but they're not here yet. - Knowing when to use // vs /\ is not obvious (using /\ when you want to override will at least give you an error, but doing the opposite will result in silently omitting files in any overridden subdirectory ( script/ in this case)) Fair point. IMHO it's generally good to start with /\, and to use // only when fields must be overridden. You could also give record completions <http://hackage.haskell.org/package/dhall-1.29.0/docs/Dhall-Tutorial.html#g:12> a try. I can see how it's awkward to have two representations for the same thing though. Yes, I think that would make things more complicate down the road. I think you should give nested records a try for now. If that turns out to be too inergonomic or cause other problems, please report back, and we'll try to find a solution. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1633?email_source=notifications&email_token=AAADOXHDTAIDP4FNQZHFMRLQ6LENRA5CNFSM4KIBRJM2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOEJJUMAY#issuecomment-575882755>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAADOXCWO2J4TIDA63LAP3TQ6LENRANCNFSM4KIBRJMQ> .

[Gabriella439]

@timbertson: For what it's worth, I think we might be able to support deep overrides after all and I have a comment related to that here: dhall-lang/dhall-lang#754 (comment)

[bfrk]

@timbertson wrote:

While I'm at it, my initial assumption was that to-directory-tree would take a map, instead of a record literal. That seems like it could be more flexible (because the keys can be programmatically generated too), and would also remove the need to support Optional Text (you could just filter out the unwanted items from the map). Should I make a separate issue for that, or does it intentionally not support maps?

+1 from me. I was just going to open an issue to request this feature.

[sjakobi]

@timbertson, @bfrk By "map" you mean this type, right?

List { mapKey : Text, mapValue : Text }

[bfrk]

Yup. Also known as https://prelude.dhall-lang.org/Map/Type

[harris-chris]

This is a very useful feature, but I seem to be hitting a bit of a limitation:

• if you're passing in a record, the keys are set at the type level, so you can't have an arbitrary or programmatically defined number of files/directories at any given level of your structure
• but if you're passing in a fileMap, then every item in the list of the fileMap has to have the same type, therefore every branch of the directory structure has to have the same depth
  I have a function which would ideally generate a config folder which looks a bit like this:

config
|
| -> sub-config-1
|         |
|         | -> _x many files_
|
| -> sub-config-2
          | 
          | -> sub-config-2-1
                    |
                    | -> _y many files_

where x and y vary across different calls of the function. I don't think this is possible from a single dhall to-directory-tree call because of the limitation above. Up until now I've been calling the function multiple times but this is quite slow (it takes a number of minutes to generate all the config).

[Gabriella439]

@harris-chris: Do you have a specific Dhall type in mind that would represent your desired directory tree? The reason I ask is that I can extend dhall to-directory-tree to support additional Dhall types for this purpose

[harris-chris]

@harris-chris: Do you have a specific Dhall type in mind that would represent your desired directory tree? The reason I ask is that I can extend dhall to-directory-tree to support additional Dhall types for this purpose

Thanks for the response. I'm sure you'd have a better idea than I, but I can't really see a particularly good solution there, at least not without a heterogenous list type. If dhall to-directory-tree accepted a fileMap in the following format then that would also work:

[ 
    { mapKey = "config/sub-config-1/file1", mapValue = "file 1 contents" }
    , { mapKey = "config/sub-config-2/sub-config-2-1/file2", mapValue = "file 2 contents" }
]

However I'd agree with the comments above, that that approach is a bit clunky compared to having nested records.

For the real-life situation I have here, I've just contrived the structure of the config files to have the same depth across the board, which is a perfectly fine work-around.

[Gabriella439]

What if dhall to-directory-tree were to support Prelude.JSON.Type? Would that work for you?

[harris-chris]

That sounds like it would be a convenient solution, yes. So am I right in thinking that from bash I'd do something like, say:

$ dhall to-directory-tree --output my-directory-tree | dhall-to-json "(./file-containing-a-normal-record-expression.dhall)"

[Gabriella439]

What I mean is that the input would be a Dhall expression of this type:

https://store.dhall-lang.org/Prelude-v21.1.0/JSON/Type.dhall.html

… but you could create a value of that type using json-to-dhall if you wanted to

[harris-chris]

Right, of course. Yes, that would work well.

…

On Sun, 3 Jul 2022, 11:48 Gabriella Gonzalez, ***@***.***> wrote: What I mean is that the input would be a Dhall expression of this type: https://store.dhall-lang.org/Prelude-v21.1.0/JSON/Type.dhall.html … but you could create a value of that type using json-to-dhall if you wanted to — Reply to this email directly, view it on GitHub <#1633 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALDMVSA2DI2JGHHQMKNGHYLVSD5ODANCNFSM4KIBRJMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>

[mmhat]

I'd vote for something like:

forall (a : Type) ->
  let EntryType = < Directory : List a | File : Text >

  let Entry = { name : Text, type : EntryType }

  in  forall (make : Entry -> a) -> List a

I'd also like to extend the Entry with fields like user : < Id : Natural | Name : Text > and permissions : Text - But that's a story for another issue.

[harris-chris]

Hello - another observation about this feature - is there a particular reason that it can't be used to create empy folders? For example,

dhall to-directory-tree <<< "[ { mapKey = \"empty_folder\", mapValue = [] : List { mapKey : Text, mapValue : Text } } ]" --output testing

is an invalid expression for to-directory-tree. Setting mapValue to None will create an empty testing folder, but with no empty_folder sub-folder.

[mmhat]

@harris-chris A workaround for that looks like:

let empty_directory = { empty = None Text }
in
{ this_is_an_empty_directory = empty_directory }

Admitted, this is not a nice solution but at least it works. I proposed a more sophisticated option for to-directory-tree in #2436. An implementation draft of that idea is here; Feel free to give that a try.

[harris-chris]

Thank you, this does seem like a viable work-around.

…

On Fri, 12 Aug 2022, 20:15 Mann mit Hut, ***@***.***> wrote: @harris-chris <https://github.com/harris-chris> A workaround for that looks like: let empty_directory = { empty = None Text }in { this_is_an_empty_directory = empty_directory } Admitted, this is not a nice solution but at least it works. I proposed a more sophisticated option for to-directory-tree in #2436 <#2436>. An implementation draft of that idea is here <https://github.com/mmhat/dhall-haskell/tree/1633-improved-to-directory-tree>; Feel free to give that a try. — Reply to this email directly, view it on GitHub <#1633 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ALDMVSFP4CHH65WCAQF6MSDVYY56FANCNFSM4KIBRJMQ> . You are receiving this because you were mentioned.Message ID: ***@***.***>