Add terminal and color escape sequences

[awvwgk]

• implement low-level API for color support
• style, foreground and background color enumerators
• true color (24-bit) types
• generation of escape strings via to_string function

Questions:

• how to test this?
• aim for compile time conversion?

Related

• ANSI Colors support #229
• https://github.com/awvwgk/fortty/blob/main/src/escape.f90
• https://github.com/urbanjost/M_attr/blob/main/src/M_attr.f90
• https://github.com/jupyter-xeus/cpp-terminal/blob/master/cpp-terminal/base.hpp

[14NGiestas]

Looking good.
Testing if they produce the expected code escapes should be enough since this is a low-level API,
users should be aware that no checks are done to ensure that the terminal will support colors.

[certik]

I am personally not a fan using derived types. The ideal for this would be to use enumerations (@FortranFan would probably approve:), in fact that's what we ended up using in C++ that you linked above.

We also have to initialize the terminal (especially on Windows).

I don't know if there is a way to simplify this somehow. If not, then the present approach is fine.

[certik]

One question: this module talks about "colors", but these are specifically terminal colors, and in fact only "ansi escape sequences", but it seems all terminals now converges to support those, including Windows. Should this be perhaps named "terminal colors"?

[certik]

And since we require to setup a terminal (on Windows) even to just print colors, we might as well use this machinery for other terminal things, such as the goal of the cpp-terminal library. Then we can write good terminal applications.

I think this is within scope of stdlib (although I always struggle if we should create a separate package or add it to stdlib).

[ivan-pi]

I am not a user of terminal colors (yet), however upon reviewing the module I was wondering if abbreviating the components as r, g, b would be just as clear but more concise. This might be just a personal preference, but I think it gives a nice clean look when initializing a color using the structure constructor:

type(fg_color24) :: my_color

my_color = fg_color24(r=34, g=12, b=10)   ! vs
my_color = fg_color24(red=34, green=12, blue=10)

[awvwgk]

Thanks for the feedback. Building something like cpp-terminal in stdlib for Fortran would be awesome, eventually we could just use stdlib_terminal for this purpose. The color escape codes would be the first step (maybe as stdlib_terminal_colors) to define our low-level API for handling escape sequences.

I usually start building this kind of projects separately and than notice that simple things like to_string for integer output are something I'm already missing. Usually, I'm just copying the snippet (at least for to_string), but I figured it would be better to start doing things properly and just contribute it directly to stdlib.

[ivan-pi]

I am personally not a fan using derived types. The ideal for this would be to use enumerations (@FortranFan would probably approve:), in fact that's what we ended up using in C++ that you linked above.

I don't think the fact Fortran doesn't support proper enums should be a show stopper. Here is a small quote from the Lua community:

Our motto in the design of Lua has always been "mechanisms instead of policies." By policy, we mean a methodical way of using existing mechanisms to build a new abstraction. Encapsulation in the C language provides a good example of a policy. The ISO C specification offers no mechanism for modules or interfaces. Nevertheless, C programmers leverage existing mechanisms (such as file inclusion and external declarations) to achieve those abstractions.

Derived types offer a nice and type-safe way to organize code, even if they might lead to reduced performance in some cases.
I don't think this usage case is so performance-critical, that derived types would represent an issue. Moreover, when the derived types are default-constructed and used as parameters, an optimizing compiler can just copy the value and insert it in the right place of the client code (at least the NAG compiler is known to do this).

For ANSI Escape sequences specifically, the other two options I see available are 1) using a character string, like the M_attr module does, or 2) use an array of 3 integers. (In principle you could encode all 24 bits in a single integer, but I guess it makes code less clear.)

Would it be feasible to overload the // operator for the foreground and background color types?

[awvwgk]

Would it be feasible to overload the // operator for the foreground and background color types?

I actually thought about overloading operator(+) to create combinations from style, fg_color and bg_color. This will create a bit of combinatorics blowup if not designed properly, therefore I just went with the simple variant first to get the discussion going.

[ivan-pi]

Here's the example @certik posted in #229:

 std::string text = "Some text with "
            + color(fg::red) + color(bg::green) + "red on green"
            + color(bg::reset) + color(fg::reset) + " and some "
            + color(style::bold) + "bold text" + color(style::reset) + ".";
        std::cout << text << std::endl;

The same can be achieved in Fortran. I guess using operator(+) between the colors and styles makes more sense, however I'd still go for // to attach the escape sequence to a string. I'd also leave it unsymmetric, i.e. the escape sequence is always the left operand:

type(fg_color) :: green, red

print *, green // "Sentence in green"   ! valid
print *, "Red sentence" // red          ! invalid 

Edit: on second thought, overloading // hides the fact these are not two strings... Perhaps that's not desirable and a function or custom operator (or even +) would be better.

[certik]

I don't think the fact Fortran doesn't support proper enums should be a show stopper.

I agree. All I am advocating for is to consider and search for the simplest solution that involves the least amount of "layers" (such as derived types or even more OO approach). Btw, I am not convinced the C++ I approach I took is the best either.

I guess the only approach I can think of is to have functions like color_fg_bright_3bit and it would accept an integer and we define integer constants like color_red, but it's easy to pass the wrong integer in. It's less safe than the current approach with derived types I think.

That's why I CCed @FortranFan, because this use case is far from unique, I think this is actually quite common.

[urbanjost]

Since Fortran now supports (some more than others) everything from procedural to functional to OOP programming there are a lot of options and all of us have different approaches (some of mine and others discussed in M_esc and to a lesser extent M_attr) but if going with OOP and user defined types I might want a string that can have attributes (bg color, fg color, blink, underline, ...) that I can set that I would otherwise use like the STRING type in stdlib but that on output I could optionally print with all the attributes applied; albeit if putting out a lot of short strings with similar attributes it would take some work to not produce redundant output when all the strings have a common attribute.

I think some example programs solving some common use cases would help.

Looks good given the approach taken, but unless you want to expand this to allow building panels and./or ASCII graphics it seems a little overkill; but since it works and something is needed and it has the attribute of being expandable to include positioning and clearing I am good with it.

Having been using an ncurses interface and more recently M_attr I am having a hard time getting through the "not what I am used to" roadblock while trying it, but everything I have tried so far has worked.

I personally need something I can easily read and write from an external file and turn off so I need more of an abstraction of the attributes I can easily embed in text files; but I think a lot of people just want an easy way to color some text and this works for that so I am good with this.

[FortranFan]

@certik wrote Nov. 28, 2021 11:52 AM EDT:

I guess the only approach I can think of is to have functions like color_fg_bright_3bit and it would accept an integer and we define integer constants like color_red, but it's easy to pass the wrong integer in. It's less safe than the current approach with derived types I think.

That's why I CCed @FortranFan, because this use case is far from unique, I think this is actually quite common.

Thanks @certik.

I agree the use case here is hardly unique: scientific and technical computing is replete with the need to work with named constants with particular scopes in type-safe manner even in compute-intensive sections of codebases.

The need here to work with color codes for terminal escape sequences is but one instance that some might consider is outside of hard-code number crunching but it should not trivialized.

Nonetheless proper ENUMs are a bridge way too far for Fortran, the solution in the next revision Fortran 202X is far from optimal.

Thus moving ahead with this Fortran stdlib item in sync with all your consensus is as good as it can get.

[jvdp1]

@awvwgk is this ready for review?

[awvwgk]

I think we have agreed that terminal escape sequences are generally in-scope for stdlib. The question now is how make best use of them in an actual application and I don't really have a definite answer, this touches points like:

1. how to represent a color (string, enum, integer, derived type, ...)
2. applying color to strings (function, binary/unary operator, ...)
3. ways to enable/disable color support at runtime (atty detection, ...)

I haven't touched point 3 here in this PR, but I think this is one of the most important points to cover. However, I don't think there is a best answer.

The most practical solution I was able to come up with is here, using a derived type to hold all escape sequences and initialize them with empty string if color support is disabled, while deferring the decision of color usage to the user of the library. But I doubt this is the best strategy, since it requires explicit initialization.

[14NGiestas]

Point 3 is not a blocking issue for this PR to go IMO, it produces all the code escapes needed to color a text. We can just let the user of the module deal with it and provide helpers later on (E.g: a portable isatty function).
Some other options that do not involve a higher-level abstraction:

• creating a private global variable (module-wise) and a function to turn on / off the colors.
• using an environment variable (I don't know if that is portable, however it worth doing it)

[jvdp1]

@awvwgk Thank you for the answers. I can't answer your questions. However, I would suggest to finish this PR as is, and ask users some feedbacks regarding these 3 questions. With fpm, people could test it quite easily IMO.

[milancurcic]

Thank you for this addition. I reviewed the files and played with it locally. It works as expected and I think it can be merged as is, and we can add any higher-level functionality in later PRs.

I agree with @certik that we should look for the simplest and lowest-level approach, though I think the derived types here are appropriate. Alternatively, the color derived types could also be expressed as integer :: fg_color24(3) arrays, but to me the current implementation is more intuitive.

What's not so intuitive to me is that we have two separate types fg_color24 and bg_color24 (and corresponding "enum" types) that are otherwise exactly the same. As I understand it, this is designed like this so that a generic to_string can figure out whether to emit a background or foreground styling. As a user, I would prefer for colors to be just colors (e.g. type color24), but use different functions (to_string_fg, to_string_bg, etc.) to apply different kind of stylings. In other words, in my mind foregroundness and backgroundness are not properties of a color, but a property of a styling operation. I hope that makes sense. But I also understand this is a matter of style, so I'm not opposed to the current approach.

One question that came up for me is if the color components of the derived types should be integer(int8) (as that would make the types truly 24-bit :)). Memory footprint would be four times smaller (are there any use cases where the memory use would be important, i.e. large arrays of color values? I can't think of it. In general graphics sure, but in terminal? I don't know. Maybe somebody will start writing rich-color roguelikes in Fortran). The default constructor would be uglier (e.g. fg_color24(0_int8, 127_int8, 255_int8)), but that could be easily fixed by providing a custom type constructor function. Another advantage could be type casting: Currently if you pass values out-of-range (e.g. < 0 or > 255) the style simply won't work but the program would otherwise be correct. But if you pass out-of-range values to int8, a compiler could emit helpful warnings if enabled.

[milancurcic]

@awvwgk Do you think there is sufficient interest and feedback to wrap up and merge this PR? If yes, I'd be happy to help resolve the merge conflicts.

[awvwgk]

Didn't have time to look into this PR for a while now.

Working with color support for a bit I was somewhat unhappy with the API defined here for my projects, as I usually need a more high-level interface to turn off the color printout. The overall structure defined here might be just right to build such a low level interface.

I'm fine with finishing this PR and merging it as is, still there is some way to go.

[awvwgk]

The only thing a little off is that the derived type is called ansi_color but it represents not only a color but an entire ansi_style or ansi_escape.

Just rebased and renamed the derived type to ansi_code to better reflect that it can contain a style while keeping the name short (and avoiding British / American English variants).

[ivan-pi]

ansi_code is very reasonable given that "ANSI code" redirects to ANSI Escape Codes on Wikipedia.

[ivan-pi]

The way this is currently written one might wrongly interpret that codes can be "added" using the the // operator. Only after I checked the interface I realized this was not the case.

I've got no good suggestions how to amend this.

[ivan-pi]

A few comments:

1. I'm not 100% convinced by the naming of the module and constants. The derived type name ansi_code is good, but I wonder if stdlib_ansi or stdlib_ansi_codes would be better for the module name? For the colors, I feel like the "color" word is just extra characters. There is the actual color name in the variable name anyway, and it could be just stated in the documentation that parameters prepended with fg_ and bg_ are colors. A second consideration I had was to prepend ansi_ to all named parameters for extra clarity, but I'm not fully decided if I like it or not.
2. I find the right-overwriting rule of the addition operator a bit nonintuitive the way it reads in the documentation. I suppose the correct usage pattern is to add up to one code from each group (fg, bg, and style). In that case the over-writing direction doesn't actually matter, unless you are "recycling" code variables. The graphics language Asymptote offers something similar for combining different colored pens:

Pens may be added together with the nonassociative binary operator +. This will add the colors of the two pens. All other non-default attributes of the rightmost pen will override those of the leftmost pen. Thus, one can obtain a yellow dashed pen by saying dashed+red+green or red+green+dashed or red+dashed+green.

3. It would be interesting to look at some micro-optimizations for the ANSI code to string translation step in the future. Granted, colors are typically not used in performance critical sections. I'm just being pedantic.
4. We can offer a functional API in addition to the operator one. This would reset the sequence by default. There could also be a second function with a logical flag for resetting.

pure function colorize(code, str)
  type(ansi_code), intent(in) :: code
  character(len=*), intent(in) :: str
  character(len=:), allocatable :: colorize
  colorize = code // str // style_reset
end function

5. Should string type be supported to? Can also be deferred to a new PR.
6. NO_COLOR could be easily supported in the future by changing the behavior of to_string_ansi_code.
7. Perhaps I missed the discussion, but what was the reason for making the structure constructor private? Is it to force users to initialize it explicitly after the declaration section? This ties back to comment 2).

Overall, I think it's a great how this new "low-level" interface turned out. I say low-level even if derived types and operator overloading is involved. After-all this is the "standard" library, and compiler vendors can choose to implement this in a more primitive fashion if they really want to. In that case ansi_code should probably be a sequence type to prevent access of the internals through inheritance.

[awvwgk]

I'm not 100% convinced by the naming of the module and constants. The derived type name ansi_code is good, but I wonder if stdlib_ansi or stdlib_ansi_codes would be better for the module name? For the colors, I feel like the "color" word is just extra characters. There is the actual color name in the variable name anyway, and it could be just stated in the documentation that parameters prepended with fg_ and bg_ are colors. A second consideration I had was to prepend ansi_ to all named parameters for extra clarity, but I'm not fully decided if I like it or not.

Naming is always hard, stdlib_ansi might work, I will rename those modules.

NO_COLOR could be easily supported in the future by changing the behavior of to_string_ansi_code.

This module is developed with support for easily disabling colors in mind, if you do not initialize your escape sequence with a color you get the NO_COLOR behavior, however you can still safely use the escape sequences as if there were color.

Perhaps I missed the discussion, but what was the reason for making the structure constructor private? Is it to force users to initialize it explicitly after the declaration section? This ties back to comment 2).

What should happen if the user initializes the escape sequence with bg=huge(1_i1). To avoid the headache of dealing with faulty user input the constructor is private.

Should string type be supported to? Can also be deferred to a new PR.

Can be supported as well, should only require two new procedures.

We can offer a functional API in addition to the operator one. This would reset the sequence by default. There could also be a second function with a logical flag for resetting.

Isn't code // str // style_reset already functional style?

[jvdp1]

@awvwgk If you are satisfied with this PR, I suggest that you merge it, such that users can test it, e.g., through fpm.

[awvwgk]

I'm already using the module in a project and will soon also adopt it in TOML Fortran, the design works nicely so far for my usage.

Now that we have the tree-shaking in fpm I could adopt it TOML Fortran via stdlib, but I'm still hesitant to limit myself in the compiler choice (I have to support GCC 5 and NAG 7 which are both not supported with stdlib but currenly work with TOML Fortran).

[milancurcic]

Thanks Sebastian for the contribution and all reviewers. I'll merge now.