module util::Formatters

rascal-Not specified

Tools for generating and debugging ((Box-Box))-based formatting pipelines.

Usage

import util::Formatters;

Dependencies

import Content;
import Exception;
import IO;
import Location;
import ParseTree;
import Set;
import String;
import analysis::diff::edits::ExecuteTextEdits;
import analysis::diff::edits::HiFiLayoutDiff;
import analysis::diff::edits::TextEdits;
import lang::box::\syntax::Box;
import lang::box::util::Box2Text;
import lang::box::util::Tree2Box;
import util::FileSystem;
import util::Highlight;
import util::IDEServices;
import util::Monitor;
import util::PathConfig;
import util::Reflective;

Description

This module provides a number of generative functions (functions that produce functions) to construct efficient and accurate formatters for programming languages and domain-specific languages.

All formatter generators take two specifications as parameter:

• type[Tree] grammar which specifies the input type and/or the parser to generate to re-parse formatted output.
• Style style, which uses the Style type to pass in your specification that maps all language constructors to a Box term. Typically this is an extension of the ToBox default style.

We generate formatting functions here with different input parameter types (Tree, loc or str) and different return types void for IO side-effects, list[TextEdit] for IDE integration and str for normal string output:

generator	generated formatter	comment
File Formatter	void (loc)	replaces the current file
String Formatter	str (str)
Sub String Formatter	str (type[Tree], str)	can be used on any non-terminal in the grammar
File Edit Formatter	list[TextEdit](loc)	use Execute Text Edits or Execute File System Changes or Apply File System Edits later
Tree Edit Formatter	list[TextEdit](Tree)
Sub Tree Edit Formatter	list[TextEdit](Tree)	this uses a subparser based on the top-level type of the input tree`
String Edit Formatter	list[TextEdit](str)
Sub String Edit Formatter	list[TextEdit](type[Tree], str)

If the pipeline you require is not in the above set, then it is advised to copy one of the above and adapt.

All of the above generators capture the current state of the grammar and the current state of the style function. So if you change the grammar or the ToBox style, then you have to "regenerate" and call these functions again.

During the development of a formatting style, regeneration is the default and visual feedback is essential. To accommodate the activity of developing a style incrementally, or by trial-and-error, the following debugging functions come in handy:

• Debug File Format generates a HTML or ANSI based preview of an example file
• Debug Files Format generates formatted versions of all files in an entire file hierarchy, or appends all their formatted examples to one big file, or prints that file to the console. Adds ANSI codes optionally.
• Debug String Format generates a formatter version in an HTML preview or on the console with ANSI codes.

The debug functions always take the grammar and the ToBox style parameter and these are never captured. This guarantees that while you are debugging you are always looking at the current version of your specifications.

As Formatting Options you can configure different parts of the pipeline in one go using the formattingOptions() constructor and these optional parameters:

• maxWidth - indicates a hard limit for wrapping and switching to vertical mode
• breakAfter - indicates hard limit under which wrapping and switching to vertical mode must be avoided
• tabSize - indicates the default indentation size (counting spaces)
• caseInsensitivity - CaseInsensitivity to influence how the state of each individual case-insensitive keyword us either propagated or normalized while formatting.
• needsConfirmation - triggers UI behavior with confirmation dialogs and possible previews
• insertSpaces - when set to true it prefers spaces over tabs, when set to false we use tabs for indentation (see tabSize)
• trimTrailingWhitespace - when true the formatter can not leave spaces or tabs after the last non-whitespace character on a line, when false it does not matter.
• insertFinalNewline, insert a newline character at the end of the file if one does not exist.
• trimFinalNewlines, trim all newlines after the final newline at the end of the file.

Benefits

• speed:
  1. the generators cache a generated parser for fast reuse between calls to formatting pipeline, and a fully composed ToBox function with a pre-computed dispatch hashtable.
  2. Layout Diff isolates only what changes and leafs what is unchanged alone.
• safety: if the generated formatters produce illegal or ambiguous syntax, a warning is printed and no changes are produced.
• accuracy: the generated formatters retain all comments, as long as comments are annotated with @category("Comment") or @category("comment") in the grammar.
• practicality: the generated formatters can produce Text Edits which are consumed by IDEs such as VScode.
• shyness: this setup guarantees a minimal amount of specification from the language engineer because:
  1. the default ToBox formatter takes typically 40% of the (more boring) work.
  2. the Layout Diff algorithm recovers comments, so case distinctions for existence of comments everywhere are unnecessary.
  3. the Box constraints allow for adaptive layout strategies, based on available space, which are still style-consistent for free.
  4. robustness: built-in post-formatter parser check guarantees the user is never confronted with syntactically broken output.

Pitfalls

• the formatters capture the current state of the grammar and the style function, so if you change their source code, you have to regenerate the formatters.
• case insensitivy normalization is implemented by both Tree2 Box and Hi Fi Layout Diff, for efficiency reasons we only use the Hi Fi Layout Diff implementation here.
• all pipelines here use Tree2 Box, then Box2 Text, and then parse again and then produce diffs using Hi Fi Layout Diff. Strictly speaking if you only need the full string, then you can use Tree2 Box and Box2 Text and get the formatted result a bit faster. We include Hi Fi Layout Diff here always because debugging and testing IDE features like formatting is then simulated as accurately as possible.

alias Style

A style specification maps a ((Tree)) to a ((Box-Box)) expression.

Box(Tree)

data FormattingOptions

Additional formatter options

data FormattingOptions (
    bool needsConfirmation=false,
    CaseInsensitivity caseInsensitivity = asIs()
)

• needsConfirmation when set to true triggers a confirmation dialog for applying the proposed edits. The UI may even show a preview or a diff editor.
• caseInsensitivity can be used to keep or normalize the style of case insensitivy keywords. See CaseInsensitivity for the options.

function fo

short-hand

FormattingOptions fo()

data FileSystemChange

Repeats this option from ((util::LanguageServer)) to avoid a cyclic dependency.

data FileSystemChange (bool needsConfirmation = false)

function fileFormatter

Generates a file formatter that immediately applies the new style to the input.

void(loc) fileFormatter(type[&G <: Tree] grammar, Style style, FormattingOptions opts=fo())

The formatter function this generates will produce formatted string directly and replace the original file with it.

• grammar should be the grammar for the entire language (a start non-terminal preferably)
• style is typically an extended version of ToBox
• opts provides the Formatting Options

Benefits

• can be used to seemlessly integrate a box-based formatter in an IDE for "format file"-purposes.
• will do nothing if the formatter breaks syntax rules or introduces ambiguity.
• will never lose any comments.
• is parameterized by a Style function to support different styles of formatting for the same language
• can be used to normalize case-insensitive literals in one go

Pitfalls

• only applicable to entire files. For selection-based formatting see Sub Tree Edit Formatter and Sub String Edit Formatter.
• in rare case the recovered comments are positioned in awkward places.
• can be slow for very very large input

function allFilesFormatter

Convenience function for batch formatting with ((fileFormatter))

void(loc) allFilesFormatter(type[&G <: Tree] grammar, Style style, FormattingOptions opts=fo(), str extension="rsc")

function stringFormatter

Generates a string formatter

str(str) stringFormatter(type[&G <: Tree] grammar, Style style, FormattingOptions opts=fo())

The formatter function this generates will produce formatted string directly.

• grammar should be the grammar for the entire language (a start non-terminal preferably)
• style is typically an extended version of ToBox
• opts provides the Formatting Options

Benefits

• can be used to seemlessly integrate a box-based formatter in an IDE for "format selection"-purposes.
• will do nothing if the formatter breaks syntax rules or introduces ambiguity.
• will never lose any comments.
• is parameterized by a Style function to support different styles of formatting for the same language
• can be used to normalize case-insensitive literals in one go

Pitfalls

• only applicable to entire files. For selection-based formatting see Sub String Edit Formatter.
• in rare case the recovered comments are positioned in awkward places.
• can be slow for very very large input

function subStringFormatter

Generates a substring formatter

str(type[Tree], str) subStringFormatter(type[&G <: Tree] grammar, Style style, FormattingOptions opts=fo())

The formatter function this generates will produce a formatted substring.

• grammar should be the grammar for the entire language (a start non-terminal preferably)
• style is typically an extended version of ToBox
• opts provides the Formatting Options

The subStringFormatter does not have to be called on a top-level parse tree (with a start non-terminal), but rather works on any sub-sentence:

1. first the given sub-sentence will be parsed and formatted with its left-most margin at column 0.
2. then the original indentation of the old sub-string will be computed.
3. then the formatted output will be indented (expect the first line).
4. then the whitespace differences between the old and the new string will be computed. Also lost comments will be re-inserted.
5. finally the resulting string is produced.

Benefits

• can be used to seemlessly integrate a box-based formatter in an IDE for "format selection"-purposes.
• learns the appropriate indentation level for the sub-tree from the input.
• learns the non-terminal to use for parsing from the input sub-tree.
• will do nothing if the formatter breaks syntax rules or introduces ambiguity.
• will never lose any comments.
• is parameterized by a Style function to support different styles of formatting for the same language
• can be used to normalize case-insensitive literals in one go

Pitfalls

• in rare case the recovered comments are positioned in awkward places.
• can be slow for very very large input

function fileEditFormatter

Generates a file formatter that produces ((TextEdit))s for downstream processing.

list[TextEdit](loc) fileEditFormatter(type[&G <: Tree] grammar, Style style, FormattingOptions opts=fo())

The formatter function this generates will produce a list of Text Edits. The edits change the original input to the formatted output.

• grammar should be the grammar for the entire language (a start non-terminal preferably)
• style is typically an extended version of ToBox
• opts provides the Formatting Options

Benefits

• can be used to seemlessly integrate a box-based formatter in an IDE for "format selection"-purposes.
• will do nothing if the formatter breaks syntax rules or introduces ambiguity.
• will never lose any comments.
• is parameterized by a Style function to support different styles of formatting for the same language
• can be used to normalize case-insensitive literals in one go

Pitfalls

• only applicable to entire files. For selection-based formatting see Sub Tree Edit Formatter and Sub String Edit Formatter.
• in rare case the recovered comments are positioned in awkward places.
• can be slow for very very large input

function treeEditFormatter

Generates a file formatter that produces ((TextEdit))s from a ((Tree)) for downstream processing.

list[TextEdit](&G <: Tree) treeEditFormatter(type[&G <: Tree] grammar, Style style, FormattingOptions opts=fo())

The formatter function this generates will produce a list of Text Edits. The edits change the original input to the formatted output.

• grammar should be the grammar for the entire language (a start non-terminal preferably)
• style is typically an extended version of ToBox
• opts provides the Formatting Options

Benefits

• can be used to seemlessly integrate a box-based formatter in an IDE for "format string"-purposes.
• will do nothing if the formatter breaks syntax rules or introduces ambiguity.
• will never lose any comments.
• is parameterized by a Style function to support different styles of formatting for the same language
• can be used to normalize case-insensitive literals in one go

Pitfalls

• only applicable to entire files. For selection-based formatting see Sub Tree Edit Formatter.
• in rare case the recovered comments are positioned in awkward places.
• can be slow for very very large input

function subTreeEditFormatter

Generates a file formatter that produces ((TextEdit))s from sub((Trees)s

list[TextEdit](Tree) subTreeEditFormatter(type[&G <: Tree] grammar, Style style, FormattingOptions opts=fo())

The formatter function this generates will produce a list of Text Edits. The edits change the original input to the formatted output.

• grammar should be the grammar for the entire language (a start non-terminal preferably)
• style is typically an extended version of ToBox
• opts provides the Formatting Options

The subTreeEdits formatter does not have to be called on a top-level parse tree (with a start non-terminal), but rather works on any sub-tree:

1. first the given sub-tree will be formatted with its left-most margin at column 0.
2. then the original indentation of the old sub-tree will be computed.
3. then the formatted output will be indented (expect the first line).
4. then the whitespace differences between the old and the new tree will be computed. Also lost comments will be re-inserted.
5. finally the resulting edits are produced.

Benefits

• can be used to seemlessly integrate a box-based formatter in an IDE for "format selection"-purposes.
• learns the appropriate indentation level for the sub-tree from the input.
• learns the non-terminal to use for parsing from the input sub-tree.
• will do nothing if the formatter breaks syntax rules or introduces ambiguity.
• will never lose any comments.
• is parameterized by a Style function to support different styles of formatting for the same language
• can be used to normalize case-insensitive literals in one go

Pitfalls

• in rare case the recovered comments are positioned in awkward places.
• can be slow for very very large input

function delabel

Symbol delabel(label(_, Symbol s))

Symbol delabel(conditional(Symbol s, _))

default Symbol delabel(Symbol s)

function stringEditFormatter

Generates a string formatter that produces ((TextEdit))s for downstream processing.

list[TextEdit](str) stringEditFormatter(type[&G <: Tree] grammar, Style style, FormattingOptions opts=fo())

The formatter function this generates will produce a list of Text Edits. The edits change the original input to the formatted output.

• grammar should be the grammar for the entire language (a start non-terminal preferably)
• style is typically an extended version of ToBox
• opts provides the Formatting Options

Benefits

• can be used to seemlessly integrate a box-based formatter in an IDE for "format-string"-purposes.
• will do nothing if the formatter breaks syntax rules or introduces ambiguity.
• will never lose any comments.
• is parameterized by a Style function to support different styles of formatting for the same language
• can be used to normalize case-insensitive literals in one go

Pitfalls

• in rare case the recovered comments are positioned in awkward places.
• can be slow for very very large input

function subStringEditFormatter

Generates a string formatter that produces ((TextEdit))s, from substrings with a given nonterminal, for downstream processing.

list[TextEdit](type[Tree], str) subStringEditFormatter(type[&G <: Tree] grammar, Style style, FormattingOptions opts=fo())

The formatter function this generates will produce a list of Text Edits. The edits change the original input to the formatted output.

• grammar should be the grammar for the entire language (a start non-terminal preferably)
• style is typically an extended version of ToBox
• opts provides the Formatting Options

The subTreeEdits formatter does not have to be called on a top-level parse tree (with a start non-terminal), but rather works on any sub-tree:

1. first the given sub-tree will be formatted with its left-most margin at column 0.
2. then the original indentation of the old sub-tree will be computed.
3. then the formatted output will be indented (expect the first line).
4. then the whitespace differences between the old and the new tree will be computed. Also lost comments will be re-inserted.
5. finally the resulting edits are produced.

Benefits

• can be used to seemlessly integrate a box-based formatter in an IDE for "format selection"-purposes.
• learns the appropriate indentation level for the sub-tree from the input.
• learns the non-terminal to use for parsing from the input sub-tree.
• will do nothing if the formatter breaks syntax rules or introduces ambiguity.
• will never lose any comments.
• is parameterized by a Style function to support different styles of formatting for the same language
• can be used to normalize case-insensitive literals in one go

Pitfalls

• in rare case the recovered comments are positioned in awkward places.
• can be slow for very very large input

function subIndent

Restores indentation for formatted sub-trees

str subIndent(loc span, str replacement, str original)

This algorithm ignores the first line, since the first line is always preceded by the layout of a parent node.

Then it measures the depth of indentation of every line in the original, and takes the minimum. That minimum indentation is stripped off every line that already has that much indentation in the replacement, and then all lines are re-indented with the discovered minimum.

function debugFileFormat

Generates an HTML-based preview of the result of formatting.

void debugFileFormat(type[&G <: Tree] grammar, Style style, loc input, FormattingOptions opts=fo(), bool HTML=true, bool console=!HTML, bool dumpEdits=false)

Benefits

• uses Highlight to create an HTML preview of the formatted code
• can also print directly to the console with ANSI coloring
• regenerates parser and style functions every time to avoid looking at stale code.

Pitfalls

• not useful for production contexts

function debugFilesFormat

Generates previews of the result of formatting for all files in a directory structure.

void debugFilesFormat(type[&G <: Tree] grammar, Style style, loc root, str extension, FormattingOptions opts=fo(), bool shadowFiles = true, bool appendFile=!shadowFiles, bool console=!shadowFiles && !appendFile, bool ansi=console || appendFile, bool dumpEdits = false)

Benefits

• uses writeFile to generate a file preview of the formatted code in a shadow directory structure (easy for using git or diff)
• can also stream to the console with ANSI colors for quickly checking formatted style.
• regenerates parser and style functions every time to avoid looking at stale code.

Pitfalls

• not useful for production contexts
• may have a very long running time

function debugStringFormat

Generates an in-memory preview of the result of formatting and opens an HTML viewer for it, or prints the results directly to the console.

void debugStringFormat(type[&G <: Tree] grammar, Style style, str input, FormattingOptions opts=fo(), bool HTML = true, bool console = !HTML)

Benefits

• opens the new file with the same file extension to activate possible language servers (syntax highlighting)
• regenerates parser and style functions every time to avoid looking at stale code.

Pitfalls

• not useful for production contexts

function debugFileToBox

Prints the indented intermediate Box format for an input file, using the given grammar and Box style.

void debugFileToBox(type[&G <: Tree] grammar, Style style, loc input, bool flatten = true)

function debugStringToBox

Prints the indented intermediate Box format for an input string, using the given grammar and Box style.

void debugStringToBox(type[&G <: Tree] grammar, Style style, str input, bool flatten = true)