module analysis::grammars::ParserTesting
Reusable test functions to run (generated) parsers on a test corpus
Usage
import analysis::grammars::ParserTesting;
Dependencies
extend lang::rascal::\syntax::Rascal;
import Exception;
import Grammar;
import IO;
import List;
import Message;
import ParseTree;
import String;
import analysis::grammars::Ambiguity;
import lang::rascal::grammar::definition::Modules;
import lang::rascal::grammar::definition::Symbols;
import util::Monitor;
import util::Reflective;
syntax ParserTestModule
start syntax ParserTestModule
= @synopsis{A syntax for files that can contain many small test cases for a parser}
Tags tags "module" QualifiedName name Import* imports ParserTestCase* cases;
syntax ParserTestCase
syntax ParserTestCase
= "syntax" "test" Name? label "(" Sym symbol ")" "expects" ParserTestExpect expect TagString content;
syntax ParserTestExpect
syntax ParserTestExpect
= ok: "ok"
| ambiguous:"ambiguous"
| error: "error"
;
alias FileParser
&T (loc org)
function reportParserTestModule
Run the tests embedded in a parser test module file and return true iff all the tests succeeded.
bool reportParserTestModule(loc m, PathConfig pcfg=pathConfig())
Benefits
- all failing tests are reported in standard fashion on stdout
- when grammars remain unchanged on disk, the tests reuse parsers from the previous run
Pitfalls
- parser test module files have to be loaded, as well as Rascal module files, and converted to a parser, which involves critical file IO and the right Path Config instance.
function runParserTestModule
Run the tests embedded in a parser test module file and return true iff all the tests succeeded.
list[Message] runParserTestModule(loc m, PathConfig pcfg=pathConfig())
Benefits
- all failing tests are reported in standard fashion as a
list[Message], which can be processed further downstream by communicating to IDE diagnostics views, or as editor annotations, for example. - when grammars remain unchanged on disk, the tests reuse parsers from the previous run
Pitfalls
- parser test module files have to be loaded, as well as Rascal module files, and converted to a parser, which involves critical file IO and the right Path Config instance.
function doRunParserTestModule
The workhorse parses the test specifications, composes the main grammar, and iterates over each individual test
list[Message] doRunParserTestModule(ParserTestModule m, PathConfig pcfg=pathConfig())
function deslash
str deslash(str input)
function collectRascalModules
set[Module] collectRascalModules(Import* imports, set[str] done, PathConfig pcfg=pathConfig())
set[Module] collectRascalModules((Import) `extend <QualifiedName name>;`, set[str] done, PathConfig pcfg=pathConfig())
set[Module] collectRascalModules((Import) `import <QualifiedName name>;`, set[str] done, PathConfig pcfg=pathConfig())
default set[Module] collectRascalModules(Import _, set[str] done, PathConfig pcfg=pathConfig())
function runParserTestCase
Message runParserTestCase(ParserTestCase testCase, &T (type[&T <: Tree], str) theParser)
function constructParsingTestCorpus
Collect files of a specific extension from a folder recursively, while filtering ignored files and folders
list[loc] constructParsingTestCorpus(loc root, set[str] extensions, set[loc] ignores={})
function testCorpusForParseErrorsAndAmbiguity
Run a generated parser on a corpus and print errors, diagnostics and summary statistics
bool testCorpusForParseErrorsAndAmbiguity(type[&T <: Tree] grammar, list[loc] corpus, str jobLabel="Testing <grammar>", bool abort=false)
This functions returns true if and only if none of the files in the corpus exhibit a parse error or an ambiguity for the given grammar
Pitfalls
- the grammar is best if it's a start symbol (since we are parsing files we need spaces before and after the top sort).
function testCorpusForParseErrorsAndAmbiguity
Run a parser on a corpus and print errors, diagnostics and summary statistics
bool testCorpusForParseErrorsAndAmbiguity(FileParser[&T] parser, list[loc] corpus, str jobLabel="Testing parser ", bool abort=false)
This functions returns true if and only if none of the files in the corpus exhibit a parse error or an ambiguity
Benefits
- this is a typical simple task for every project that has a grammar
- ambiguity reported here can be analyzed using analysis::grammars::Ambiguity or the drambiguity project
Pitfalls
- the parser must not
allowAmbiguity=true - the parser must not
allowRecovery=trueeither - the parser must accept whitespace and comments at the beginning and end of the file