I think the real win will be in third-party parsers of provable 100% accuracy. PHP in the default install, compiled C on a heavy-duty site or for intensive post-processing.

Tim has said we don’t necessarily have to preserve every stupid corner case and piece of emergent behaviour … unless they’re used by people and considered part of wikitext. This is a soft boundary.

As to parsing: converting regular expressions to EBNF is relatively trivial, in might not even be needed, depending on what framework you use to build the parser. The problem is that mediawiki syntax is not regular [2], and very likely not LL-1 or LR-1 [1] either. I’m not even convinced that it’s even context-free in all cases [4] (parsing non-CF grammar is very hard, even theoretically). It also has an annyoing tendency to depend on localization and customization: a promintent example are image links. They follow a different grammar than normal links, but can only be detected when knowing all local names of the image namespace. Another problems are templates that are “syntactically incomplete”: you can not parse the template code individually, and then use the result. You need a separate preprocessor pass (using a limited grammar) to resolve them (which requires database access). Same goes for some parser functions. This makes it very hard to detach the parser from the rest of the system.

For the parser tests: the are badly named, really — what they test is code generation, not parsing as such. What we currently have is a “munger” that mogrifies the wikitext until it resembles html — a real parser would not generate html. It would generate a parse tree (or parse events), and a code generator would be plugged into that. For a wysiwyg engine, it might not generate html code at all, but a DOM tree or something like that. And for exporting, mit might generate something else, like TeX (that would rock).

The advantage of having a formal grammar would be to allow people to build parsers on different platforms — for php to be used in MediaWiki itself, in javascript for a web based wysiwym-editor, in python for use with bots, etc.

My point is: it’s not just a lot of work someone needs to do. There are conceptual problems with this. On of the biggest is that due to the reliance on configuration, localization, database content and extension-defined syntax, a formal grammar in the academic sense (EBNF or a production grammar [5]) is not even possible.

To me this means that we can not hope for a simple or clean solution. We can only try to build something that is better than what we have, and live with the quirks.

[1] http://en.wikipedia.org/wiki/User:Cacycle/wikEd
[2] http://en.wikipedia.org/wiki/Regular_grammar
[3] http://en.wikipedia.org/wiki/LL_parser resp. http://en.wikipedia.org/wiki/LR_parser
[4] http://en.wikipedia.org/wiki/Context_free_grammar
[5] http://en.wikipedia.org/wiki/Production_%28computer_science%29