[MLton] fix for bug in mlyacc

[Michael Norrish]

Fix bug in comment-handling in lexer for mlyacc's input language.

In particular, if a comment was successfully lexed, it returned a BOGUS_VALUE
token, which then messed up the higher-level parsing. This was due to
the use of the idiom

continue() before YYBEGIN SOMESTATE

making the return value of the continue the return value for the lexer at
that point.

This bug prevented the use of comments in code sections entirely.

It was somewhat masked in the prefix section by the fact that most
supposed comments there were never lexed as such anyway. This could
be observed by a file that included %% in a comment in the prefix
section.


diff --git a/mlyacc/src/yacc.lex b/mlyacc/src/yacc.lex
--- a/mlyacc/src/yacc.lex
+++ b/mlyacc/src/yacc.lex
@@ -71,11 +71,11 @@
qualid ={id}".";
%%
<INITIAL>"(*" => (Add yytext; YYBEGIN COMMENT; commentLevel := 1;
- continue() before YYBEGIN INITIAL);
+ continue(); YYBEGIN INITIAL; continue());
<A>"(*" => (YYBEGIN EMPTYCOMMENT; commentLevel := 1; continue());
<CODE>"(*" => (Add yytext; YYBEGIN COMMENT; commentLevel := 1;
- continue() before YYBEGIN CODE);
-<INITIAL>[^%\n]+ => (Add yytext; continue());
+ continue(); YYBEGIN CODE; continue());
+<INITIAL>[^(%\n]+ => (Add yytext; continue());
<INITIAL>"%%" => (YYBEGIN A; HEADER (concat (rev (!text)),!lineno,!lineno));
<INITIAL,CODE,COMMENT,F,EMPTYCOMMENT>\n => (Add yytext; inc lineno; continue());
<INITIAL>. => (Add yytext; continue());

[Matthew Fluet]

Thanks Michael. I committed your patch.

I guess it says something that no one before ever felt compelled to
comment their semantic actions. I'm just not sure what the something
is...

The SML/NJ version of the tool will still have this bug; you might
push the patch to them as well.

On Wed, Jan 12, 2011 at 12:26 AM, Michael Norrish

Post by Michael Norrish
Fix bug in comment-handling in lexer for mlyacc's input language.
In particular, if a comment was successfully lexed, it returned a BOGUS_VALUE
token, which then messed up the higher-level parsing. ?This was due to
the use of the idiom
?continue() before YYBEGIN SOMESTATE
making the return value of the continue the return value for the lexer at
that point.
This bug prevented the use of comments in code sections entirely.
It was somewhat masked in the prefix section by the fact that most
supposed comments there were never lexed as such anyway. ?This could
be observed by a file that included %% in a comment in the prefix
section.
diff --git a/mlyacc/src/yacc.lex b/mlyacc/src/yacc.lex
--- a/mlyacc/src/yacc.lex
+++ b/mlyacc/src/yacc.lex
@@ -71,11 +71,11 @@
?qualid ={id}".";
?%%
?<INITIAL>"(*" ?=> (Add yytext; YYBEGIN COMMENT; commentLevel := 1;
- ? ? ? ? ? ? ? ? ? continue() before YYBEGIN INITIAL);
+ ? ? ? ? ? ? ? ? ? continue(); YYBEGIN INITIAL; continue());
?<A>"(*" ? ? ? ? ? ? ? ?=> (YYBEGIN EMPTYCOMMENT; commentLevel := 1;
continue());
?<CODE>"(*" ? ? => (Add yytext; YYBEGIN COMMENT; commentLevel := 1;
- ? ? ? ? ? ? ? ? ? continue() before YYBEGIN CODE);
-<INITIAL>[^%\n]+ => (Add yytext; continue());
+ ? ? ? ? ? ? ? ? ? continue(); YYBEGIN CODE; continue());
+<INITIAL>[^(%\n]+ => (Add yytext; continue());
?<INITIAL>"%%" ? => (YYBEGIN A; HEADER (concat (rev
(!text)),!lineno,!lineno));
?<INITIAL,CODE,COMMENT,F,EMPTYCOMMENT>\n ?=> (Add yytext; inc lineno;
continue());
?<INITIAL>. ? ? ?=> (Add yytext; continue());
_______________________________________________
MLton mailing list
http://mlton.org/mailman/listinfo/mlton