During the parsing of doc comments, backtracking is off on purpose, as it won't consume syntactic contents, merely whitespaces (while doc comment is a special form of block comment), the desired result is `Nothing` or `Just docCmt` without backtracking of whitespaces.

On 2020-10-25, at 18:53, Jaro Reinders wrote:

I think you have to insert a 'try' manually inside the optional, because megaparsec doesn't do backtracking by default.

On 10/25/20 11:27 AM, YueCompl via Haskell-Cafe wrote:

...

Dear Cafe, I find myself unable to reason about an error that `optional` in a parser err out instead of return Nothing, I asked the question at StackOverflow, and would like to seek your help here as well. https://stackoverflow.com/questions/64522568/why-optional-in-a-parser-can-er... https://stackoverflow.com/questions/64522568/why-optional-in-a-parser-can-er... https://github.com/complyue/dcp https://github.com/complyue/dcp is a minimum working example to reprod this error $ cabal run dcp:dcp < samples/basic.txt Up to date dcp: 10:1: | 10 | method doXXX() pass | ^ unexpected 'm' expecting ';' CallStack (from HasCallStack): error, called at src/Parser.hs:149:14 in main:Parser $ I believe it's optionalSemicolon causing the failure: https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... findIt = do -- ignore leading whitespaces and an optional semicolon in between nbsc >> optionalSemicolon >> nbsc -- try get a doc comment block getIt >>= \case And it's defined like this: https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... optionalSemicolon :: Parser Bool optionalSemicolon = fromMaybe False <$> optional (True <$ symbol ";") I can't reason about why it can fail like this. Best regards, Compl _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

---

Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

Unlike `sc`, `nbsc` won't consume block comments, as it's defined as `L.space space1 (L.skipLineComment "#") empty`

On 2020-10-25, at 19:07, Jaro Reinders wrote:

But in the basic.txt example, won't the first nbsc consume everything up to 'method doXXX() pass'. Then the semicolon parser consumes the 'm' character, and that will never backtrack. So that 'm' character is lost.

On 10/25/20 12:00 PM, YueCompl wrote:

...

During the parsing of doc comments, backtracking is off on purpose, as it won't consume syntactic contents, merely whitespaces (while doc comment is a special form of block comment), the desired result is `Nothing` or `Just docCmt` without backtracking of whitespaces.

...

On 2020-10-25, at 18:53, Jaro Reinders wrote:

I think you have to insert a 'try' manually inside the optional, because megaparsec doesn't do backtracking by default.

On 10/25/20 11:27 AM, YueCompl via Haskell-Cafe wrote:

...

Dear Cafe, I find myself unable to reason about an error that `optional` in a parser err out instead of return Nothing, I asked the question at StackOverflow, and would like to seek your help here as well. https://stackoverflow.com/questions/64522568/why-optional-in-a-parser-can-er... https://stackoverflow.com/questions/64522568/why-optional-in-a-parser-can-er... https://github.com/complyue/dcp https://github.com/complyue/dcp is a minimum working example to reprod this error $ cabal run dcp:dcp < samples/basic.txt Up to date dcp: 10:1: | 10 | method doXXX() pass | ^ unexpected 'm' expecting ';' CallStack (from HasCallStack): error, called at src/Parser.hs:149:14 in main:Parser $ I believe it's optionalSemicolon causing the failure: https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... findIt = do -- ignore leading whitespaces and an optional semicolon in between nbsc >> optionalSemicolon >> nbsc -- try get a doc comment block getIt >>= \case And it's defined like this: https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... optionalSemicolon :: Parser Bool optionalSemicolon = fromMaybe False <$> optional (True <$ symbol ";") I can't reason about why it can fail like this. Best regards, Compl _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

---

Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

The rerun is guarded behind an optional `L.skipBlockCommentNested "{#" "#}"` got a `Just{}` result, and only after `getIt` got a `Nothing` result.

On 2020-10-25, at 19:16, Jaro Reinders wrote:

But the 'findIt' parser will be rerun after every block comment right?

On 10/25/20 12:10 PM, YueCompl wrote:

...

Unlike `sc`, `nbsc` won't consume block comments, as it's defined as `L.space space1 (L.skipLineComment "#") empty`

...

On 2020-10-25, at 19:07, Jaro Reinders wrote:

But in the basic.txt example, won't the first nbsc consume everything up to 'method doXXX() pass'. Then the semicolon parser consumes the 'm' character, and that will never backtrack. So that 'm' character is lost.

On 10/25/20 12:00 PM, YueCompl wrote:

...

During the parsing of doc comments, backtracking is off on purpose, as it won't consume syntactic contents, merely whitespaces (while doc comment is a special form of block comment), the desired result is `Nothing` or `Just docCmt` without backtracking of whitespaces.

...

On 2020-10-25, at 18:53, Jaro Reinders wrote:

I think you have to insert a 'try' manually inside the optional, because megaparsec doesn't do backtracking by default.

On 10/25/20 11:27 AM, YueCompl via Haskell-Cafe wrote:

...

Dear Cafe, I find myself unable to reason about an error that `optional` in a parser err out instead of return Nothing, I asked the question at StackOverflow, and would like to seek your help here as well. https://stackoverflow.com/questions/64522568/why-optional-in-a-parser-can-er... https://stackoverflow.com/questions/64522568/why-optional-in-a-parser-can-er... https://github.com/complyue/dcp https://github.com/complyue/dcp is a minimum working example to reprod this error $ cabal run dcp:dcp < samples/basic.txt Up to date dcp: 10:1: | 10 | method doXXX() pass | ^ unexpected 'm' expecting ';' CallStack (from HasCallStack): error, called at src/Parser.hs:149:14 in main:Parser $ I believe it's optionalSemicolon causing the failure: https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... findIt = do -- ignore leading whitespaces and an optional semicolon in between nbsc >> optionalSemicolon >> nbsc -- try get a doc comment block getIt >>= \case And it's defined like this: https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... optionalSemicolon :: Parser Bool optionalSemicolon = fromMaybe False <$> optional (True <$ symbol ";") I can't reason about why it can fail like this. Best regards, Compl _______________________________________________ Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

---

Haskell-Cafe mailing list To (un)subscribe, modify options or view archives go to: http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe Only members subscribed via the mailman list are allowed to post.

Tried this: ```hs findIt = do -- ignore leading whitespaces and an optional semicolon in between nbsc ii0 <- getInput trace ("**about*to*match*semicolon*[" <> T.unpack ii0 <> "]") $ pure () optionalSemicolon trace ("**after*semicolon*attempt**") $ pure () nbsc -- try get a doc comment block getIt >>= \case ``` ```console $ cabal run dcp < samples/basic.txt Up to date **about*to*match*semicolon*[{## # this is the module's doc comment #} # this is regular line comment to be dropped in parsing {## # this is the doc comment for `doXXX()` #} method doXXX() pass {# # this is regular block comment to be dropped in parsing #} {## # this is the doc comment for `doYYY()` #} method doYYY() pass ] dcp: 10:1: | 10 | method doXXX() pass | ^ unexpected 'm' expecting ';' CallStack (from HasCallStack): error, called at src/Parser.hs:154:14 in main:Parser ``` So it seems the very first optionalSemicolon failed without consuming anything ... And actually I totally have no clue how the content before `method doXXX` is consumed.

On 2020-10-25, at 19:26, Jaro Reinders wrote:

I think the optionalSemicolon will consume the '{' of the first comment and then the 'getIt' will produce Nothing because the input doesn't start with '{##' anymore, and then it will go on like that until the optionalSemicolon consumes that 'm'.

On 10/25/20 12:21 PM, YueCompl wrote:

...

The rerun is guarded behind an optional `L.skipBlockCommentNested "{#" "#}"` got a `Just{}` result, and only after `getIt` got a `Nothing` result.

...

On 2020-10-25, at 19:16, Jaro Reinders wrote:

But the 'findIt' parser will be rerun after every block comment right?

On 10/25/20 12:10 PM, YueCompl wrote:

...

Unlike `sc`, `nbsc` won't consume block comments, as it's defined as `L.space space1 (L.skipLineComment "#") empty`

...

On 2020-10-25, at 19:07, Jaro Reinders wrote:

But in the basic.txt example, won't the first nbsc consume everything up to 'method doXXX() pass'. Then the semicolon parser consumes the 'm' character, and that will never backtrack. So that 'm' character is lost.

On 10/25/20 12:00 PM, YueCompl wrote:

...

During the parsing of doc comments, backtracking is off on purpose, as it won't consume syntactic contents, merely whitespaces (while doc comment is a special form of block comment), the desired result is `Nothing` or `Just docCmt` without backtracking of whitespaces. > On 2020-10-25, at 18:53, Jaro Reinders wrote: > > I think you have to insert a 'try' manually inside the optional, because megaparsec doesn't do backtracking by default. > > On 10/25/20 11:27 AM, YueCompl via Haskell-Cafe wrote: >> Dear Cafe, >> I find myself unable to reason about an error that `optional` in a parser err out instead of return Nothing, I asked the question at StackOverflow, and would like to seek your help here as well. >> https://stackoverflow.com/questions/64522568/why-optional-in-a-parser-can-er... https://stackoverflow.com/questions/64522568/why-optional-in-a-parser-can-er... >> https://github.com/complyue/dcp https://github.com/complyue/dcp is a minimum working example to reprod this error >> $ cabal run dcp:dcp < samples/basic.txt >> Up to date >> dcp: 10:1: >> | >> 10 | method doXXX() pass >> | ^ >> unexpected 'm' >> expecting ';' >> CallStack (from HasCallStack): >> error, called at src/Parser.hs:149:14 in main:Parser >> $ >> I believe it's optionalSemicolon causing the failure: >> https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... >> findIt = do >> -- ignore leading whitespaces and an optional semicolon in between >> nbsc >> optionalSemicolon >> nbsc >> -- try get a doc comment block >> getIt >>= \case >> And it's defined like this: https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... >> optionalSemicolon :: Parser Bool >> optionalSemicolon = fromMaybe False <$> optional (True <$ symbol ";") >> I can't reason about why it can fail like this. >> Best regards, >> Compl >> _______________________________________________ >> Haskell-Cafe mailing list >> To (un)subscribe, modify options or view archives go to: >> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe >> Only members subscribed via the mailman list are allowed to post. > _______________________________________________ > Haskell-Cafe mailing list > To (un)subscribe, modify options or view archives go to: > http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe > Only members subscribed via the mailman list are allowed to post.

Yeah, you are so right! It's working now: ```console $ cabal run dcp < samples/basic.txt Up to date * Module parsed as: (Just ["this is the module's doc comment"],[(Just ["this is the doc comment for `doXXX()`"],"method doXXX() pass\n\n"),(Just ["this is the doc comment for `doYYY()`"],"method doYYY() pass\n")]) ``` https://github.com/complyue/dcp/commit/fd2df02f7218e59db2a732d5de74acedfefef... https://github.com/complyue/dcp/commit/fd2df02f7218e59db2a732d5de74acedfefef... ```diff optionalComma :: Parser Bool -optionalComma = fromMaybe False <$> optional (True <$ symbol ",") +optionalComma = fromMaybe False <$> optional (True <$ string ",") optionalSemicolon :: Parser Bool -optionalSemicolon = fromMaybe False <$> optional (True <$ symbol ";") +optionalSemicolon = fromMaybe False <$> optional (True <$ string ";") ``` Thanks!

On 2020-10-25, at 19:42, Jaro Reinders wrote:

Aha, I think it is the 'lexeme' function that causes these problems. 'symbol = lexeme . string', but lexeme first skips over all the comments (even the multiline comments).

On 10/25/20 12:36 PM, YueCompl wrote:

...

Tried this: ```hs findIt = do -- ignore leading whitespaces and an optional semicolon in between nbsc ii0 <- getInput trace ("**about*to*match*semicolon*[" <> T.unpack ii0 <> "]") $ pure () optionalSemicolon trace ("**after*semicolon*attempt**") $ pure () nbsc -- try get a doc comment block getIt >>= \case ``` ```console $ cabal run dcp < samples/basic.txt Up to date **about*to*match*semicolon*[{## # this is the module's doc comment #} # this is regular line comment to be dropped in parsing {## # this is the doc comment for `doXXX()` #} method doXXX() pass {# # this is regular block comment to be dropped in parsing #} {## # this is the doc comment for `doYYY()` #} method doYYY() pass ] dcp: 10:1: | 10 | method doXXX() pass | ^ unexpected 'm' expecting ';' CallStack (from HasCallStack): error, called at src/Parser.hs:154:14 in main:Parser ``` So it seems the very first optionalSemicolon failed without consuming anything ... And actually I totally have no clue how the content before `method doXXX` is consumed.

...

On 2020-10-25, at 19:26, Jaro Reinders wrote:

I think the optionalSemicolon will consume the '{' of the first comment and then the 'getIt' will produce Nothing because the input doesn't start with '{##' anymore, and then it will go on like that until the optionalSemicolon consumes that 'm'.

On 10/25/20 12:21 PM, YueCompl wrote:

...

The rerun is guarded behind an optional `L.skipBlockCommentNested "{#" "#}"` got a `Just{}` result, and only after `getIt` got a `Nothing` result.

...

On 2020-10-25, at 19:16, Jaro Reinders wrote:

But the 'findIt' parser will be rerun after every block comment right?

On 10/25/20 12:10 PM, YueCompl wrote:

...

Unlike `sc`, `nbsc` won't consume block comments, as it's defined as `L.space space1 (L.skipLineComment "#") empty` > On 2020-10-25, at 19:07, Jaro Reinders wrote: > > But in the basic.txt example, won't the first nbsc consume everything up to 'method doXXX() pass'. Then the semicolon parser consumes the 'm' character, and that will never backtrack. So that 'm' character is lost. > > On 10/25/20 12:00 PM, YueCompl wrote: >> During the parsing of doc comments, backtracking is off on purpose, as it won't consume syntactic contents, merely whitespaces (while doc comment is a special form of block comment), the desired result is `Nothing` or `Just docCmt` without backtracking of whitespaces. >>> On 2020-10-25, at 18:53, Jaro Reinders wrote: >>> >>> I think you have to insert a 'try' manually inside the optional, because megaparsec doesn't do backtracking by default. >>> >>> On 10/25/20 11:27 AM, YueCompl via Haskell-Cafe wrote: >>>> Dear Cafe, >>>> I find myself unable to reason about an error that `optional` in a parser err out instead of return Nothing, I asked the question at StackOverflow, and would like to seek your help here as well. >>>> https://stackoverflow.com/questions/64522568/why-optional-in-a-parser-can-er... https://stackoverflow.com/questions/64522568/why-optional-in-a-parser-can-er... >>>> https://github.com/complyue/dcp https://github.com/complyue/dcp is a minimum working example to reprod this error >>>> $ cabal run dcp:dcp < samples/basic.txt >>>> Up to date >>>> dcp: 10:1: >>>> | >>>> 10 | method doXXX() pass >>>> | ^ >>>> unexpected 'm' >>>> expecting ';' >>>> CallStack (from HasCallStack): >>>> error, called at src/Parser.hs:149:14 in main:Parser >>>> $ >>>> I believe it's optionalSemicolon causing the failure: >>>> https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... >>>> findIt = do >>>> -- ignore leading whitespaces and an optional semicolon in between >>>> nbsc >> optionalSemicolon >> nbsc >>>> -- try get a doc comment block >>>> getIt >>= \case >>>> And it's defined like this: https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... https://github.com/complyue/dcp/blob/1df7ad590d78d4fa9a017eb53f9f265e291bdfa... >>>> optionalSemicolon :: Parser Bool >>>> optionalSemicolon = fromMaybe False <$> optional (True <$ symbol ";") >>>> I can't reason about why it can fail like this. >>>> Best regards, >>>> Compl >>>> _______________________________________________ >>>> Haskell-Cafe mailing list >>>> To (un)subscribe, modify options or view archives go to: >>>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe >>>> Only members subscribed via the mailman list are allowed to post. >>> _______________________________________________ >>> Haskell-Cafe mailing list >>> To (un)subscribe, modify options or view archives go to: >>> http://mail.haskell.org/cgi-bin/mailman/listinfo/haskell-cafe >>> Only members subscribed via the mailman list are allowed to post.