Skip to content

"Back"/"Undo" don't undo syntax extensions #12575

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
jfehrle opened this issue Jun 23, 2020 · 32 comments · Fixed by #17069
Closed

"Back"/"Undo" don't undo syntax extensions #12575

jfehrle opened this issue Jun 23, 2020 · 32 comments · Fixed by #17069
Labels
kind: anomaly An uncaught exception has been raised. kind: bug An error, flaw, fault or unintended behaviour. part: parser The parser (also called gramlib, forked from camlp5)
Milestone

Comments

@jfehrle
Copy link
Member

jfehrle commented Jun 23, 2020

If you load a plugin such as, say, Ltac2 and then backtrack in CoqIDE, the syntax extensions from the plugin are not undone. This causes unexpected and confusing errors when you try to redo those lines.

I ran into this while experimenting with Ltac2 Notation. Here's an example in 8.12 from @tchajed:

(* evaluate these two lines, then delete the first and run the second *)
From Ltac2 Require Ltac2.
Ltac2 Eval ().
(* Error: Anomaly "Uncaught exception Not_found."
Please report at http://coq.inria.fr/bugs/. *)

If you do the Ltac2 Eval () in a fresh buffer, you get Syntax error: illegal begin of vernac.

From @ppedrot #11549 (comment):

vernacular extensions should be backtracked upon by plugins. I wonder why nobody ever complained about that so far.

UPDATE: Backtracking seems to work correctly for both Notation and Ltac2 Notation, so apparently some things are being backtracked.

@jfehrle jfehrle added the kind: bug An error, flaw, fault or unintended behaviour. label Jun 23, 2020
@tchajed
Copy link
Contributor

tchajed commented Jun 23, 2020

Excellent, thanks for putting this into an issue @jfehrle!

@jfehrle
Copy link
Member Author

jfehrle commented Jun 27, 2020

Stack trace, partly annotated, using 8.12 beta. Not sure it tells us much.

Anomaly "Uncaught exception Not_found."
Please report at http://coq.inria.fr/bugs/.
Raised at file "clib/int.ml", line 39, characters 14-29
Called from file "clib/hMap.ml", line 361, characters 12-28
Called from file "user-contrib/Ltac2/tac2env.ml", line 220, characters 11-41  shortest_qualid_of_type
Called from file "user-contrib/Ltac2/tac2print.ml", line 68, characters 23-63
Called from file "user-contrib/Ltac2/tac2print.ml", line 76, characters 8-26
Called from file "user-contrib/Ltac2/tac2print.ml" (inlined), line 78, characters 22-46
Called from file "user-contrib/Ltac2/tac2entries.ml", line 823, characters 37-61  perform_eval
Called from file "vernac/vernacinterp.ml", line 45, characters 4-13  interp_typed_vernac VTReadProofOpt
Called from file "vernac/vernacinterp.ml", line 210, characters 20-60  interp_control
Called from file "lib/flags.ml", line 17, characters 14-17
Re-raised at file "clib/exninfo.ml", line 82, characters 4-38
Called from file "vernac/vernacinterp.ml", line 253, characters 18-43
Called from file "vernac/vernacinterp.ml", line 251, characters 6-279
Re-raised at file "clib/exninfo.ml", line 82, characters 4-38
Called from file "stm/stm.ml", line 2426, characters 16-43
Called from file "stm/stm.ml", line 1006, characters 6-10
Re-raised at file "clib/exninfo.ml", line 82, characters 4-38
Called from file "stm/stm.ml", line 2567, characters 4-105
Called from file "stm/stm.ml", line 2748, characters 4-60
Re-raised at file "clib/exninfo.ml", line 82, characters 4-38
Called from file "stm/stm.ml", line 2759, characters 12-50
Called from file "ide/idetop.ml", line 233, characters 13-28
Called from file "ide/idetop.ml", line 445, characters 12-15
Called from file "ide/protocol/xmlprotocol.ml", line 665, characters 29-46

@JasonGross
Copy link
Member

Interestingly, I believe this used to be an issue with Reserved Notation. That was fixed, and sometime later, custom entries were introduced, and I think this issue popped up when backtracking across custom entry creation. I guess it was never discovered for plugin syntax extensions because, before Ltac2, plugins mostly didn't extend the syntax in ways that overlapped with existing syntax (though I wonder if you can hit a similar issue by backtracking across loading of ssreflect?), and in any case backtracking across the plugin that extended syntax was probably a relatively rare occurrence.

@jfehrle
Copy link
Member Author

jfehrle commented Jun 27, 2020

The CoqIDE behavior is confusing. If I CTRL-up to the beginning of the file after the first try of executing the Ltac2 Eval, CoqIDE spawns a new worker process for the buffer, which won't ever have loaded Ltac2, so there's no anomaly and it correctly gives a syntax error for the Ltac2 Eval.

If I instead move the cursor to the beginning of the buffer and comment out the From ... Require, CoqIDE keeps the same worker process, so you get the anomaly.

The command entry for Print Grammar vernac is consistent with whether you get or don't get an anomaly.

I see the same problem with the Derive plugin with this input. So not Ltac2 specific:

(* evaluate these two lines, then delete the first and run the second *)
Require Derive.
Derive X SuchThat True As L.

@JasonGross Any ideas on what to look at next or where the Reserved Notation fix is? You think the fix for this would be in Stm or something the Require command or general command code?

@Zimmi48
Copy link
Member

Zimmi48 commented Jun 29, 2020

cc @coq/parsing-maintainers

@jfehrle jfehrle added the part: parser The parser (also called gramlib, forked from camlp5) label Jun 29, 2020
@jfehrle
Copy link
Member Author

jfehrle commented Jul 8, 2020

@gares or @maximedenes:

I verified with a printf that Stm.process_transaction doesn't call VCS.set_parsing_state for a From ... Require command. Shouldn't it be doing that since the command can modify the parser? If that seems right and you give me a little guidance, I'll prepare a PR. (Of course I would not object if one of you submitted a fix yourself.) Or if this is not the issue, perhaps you can suggest where to look next.

FWIW, another example that probably has the same root cause. The following gets a syntax error on the Ltac2 Notation if the 2 lines are uncommented (and works correctly otherwise).

From Ltac2 Require Import Ltac2.
(*Back.
From Ltac2 Require Import Ltac2.*)
From Ltac2 Require Import Message.

Ltac2 Notation "ex" := print (of_string "x").
Goal True.
ex.

@gares
Copy link
Member

gares commented Jul 9, 2020

IIRC set parsing state is called after each step, and the code seems to do that. This should be the line reached on commands such as Require: https://github.com/coq/coq/blob/master/stm/stm.ml#L2999 (since VernacRequire is classified as a Sideff with VtNow https://github.com/coq/coq/blob/master/stm/vernac_classifier.ml#L182).

I don't know if this is problem, but ML code cannot be unloaded and I'm pretty sure Ltac2 defines some parsing rules in ML (it must really). I'm pretty sure the "undoing" a regular notation works.

@herbelin
Copy link
Member

herbelin commented Jul 9, 2020

With gramlib in the archive, its state could be registered to the global summary and I suspect this would solve the problem.

@jfehrle
Copy link
Member Author

jfehrle commented Jul 9, 2020

@gares That line is not executed for From ... Require.

                (* If execution has not been put in cache, we need to save the parsing state *)
                if (VCS.get_info id).state == EmptyState then begin
                try let _ = Sys.getenv("STM_INFO") in
                  Printf.printf "Call set_parsing_state at %d\n%!" __LINE__
                with Not_found -> ();
                VCS.set_parsing_state id parsing;
                end;
$ bin/coqtop
Welcome to Coq LAPTOP-LB5T3R83:/mnt/c/proj/coq,proof_state_info (0255d9592ba40d02c1c1ab5a27ebb1de5166ed9c)

Coq < From Ltac2 Require Import Ltac2.
[Loading ML file ltac2_plugin.cmxs ... done]

Coq < From Ltac2 Require Import Message.

Coq < Ltac2 Notation "ex" := print (of_string "x").
Goal True.
ex.
Coq < 1 subgoal

  ============================
  True

Unnamed_thm <
x

Unnamed_thm < Quit.
print stats
vcs state size = 946646 backup1 size = 946646 backup2 size = 28
jfehrle@LAPTOP-LB5T3R83:/mnt/c/proj/coq$ echo $STM_INFO
1

I don't know if this is problem, but ML code cannot be unloaded

I don't see how that would be a problem for this use case.

I'm pretty sure the "undoing" a regular notation works.

Yes, I tried undoing both regular notations and Ltac2 notations and it seemed to work correctly. The Ltac2 code path looks a bit different from regular notations but I didn't investigate since it seems to work.

@ejgallego
Copy link
Member

ejgallego commented Jul 10, 2020

With gramlib in the archive, its state could be registered to the global summary and I suspect this would solve the problem.

@herbelin even so that's a bit bizarre that pcoq already is not capturing this; the vernacular extension point already pushed to the internal undo/redo table.

Maybe we could have a look at it in person some day?

@herbelin
Copy link
Member

@ejgallego: you're right, these syntax extensions goes through Pcoq.grammar_extend which registers how to undo them when backtracking (even if this undoing is ad hoc, using safe_delete_rule rather than restoring a state as done in other places). So, why is this undoing not working?

herbelin added a commit to herbelin/github-coq that referenced this issue Nov 14, 2020
…ktracking mechanism.

In particular, we remove the "ByEXTEND" mechanism.
@herbelin
Copy link
Member

For reproducibility in coqtop, do:

From Ltac2 Require Ltac2.
Back 1.
Ltac2 Eval ().

The point is that GRAMMAR EXTEND rules are not (any more?) synchronized. According to pcoq.mli (comment on top of grammar_extend) this is on purpose. If I plug GRAMMAR EXTEND to the synchronization mechanism, this fixes the bug (see #13382). Not sure about what is the "right" thing to do.

@ejgallego
Copy link
Member

See also #5234

@MSoegtropIMC
Copy link
Contributor

@jfehrle : this appears to be fixed in CoqIDE 8.15, but not in VsCoq. Do you remember how this has been fixed?

@jfehrle
Copy link
Member Author

jfehrle commented Mar 8, 2022

@MSoegtropIMC You mean the anomaly is gone or that this is completely fixed? I haven't made any code changes to address this. I think the fix would be entirely on the server side, so I don't see why VsCoq would behave any differently. Note my comments above:

If I CTRL-up to the beginning of the file after the first try of executing the Ltac2 Eval, CoqIDE spawns a new worker process for the buffer, which won't ever have loaded Ltac2, so there's no anomaly and it correctly gives a syntax error for the Ltac2 Eval.

If I instead move the cursor to the beginning of the buffer and comment out the From ... Require, CoqIDE keeps the same worker process, so you get the anomaly.

Perhaps you're spawning a new process on CoqIDE but not on VsCoq?

@MSoegtropIMC
Copy link
Contributor

Actually my older issue (#11793) is fixed, but this one appears to be not fixed.

@MSoegtropIMC
Copy link
Contributor

More specifically what is fixed that stepping hence and forth over valid code including the Ltac2 require did not work in the past, but now it does. I find this more important than the error messages one gets when stepping over code which got invalid cause the Ltac2 require was removed.

@ejgallego
Copy link
Member

I am positive we still have problems handling the state in the parsing engine, this should be IMHO a top priority in terms of engineering power.

@SkySkimmer
Copy link
Contributor

SkySkimmer commented Jan 3, 2023

A very similar issue is still present in CoqIDE as well. To reproduce on the following code:

Can't reproduce since 89ad33f
Original example (with the Not_found) still reproduces in master

@ejgallego
Copy link
Member

@SkySkimmer there is an easy reproduction case:

  • do Require Import ssreflect, execute.
  • then remove that, try some code that uses unfold foo in |- * (like Rtrigo1.v`

Coq will fail to parse it. Can be tested very easily with coq-lsp for example.

@SkySkimmer
Copy link
Contributor

Require Import ssreflect.

Goal True.
  unfold idProp in |- *.

parses for me with whatever combination of executing then backtracking that I tried.

@ejgallego
Copy link
Member

ejgallego commented Jan 3, 2023

Actually this seems quite tricky to reproduce / minimize indeed, let me research a bit more, as it seems the bug only happens with a very particular sequence of plugin loading / freeze calls.

I see the bug always in coq-lsp when I open examples/ex0.v, then I open examples/Rtrigo1.v

@ejgallego
Copy link
Member

The problem is pretty hard to debug due to all the hack in ssr, but here is a diff of Print Grammar and Print Keywords when things go good vs when they go bad:

--- r_right.txt	2023-01-03 20:11:56.617361667 +0100
+++ r_wrong.txt	2023-01-03 20:13:36.729364722 +0100
@@ -1975,7 +1975,7 @@
   | IDENT "swap"; int_or_var; int_or_var
   | IDENT "cycle"; int_or_var
   | IDENT "unshelve"; ltac_expr LEVEL "1"
-  | IDENT "transparent_abstract"; ltac_expr LEVEL "3"; "using"; ident
+  | IDENT "transparent_abstract"; ltac_expr LEVEL "3"; IDENT "using"; ident
   | IDENT "transparent_abstract"; ltac_expr LEVEL "3"
   | IDENT "destauto"; "in"; hyp
   | IDENT "destauto"
@@ -1986,9 +1986,9 @@
   | IDENT "specialize_eqs"; hyp
   | IDENT "generalize_eqs_vars"; hyp
   | IDENT "generalize_eqs"; hyp
-  | IDENT "stepr"; constr; "by"; tactic
+  | IDENT "stepr"; constr; IDENT "by"; tactic
   | IDENT "stepr"; constr
-  | IDENT "stepl"; constr; "by"; tactic
+  | IDENT "stepl"; constr; IDENT "by"; tactic
   | IDENT "stepl"; constr
   | IDENT "instantiate"; "("; ident; ":="; lconstr; ")"
   | IDENT "instantiate"; "("; natural; ":="; lconstr; ")"; hloc
@@ -1998,9 +1998,11 @@
   | IDENT "subst"
   | IDENT "notypeclasses"; IDENT "refine"; uconstr
   | IDENT "refine"; uconstr
-  | IDENT "autorewrite"; "*"; "with"; LIST1 preident; clause; "using"; tactic
+  | IDENT "autorewrite"; "*"; "with"; LIST1 preident; clause; IDENT "using";
+    tactic
   | IDENT "autorewrite"; "*"; "with"; LIST1 preident; clause
-  | IDENT "autorewrite"; "with"; LIST1 preident; clause; "using"; tactic
+  | IDENT "autorewrite"; "with"; LIST1 preident; clause; IDENT "using";
+    tactic
   | IDENT "autorewrite"; "with"; LIST1 preident; clause
   | IDENT "contradiction"; OPT constr_with_bindings
   | IDENT "decompose"; "["; LIST1 constr; "]"; constr
@@ -2689,7 +2691,6 @@
 ()
 )
 *
-**
 +
 ++
 ,
@@ -2698,12 +2699,10 @@
 .
 .(
 ..
-...
 /
 /\
 :
 ::
-::=
 ::>
 :=
 :>
@@ -2741,7 +2740,6 @@
 Type
 Variable
 [
-[=
 \/
 ]
 ^
@@ -2751,7 +2749,6 @@
 `{
 as
 at
-by
 cofix
 else
 end
@@ -2768,13 +2765,11 @@
 mod
 return
 then
-using
 where
 with
 {
 {|
 |
-|-
 ||
 }
 ~

So starting with a state with the ssrplugin loaded, then switching to a state before it was loaded, somehow the keywords and grammar rules are affected. So this bug may be specific to ssreflect actually.

@ejgallego
Copy link
Member

cc @artagnon , here is the error you've been suffering.

@ppedrot
Copy link
Member

ppedrot commented Jan 3, 2023

Hmm, isn't the fix just to wrap the keyword declaration through CLexer.set_keyword_state in ssrvernac.mlg and g_ssrmatching.mlg into a dummy libobject using Mltop.declare_cache_obj? See e.g. the corresponding call in tac2entries.ml.

@ejgallego
Copy link
Member

Thanks @ppedrot , I hope that works, the bug is very subtle and is giving me a headache (the dance done with the lexer and ssr_is_loaded is not very, ehem, pleasant)

I think the only way we can hope to have a reasonably maintainable setup is to do require writers to parsing and lexing state to grab a handle so they can update it functionally.

@SkySkimmer
Copy link
Contributor

This may be related to calling redo in this remove_grammar code https://github.com/coq/coq/blob/6424bbc81e03447198784cf3dc56934a45145fdf/parsing/pcoq.ml#L169

@ejgallego
Copy link
Member

See also #10146

@ejgallego
Copy link
Member

This may be related to calling redo in this remove_grammar code

https://github.com/coq/coq/blob/6424bbc81e03447198784cf3dc56934a45145fdf/parsing/pcoq.ml#L169

I think you may be into something @SkySkimmer , thanks; I can reproduce this better, and without involving ssreflect (still not a simple repro case yet).

So indeed, that happens with regular Coq plugins so seems like a core issue.

@SkySkimmer
Copy link
Contributor

Definitely related as if I remove the redo code and test

Reserved Notation "& $ k" (at level 300).

Require Import Ltac2.Ltac2.

Ltac2 Eval ().

run everything, backtrack to before the Reserved Notation, run just Ltac2 Eval
then I get the expected illegal begin of toplevel:vernac_toplevel. (with master: uncaught Not_found)
(the Reserved Notation is needed because otherwise remove_grammar stops too early)

Turning this info into a fix is harder though.

SkySkimmer added a commit to SkySkimmer/rocq that referenced this issue Jan 5, 2023
ie to fix rocq-prover#12575

This is in a hacky state currently, mostly because coqpp is not adapted (so
the g_ltac2.ml file was manually modified for the proof of concept)
and because GRAMMAR EXTEND is not properly treated but was just hacked
up (but the same method should be applicable).

This was hand tested using
~~~coq
Require Import Ltac2.Ltac2.

Ltac2 Eval ().
~~~
we must correctly handle
- eval all, backtrack before the Require, eval just Ltac2 Eval: should
  parsing error illegal begin etc
- eval all, backtrack before the Require, eval all again: should
  work (ie the parsing rules must be re-applied)

g_ltac2.ml was generated by unmodified coqpp then manually edited in 2
ways:
- vernac_extend calls have an added `~plugin:__coq_plugin_name` argument
- the code generated from
~~~
GRAMMAR EXTEND Gram
  GLOBAL: tac2mode;
  tac2mode: TOP
    [ [ tac = G_vernac.query_command -> { tac None } ] ];
END
~~~
  was wrapped in Mltop.add_init_function and uses `Pcoq.grammar_extend ~redo:false`
  (without this it complains about mssing levels because it executes
  before the vernac extensions)
  This is just a hack and grammar_extend should not have the redo
  argument once this is in a mergeable state.

The interesting part is what happens in vernac_extend and the
associated egramml functions:
- an indirection is added from the command name (Foo in
  `VERNAC COMMAND EXTEND Foo`) to the extension data (which contains a
  closure: not marshallable).
  This is non-backtracking, and there is no use to making it backtrack
  since we can't unlink plugins.
- vernac_extend calls `Egramml.extend_vernac_command` with `undoable:false`
  when there is no `plugin` argument, this is the old non
  backtrackable behaviour. Typically for non-plugin callers like
  g_obligations.
  Eventually we will clean up so that rules added this way produce an
  error when backtracked over instead of having the redo code in pcoq,
  until then it's also available for plugin hacks which don't mind the
  backtrack bug I guess.
- when there is a `plugin` argument, vernac_extend wraps the call to
  `Egramml.extend_vernac_command` in `Mltop.add_init_function` and
  uses `undoable:true`. In undoable mode, egramml uses the
  Pcoq.create_grammar_command system which backtracks properly.
  Because this system puts its input in the grammar_stack which ends
  up in the summary, we need the indirection mentioned above to keep
  the input marshallable.

To summarize TODOs:
- use some similar create_command_grammar mechanism to avoid raw
  grammar_extend calls from GRAMMAR EXTEND (?, not sure I fully
  understand GRAMMAR EXTEND)
- adapt coqpp to use the new mechanism instead of manually editing
  g_ltac2
- remove hack "redo" argument of grammar_extend
SkySkimmer added a commit to SkySkimmer/rocq that referenced this issue Jan 5, 2023
ie to fix rocq-prover#12575

This is in a hacky state currently, mostly because coqpp is not adapted (so
the g_ltac2.ml file was manually modified for the proof of concept)
and because GRAMMAR EXTEND is not properly treated but was just hacked
up (but the same method should be applicable).

This was hand tested using
~~~coq
Require Import Ltac2.Ltac2.

Ltac2 Eval ().
~~~
we must correctly handle
- eval all, backtrack before the Require, eval just Ltac2 Eval: should
  parsing error illegal begin etc
- eval all, backtrack before the Require, eval all again: should
  work (ie the parsing rules must be re-applied)

g_ltac2.ml was generated by unmodified coqpp then manually edited in 2
ways:
- vernac_extend calls have an added `~plugin:__coq_plugin_name` argument
- the code generated from
~~~
GRAMMAR EXTEND Gram
  GLOBAL: tac2mode;
  tac2mode: TOP
    [ [ tac = G_vernac.query_command -> { tac None } ] ];
END
~~~
  was wrapped in Mltop.add_init_function and uses `Pcoq.grammar_extend ~redo:false`
  (without this it complains about mssing levels because it executes
  before the vernac extensions)
  This is just a hack and grammar_extend should not have the redo
  argument once this is in a mergeable state.

The interesting part is what happens in vernac_extend and the
associated egramml functions:
- an indirection is added from the command name (Foo in
  `VERNAC COMMAND EXTEND Foo`) to the extension data (which contains a
  closure: not marshallable).
  This is non-backtracking, and there is no use to making it backtrack
  since we can't unlink plugins.
- vernac_extend calls `Egramml.extend_vernac_command` with `undoable:false`
  when there is no `plugin` argument, this is the old non
  backtrackable behaviour. Typically for non-plugin callers like
  g_obligations.
  Eventually we will clean up so that rules added this way produce an
  error when backtracked over instead of having the redo code in pcoq,
  until then it's also available for plugin hacks which don't mind the
  backtrack bug I guess.
- when there is a `plugin` argument, vernac_extend wraps the call to
  `Egramml.extend_vernac_command` in `Mltop.add_init_function` and
  uses `undoable:true`. In undoable mode, egramml uses the
  Pcoq.create_grammar_command system which backtracks properly.
  Because this system puts its input in the grammar_stack which ends
  up in the summary, we need the indirection mentioned above to keep
  the input marshallable.

To summarize TODOs:
- use some similar create_command_grammar mechanism to avoid raw
  grammar_extend calls from GRAMMAR EXTEND (?, not sure I fully
  understand GRAMMAR EXTEND)
- adapt coqpp to use the new mechanism instead of manually editing
  g_ltac2
- remove hack "redo" argument of grammar_extend
@SkySkimmer
Copy link
Contributor

#17069

SkySkimmer added a commit to SkySkimmer/rocq that referenced this issue Jan 6, 2023
Fix rocq-prover#12575

This was hand tested using
~~~coq
Require Import Ltac2.Ltac2.

Ltac2 Eval ().
~~~
we must correctly handle
- eval all, backtrack before the Require, eval just Ltac2 Eval: should
  parsing error illegal begin etc
- eval all, backtrack before the Require, eval all again: should
  work (ie the parsing rules must be re-applied)

The interesting part is what happens in vernac_extend and the
associated egramml functions:
- an indirection is added from the command name (Foo in
  `VERNAC COMMAND EXTEND Foo`) to the extension data (which contains a
  closure: not marshallable).
  This is non-backtracking, and there is no use to making it backtrack
  since we can't unlink plugins.
- vernac_extend calls `Egramml.extend_vernac_command` with `undoable:false`
  when there is no `plugin` argument, this is the old non
  backtrackable behaviour. Typically for non-plugin callers like
  g_obligations.
  Eventually we will clean up so that rules added this way produce an
  error when backtracked over instead of having the redo code in pcoq,
  until then it's also available for plugin hacks which don't mind the
  backtrack bug I guess.
- when there is a `plugin` argument, vernac_extend wraps the call to
  `Egramml.extend_vernac_command` in `Mltop.add_init_function` and
  uses `undoable:true`. In undoable mode, egramml uses the
  Pcoq.create_grammar_command system which backtracks properly.
  Because this system puts its input in the grammar_stack which ends
  up in the summary, we need the indirection mentioned above to keep
  the input marshallable.

For GRAMMAR EXTEND there is no user-given name so we produce a uid
from DECLARE PLUGIN, the file name and the number of occurences of
GRAMMAR EXTEND in the file (we end up with eg
`coq.plugins.ltac:coretactics.mlg:42`).

In the future we will probably try to replace the redo code of
ByEXTEND with an error and make the users in plugins switch to the new
mechanism.
For now it is too much work, when a plugin is modifying its own
entries it doesn't matter if backtrack is undone.
SkySkimmer added a commit to SkySkimmer/rocq that referenced this issue Jan 18, 2023
Fix rocq-prover#12575

This was hand tested using
~~~coq
Require Import Ltac2.Ltac2.

Ltac2 Eval ().
~~~
we must correctly handle
- eval all, backtrack before the Require, eval just Ltac2 Eval: should
  parsing error illegal begin etc
- eval all, backtrack before the Require, eval all again: should
  work (ie the parsing rules must be re-applied)

The interesting part is what happens in vernac_extend and the
associated egramml functions:
- an indirection is added from the command name (Foo in
  `VERNAC COMMAND EXTEND Foo`) to the extension data (which contains a
  closure: not marshallable).
  This is non-backtracking, and there is no use to making it backtrack
  since we can't unlink plugins.
- vernac_extend calls `Egramml.extend_vernac_command` with `undoable:false`
  when there is no `plugin` argument, this is the old non
  backtrackable behaviour. Typically for non-plugin callers like
  g_obligations.
  Eventually we will clean up so that rules added this way produce an
  error when backtracked over instead of having the redo code in pcoq,
  until then it's also available for plugin hacks which don't mind the
  backtrack bug I guess.
- when there is a `plugin` argument, vernac_extend wraps the call to
  `Egramml.extend_vernac_command` in `Mltop.add_init_function` and
  uses `undoable:true`. In undoable mode, egramml uses the
  Pcoq.create_grammar_command system which backtracks properly.
  Because this system puts its input in the grammar_stack which ends
  up in the summary, we need the indirection mentioned above to keep
  the input marshallable.

For GRAMMAR EXTEND there is no user-given name so we produce a uid
from DECLARE PLUGIN, the file name and the number of occurences of
GRAMMAR EXTEND in the file (we end up with eg
`coq.plugins.ltac:coretactics.mlg:42`).

In the future we will probably try to replace the redo code of
ByEXTEND with an error and make the users in plugins switch to the new
mechanism.
For now it is too much work, when a plugin is modifying its own
entries it doesn't matter if backtrack is undone.
SkySkimmer added a commit to SkySkimmer/rocq that referenced this issue Jan 20, 2023
Fix rocq-prover#12575

This was hand tested using
~~~coq
Require Import Ltac2.Ltac2.

Ltac2 Eval ().
~~~
we must correctly handle
- eval all, backtrack before the Require, eval just Ltac2 Eval: should
  parsing error illegal begin etc
- eval all, backtrack before the Require, eval all again: should
  work (ie the parsing rules must be re-applied)

The interesting part is what happens in vernac_extend and the
associated egramml functions:
- an indirection is added from the command name (Foo in
  `VERNAC COMMAND EXTEND Foo`) to the extension data (which contains a
  closure: not marshallable).
  This is non-backtracking, and there is no use to making it backtrack
  since we can't unlink plugins.
- vernac_extend calls `Egramml.extend_vernac_command` with `undoable:false`
  when there is no `plugin` argument, this is the old non
  backtrackable behaviour. Typically for non-plugin callers like
  g_obligations.
  Eventually we will clean up so that rules added this way produce an
  error when backtracked over instead of having the redo code in pcoq,
  until then it's also available for plugin hacks which don't mind the
  backtrack bug I guess.
- when there is a `plugin` argument, vernac_extend wraps the call to
  `Egramml.extend_vernac_command` in `Mltop.add_init_function` and
  uses `undoable:true`. In undoable mode, egramml uses the
  Pcoq.create_grammar_command system which backtracks properly.
  Because this system puts its input in the grammar_stack which ends
  up in the summary, we need the indirection mentioned above to keep
  the input marshallable.

For GRAMMAR EXTEND there is no user-given name so we produce a uid
from DECLARE PLUGIN, the file name and the number of occurences of
GRAMMAR EXTEND in the file (we end up with eg
`coq.plugins.ltac:coretactics.mlg:42`).

In the future we will probably try to replace the redo code of
ByEXTEND with an error and make the users in plugins switch to the new
mechanism.
For now it is too much work, when a plugin is modifying its own
entries it doesn't matter if backtrack is undone.
SkySkimmer added a commit to SkySkimmer/rocq that referenced this issue Jan 23, 2023
Fix rocq-prover#12575

This was hand tested using
~~~coq
Require Import Ltac2.Ltac2.

Ltac2 Eval ().
~~~
we must correctly handle
- eval all, backtrack before the Require, eval just Ltac2 Eval: should
  parsing error illegal begin etc
- eval all, backtrack before the Require, eval all again: should
  work (ie the parsing rules must be re-applied)

The interesting part is what happens in vernac_extend and the
associated egramml functions:
- an indirection is added from the command name (Foo in
  `VERNAC COMMAND EXTEND Foo`) to the extension data (which contains a
  closure: not marshallable).
  This is non-backtracking, and there is no use to making it backtrack
  since we can't unlink plugins.
- vernac_extend calls `Egramml.extend_vernac_command` with `undoable:false`
  when there is no `plugin` argument, this is the old non
  backtrackable behaviour. Typically for non-plugin callers like
  g_obligations.
  Eventually we will clean up so that rules added this way produce an
  error when backtracked over instead of having the redo code in pcoq,
  until then it's also available for plugin hacks which don't mind the
  backtrack bug I guess.
- when there is a `plugin` argument, vernac_extend wraps the call to
  `Egramml.extend_vernac_command` in `Mltop.add_init_function` and
  uses `undoable:true`. In undoable mode, egramml uses the
  Pcoq.create_grammar_command system which backtracks properly.
  Because this system puts its input in the grammar_stack which ends
  up in the summary, we need the indirection mentioned above to keep
  the input marshallable.

For GRAMMAR EXTEND there is no user-given name so we produce a uid
from DECLARE PLUGIN, the file name and the number of occurences of
GRAMMAR EXTEND in the file (we end up with eg
`coq.plugins.ltac:coretactics.mlg:42`).

In the future we will probably try to replace the redo code of
ByEXTEND with an error and make the users in plugins switch to the new
mechanism.
For now it is too much work, when a plugin is modifying its own
entries it doesn't matter if backtrack is undone.
@coqbot-app coqbot-app bot added this to the 8.18+rc1 milestone Jan 27, 2023
ejgallego pushed a commit to ejgallego/coq that referenced this issue Feb 2, 2023
Fix rocq-prover#12575

This was hand tested using
~~~coq
Require Import Ltac2.Ltac2.

Ltac2 Eval ().
~~~
we must correctly handle
- eval all, backtrack before the Require, eval just Ltac2 Eval: should
  parsing error illegal begin etc
- eval all, backtrack before the Require, eval all again: should
  work (ie the parsing rules must be re-applied)

The interesting part is what happens in vernac_extend and the
associated egramml functions:
- an indirection is added from the command name (Foo in
  `VERNAC COMMAND EXTEND Foo`) to the extension data (which contains a
  closure: not marshallable).
  This is non-backtracking, and there is no use to making it backtrack
  since we can't unlink plugins.
- vernac_extend calls `Egramml.extend_vernac_command` with `undoable:false`
  when there is no `plugin` argument, this is the old non
  backtrackable behaviour. Typically for non-plugin callers like
  g_obligations.
  Eventually we will clean up so that rules added this way produce an
  error when backtracked over instead of having the redo code in pcoq,
  until then it's also available for plugin hacks which don't mind the
  backtrack bug I guess.
- when there is a `plugin` argument, vernac_extend wraps the call to
  `Egramml.extend_vernac_command` in `Mltop.add_init_function` and
  uses `undoable:true`. In undoable mode, egramml uses the
  Pcoq.create_grammar_command system which backtracks properly.
  Because this system puts its input in the grammar_stack which ends
  up in the summary, we need the indirection mentioned above to keep
  the input marshallable.

For GRAMMAR EXTEND there is no user-given name so we produce a uid
from DECLARE PLUGIN, the file name and the number of occurences of
GRAMMAR EXTEND in the file (we end up with eg
`coq.plugins.ltac:coretactics.mlg:42`).

In the future we will probably try to replace the redo code of
ByEXTEND with an error and make the users in plugins switch to the new
mechanism.
For now it is too much work, when a plugin is modifying its own
entries it doesn't matter if backtrack is undone.
ejgallego pushed a commit to ejgallego/coq that referenced this issue Feb 2, 2023
Fix rocq-prover#12575

This was hand tested using
~~~coq
Require Import Ltac2.Ltac2.

Ltac2 Eval ().
~~~
we must correctly handle
- eval all, backtrack before the Require, eval just Ltac2 Eval: should
  parsing error illegal begin etc
- eval all, backtrack before the Require, eval all again: should
  work (ie the parsing rules must be re-applied)

The interesting part is what happens in vernac_extend and the
associated egramml functions:
- an indirection is added from the command name (Foo in
  `VERNAC COMMAND EXTEND Foo`) to the extension data (which contains a
  closure: not marshallable).
  This is non-backtracking, and there is no use to making it backtrack
  since we can't unlink plugins.
- vernac_extend calls `Egramml.extend_vernac_command` with `undoable:false`
  when there is no `plugin` argument, this is the old non
  backtrackable behaviour. Typically for non-plugin callers like
  g_obligations.
  Eventually we will clean up so that rules added this way produce an
  error when backtracked over instead of having the redo code in pcoq,
  until then it's also available for plugin hacks which don't mind the
  backtrack bug I guess.
- when there is a `plugin` argument, vernac_extend wraps the call to
  `Egramml.extend_vernac_command` in `Mltop.add_init_function` and
  uses `undoable:true`. In undoable mode, egramml uses the
  Pcoq.create_grammar_command system which backtracks properly.
  Because this system puts its input in the grammar_stack which ends
  up in the summary, we need the indirection mentioned above to keep
  the input marshallable.

For GRAMMAR EXTEND there is no user-given name so we produce a uid
from DECLARE PLUGIN, the file name and the number of occurences of
GRAMMAR EXTEND in the file (we end up with eg
`coq.plugins.ltac:coretactics.mlg:42`).

In the future we will probably try to replace the redo code of
ByEXTEND with an error and make the users in plugins switch to the new
mechanism.
For now it is too much work, when a plugin is modifying its own
entries it doesn't matter if backtrack is undone.
ejgallego pushed a commit to ejgallego/coq that referenced this issue Feb 9, 2023
Fix rocq-prover#12575

This was hand tested using
~~~coq
Require Import Ltac2.Ltac2.

Ltac2 Eval ().
~~~
we must correctly handle
- eval all, backtrack before the Require, eval just Ltac2 Eval: should
  parsing error illegal begin etc
- eval all, backtrack before the Require, eval all again: should
  work (ie the parsing rules must be re-applied)

The interesting part is what happens in vernac_extend and the
associated egramml functions:
- an indirection is added from the command name (Foo in
  `VERNAC COMMAND EXTEND Foo`) to the extension data (which contains a
  closure: not marshallable).
  This is non-backtracking, and there is no use to making it backtrack
  since we can't unlink plugins.
- vernac_extend calls `Egramml.extend_vernac_command` with `undoable:false`
  when there is no `plugin` argument, this is the old non
  backtrackable behaviour. Typically for non-plugin callers like
  g_obligations.
  Eventually we will clean up so that rules added this way produce an
  error when backtracked over instead of having the redo code in pcoq,
  until then it's also available for plugin hacks which don't mind the
  backtrack bug I guess.
- when there is a `plugin` argument, vernac_extend wraps the call to
  `Egramml.extend_vernac_command` in `Mltop.add_init_function` and
  uses `undoable:true`. In undoable mode, egramml uses the
  Pcoq.create_grammar_command system which backtracks properly.
  Because this system puts its input in the grammar_stack which ends
  up in the summary, we need the indirection mentioned above to keep
  the input marshallable.

For GRAMMAR EXTEND there is no user-given name so we produce a uid
from DECLARE PLUGIN, the file name and the number of occurences of
GRAMMAR EXTEND in the file (we end up with eg
`coq.plugins.ltac:coretactics.mlg:42`).

In the future we will probably try to replace the redo code of
ByEXTEND with an error and make the users in plugins switch to the new
mechanism.
For now it is too much work, when a plugin is modifying its own
entries it doesn't matter if backtrack is undone.
ejgallego pushed a commit to ejgallego/coq that referenced this issue Feb 14, 2023
Fix rocq-prover#12575

This was hand tested using
~~~coq
Require Import Ltac2.Ltac2.

Ltac2 Eval ().
~~~
we must correctly handle
- eval all, backtrack before the Require, eval just Ltac2 Eval: should
  parsing error illegal begin etc
- eval all, backtrack before the Require, eval all again: should
  work (ie the parsing rules must be re-applied)

The interesting part is what happens in vernac_extend and the
associated egramml functions:
- an indirection is added from the command name (Foo in
  `VERNAC COMMAND EXTEND Foo`) to the extension data (which contains a
  closure: not marshallable).
  This is non-backtracking, and there is no use to making it backtrack
  since we can't unlink plugins.
- vernac_extend calls `Egramml.extend_vernac_command` with `undoable:false`
  when there is no `plugin` argument, this is the old non
  backtrackable behaviour. Typically for non-plugin callers like
  g_obligations.
  Eventually we will clean up so that rules added this way produce an
  error when backtracked over instead of having the redo code in pcoq,
  until then it's also available for plugin hacks which don't mind the
  backtrack bug I guess.
- when there is a `plugin` argument, vernac_extend wraps the call to
  `Egramml.extend_vernac_command` in `Mltop.add_init_function` and
  uses `undoable:true`. In undoable mode, egramml uses the
  Pcoq.create_grammar_command system which backtracks properly.
  Because this system puts its input in the grammar_stack which ends
  up in the summary, we need the indirection mentioned above to keep
  the input marshallable.

For GRAMMAR EXTEND there is no user-given name so we produce a uid
from DECLARE PLUGIN, the file name and the number of occurences of
GRAMMAR EXTEND in the file (we end up with eg
`coq.plugins.ltac:coretactics.mlg:42`).

In the future we will probably try to replace the redo code of
ByEXTEND with an error and make the users in plugins switch to the new
mechanism.
For now it is too much work, when a plugin is modifying its own
entries it doesn't matter if backtrack is undone.
ejgallego pushed a commit to ejgallego/coq that referenced this issue Feb 14, 2023
Fix rocq-prover#12575

This was hand tested using
~~~coq
Require Import Ltac2.Ltac2.

Ltac2 Eval ().
~~~
we must correctly handle
- eval all, backtrack before the Require, eval just Ltac2 Eval: should
  parsing error illegal begin etc
- eval all, backtrack before the Require, eval all again: should
  work (ie the parsing rules must be re-applied)

The interesting part is what happens in vernac_extend and the
associated egramml functions:
- an indirection is added from the command name (Foo in
  `VERNAC COMMAND EXTEND Foo`) to the extension data (which contains a
  closure: not marshallable).
  This is non-backtracking, and there is no use to making it backtrack
  since we can't unlink plugins.
- vernac_extend calls `Egramml.extend_vernac_command` with `undoable:false`
  when there is no `plugin` argument, this is the old non
  backtrackable behaviour. Typically for non-plugin callers like
  g_obligations.
  Eventually we will clean up so that rules added this way produce an
  error when backtracked over instead of having the redo code in pcoq,
  until then it's also available for plugin hacks which don't mind the
  backtrack bug I guess.
- when there is a `plugin` argument, vernac_extend wraps the call to
  `Egramml.extend_vernac_command` in `Mltop.add_init_function` and
  uses `undoable:true`. In undoable mode, egramml uses the
  Pcoq.create_grammar_command system which backtracks properly.
  Because this system puts its input in the grammar_stack which ends
  up in the summary, we need the indirection mentioned above to keep
  the input marshallable.

For GRAMMAR EXTEND there is no user-given name so we produce a uid
from DECLARE PLUGIN, the file name and the number of occurences of
GRAMMAR EXTEND in the file (we end up with eg
`coq.plugins.ltac:coretactics.mlg:42`).

In the future we will probably try to replace the redo code of
ByEXTEND with an error and make the users in plugins switch to the new
mechanism.
For now it is too much work, when a plugin is modifying its own
entries it doesn't matter if backtrack is undone.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind: anomaly An uncaught exception has been raised. kind: bug An error, flaw, fault or unintended behaviour. part: parser The parser (also called gramlib, forked from camlp5)
Projects
None yet