Skip to content

[WIP] Modified Lua 5.4.7 test suite #219

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: develop
Choose a base branch
from

Conversation

cmdrk
Copy link
Contributor

@cmdrk cmdrk commented Jun 29, 2025

This PR attempts to lightly adapt the Lua 5.4.7 test suite to Luerl. I have gone through most (if not all) of the tests and commented out the tests that are not functional and added an appropriate print() noting that the test is non-functional and a TODO in the comments where applicable. As we've discussed in Discord, a lot of the failing tests may not make sense to fix in Luerl, but I do think I've identified a few a few genuine compatibility bugs here and there.

I don't expect this PR to get merged as-is, but I think it's a starting point for building something Luerl-specific. Maybe we could discuss an appropriate structure (eunit-based, perhaps?) for Luerl-specific tests, based off of the Lua test suite, that could run as part of the GitHub action, or separately. That's my thinking for the ultimate direction of this PR (or a new PR after closing this one).

Here's how I've been approaching each file:

D = code:priv_dir(luerl) ++ "/lua-5.4.7-tests/".
F = fun(File) -> 
    case luerl:dofile(D ++ File, luerl:init()) of
        {ok, _, _} -> ok;
        {lua_error, E, _St} -> E
  end
end.

Which gives an output like this:

F("math.lua").                 
testing numbers and math lib        
NOT integer testing overflow properties 
NOT testing NaN                   
testing floating point precision limit 
64-bit integers, 53-bit (mantissa) floats 
testing types of integers and floats                                                          
testing basic float notation                                                                  
string string to number coercion      
testing minus zero as table key     
testing modf                 
NOT testing modf with NaN (division by zero) 
testing the size of positive and negative math.huge 
testing integer arithmetic for max/min integers  
...and so on

Where possible, I've tried to describe the test in print() if an existing comment didn't exist -- some of these may not be accurate. I've also annotated some of the nonfunctioning tests with a TODO comment, which you can grep through:

$ grep -B1 "TODO" *.lua
# ...
math.lua-print('NOT testing negative exponents')
math.lua:-- TODO: {badarith,'^',[0,-3]}
--
math.lua-print("NOT testing precision of module for large numbers")
math.lua:--TODO: shell hang
--
math.lua-print("NOT testing return of randomseed()")
math.lua:--TODO: {badarg,randomseed,[]}
--
math.lua-print("NOT testing random for floats")
math.lua:--TODO: relies on goto behavior
# ...

Broadly summarizing the categories of non-functional tests:

  1. Unimplemented features. This includes goto, coroutine, etc. Some of the tests also (as you might expect) depend on functionality that hasn't been implemented in debug or io, for instance. I don't see these as particularly urgent.
  2. Shell hangs, evidenced by the function not returning and my CPU/RAM going to 100% utilization. I am not sure how many of these are unbounded vs very heavy resource consumption. Some of the tests in heavy.lua, for instance, hung up the shell for a bit, but were OOM-killed on a machine with 32GB RAM. Other tests unexpectedly hung the shell for reasons I couldn't understand. I've tried to note these in TODOs where I remembered to do so :)
  3. Assertion failures due to mismatched error messages between Lua and Luerl. The Lua test suite does a lot of string matching on error messages which don't match in Luerl. There are a lot of these, and personally I would not prioritize them.
  4. Assertion failures due to implementation differences or gaps. These seem like the important ones to consider changing to match PUC Lua if possible.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

1 participant