Unit Testing in Corona SDK Using Lunatest

In this article I ‘ll give you an overview of why you’d want to unit test your Lua code in Corona SDK, what features Lunatest has, a walk you through some real-world unit tests and suites I’ve written. I’ve forked Lunatest on Github to make it work with Corona SDK. This blog post has a companion video embedded below.

 

Introduction: Why Unit Test?

Lua in Corona SDK has 3 issues when your game/application code base grows to any large size.

First, Lua is a dynamic language much like JavaScript, Ruby, and Python. As such, you have little to no help from the compiler to ensure your code expects even basic parameters being passed to your functions as well as misspellings.

Secondly, Lua is dynamic at runtime. You actually need to run the code to see how it actually works. Examples include combining strings together to dynamically access properties on dynamically created objects in an asynchronous way.

Third, even if encapsulated, you have no way of knowing if your new code breaks existing code, or if modifying existing code allows it to work the same way with some assurance. This problem is not unique to Lua, it’s unique to any complex code base regardless of language.

All 3 combined necessitate a need for unit tests. Some would argue even just 1 of the above.

Anatomy of Unit Testing

Unit testing, as far as we’re concerned with Lua, are a bunch of functions that make sure things work as expected. Inside of those functions, you write those expectations about how you expect things to work as a bunch of assert functions. These groups of functions aren’t actually part of your main code base; they’re grouped together in a test folder and called a test suite. The more of them you have, the more test coverage you have.

TDD or Test Driven Development is where you actually write the above paragraph test code first, and THEN actually write the code it’s testing afterwards. You start with writing a test that fails, you write the code to make it pass, and then ensure the test passes. Then you repeat.

The above 2 paragraphs can, and do fill books with the details of TDD & unit testing… but that’s all you need to know for now. Heck, you can ignore everything I write here, start to proactively use assert, and your coding life will improve a lot.

Asserts
Let’s start at the beginning. Lua already has an expectation function, called “assert”. It looks like this:

local wubWubWub = true
assert(wubWubWub, "Someone forgot to drop the bass.")

This code asserts or “states positively” that the variable is true. If it were false, an error would be thrown with that string “Someone forgot to drop the bass.” passed into the error function. If you’re not aware, the error function in Lua stops execution of the current script and prints whatever string it has to the console/terminal. Secondly, all functions in Lua are implied to be successful if they return true.

Even without unit testing, using assert, you can be proactive in finding errors. For example, I use it all the time in manual getter/setters in Lua (not metatable ones, the set/get + method name convention). Since Lua has no typing system, I ensure my inputs are sanitized like so:

function setSpeed(newValue)
   assert(newValue ~= nil, "You cannot set speed to nil.")
   assert(type(newValue) == "number", "You must set speed to a number value.")
   self.speed = newValue
end

Those 2 functions alone find ALL kinds of silly bugs where someone, somewhere accidentally set a value to nil and it made it’s way deep into the bowels of some class, and I’m wondering who’s fault it is. When everyone starts doing this, it makes it really easy to find who caused the bug, and quickly.

Yes, there is a performance cost to assert. However, it’s easier to remove a bunch of assert functions scattered everywhere via a Find / Replace from solid code then it is at add them to code that runs very fast with lots of unknown bugs.

Moar Asserts

Just about every testing framework expands upon the above function. In fact, 80% of what they offer in their libraries and frameworks are additional assert functions. Why? Three reasons. First, would you rather write this?

local icon = display.newImage("icon.png")
assert(icon ~= nil, "Icon image cannot be found.")

or this:

local icon = display.newImage("icon.png")
assert_not_nil(icon, "Icon image cannot be found.")

The second reason is it’s more clear what that is testing; the function name alone tells you what you need to know before you’ve even made it to the parameters.

Here’s another example:

local currentSpeed = character.speed
character:setSpeed(currentSpeed + 1)
assert(character.speed > currentSpeed, "Character speed did not increase.")

vs writing:

local currentSpeed = character.speed
character:setSpeed(currentSpeed + 1)
assert_gt(character.speed, currentSpeed, "Character speed did not increase.")

Now this one may appear backwards; “A comma vs. using the normal mathematical construct!?”

Keep in mind, however, many bugs are caused by misplaced operators. For example, developers often will use = instead of == to assign a value, or they’ll use > vs. >= for edge cases, etc. In writing that code, it’s completely legal, and Lua’s interpreter won’t give you an error when you run it. Using assert_gt in this instance will ensure your expected value is there, as well as ensure you’ve use the operator correctly. If not, it’ll actually… you know… help you out!

And although I bash Lua for not having a type system, it does have a type function for a reason. You can test a variety of types such as assert_not_boolean for a return value that’s supposed to be a number, and assert_metatable if you’re doing funky metatable magic.

Suites

A suite is just a series of functions that are related. For example, in my example video I show my file suite which has unit tests for testing my DeleteFileService class and my ReadFileContentsService class. Since reading and writing and deleting files are something I consider related, I created a file suite, or “a bunch of functions that test file shiz”.

While lunatest can test functions inline, as an OOP guy, to me that’s just gross. I like the more organized suites where all the functions are in their own file. Lua calls it a file, I call it a Class, Lunatest calls it a suite… we’re all happy.

Here’s my file one:

module(..., package.seeall)

function test_saveFile()
	require "com.jxl.core.services.ReadFileContentsService"

	local saveFile = SaveFileService:new()
	local data = "moo"
	assert_true(saveFile:saveFile("test.txt", system.DocumentsDirectory, data), "Failed to save test.txt")
end

function test_readFile()
	require "com.jxl.core.services.ReadFileContentsService"
	require "com.jxl.core.services.SaveFileService"

	local saveFile = SaveFileService:new()
	local data = "moo"
	assert_true(saveFile:saveFile("test.txt", system.DocumentsDirectory, data), "Failed to save test.txt for use in reading.")

	local readFile = ReadFileContentsService:new()
	local contents = readFile:readFileContents("test.txt", system.DocumentsDirectory)
	assert_string(contents, "contents are not a string.")
	assert_equal(contents, data, "Data written to file does not match what we just read out of it.")
end

2 things to note about this file. First, I use the much loathed module(…, package.seeall) at the top. This is what lunatest wants in how it loads modules so… you’ll need to do this too (yes, I temporarily hacked lunatest to support Corona’s packages, but it wasn’t worth it).

Second, notice the 2 functions are both global and start with “test_”. Any function you want Lunatest to run as a test, you need to prefix it with “test_” in the function name. Notice they aren’t prefixed with local, they’re simple global functions.

That’s it. Put a bunch of functions in that file, make an assert or two within them, save it, and then in your main file, tell luna test about it.

Advanced Suites

The above is good enough for writing suites that’ll help you. Here’s 4 more cool things suites can do.

If you have repetitive code that needs to be run before every test, you can run it in the test setup. This ensures you only have to run it once, you have the option of aborting the test, and it’s run in a protected manner (aka pcall). Additionally, you don’t have to call this yourself, nor remember to do so, Lunatest does this for you. In my StateMachineSuite, I have to instantiate the StateMachine class for every call so I put it in setup.

If you have code that needs to be cleaned up a lot or in a particular order, you can utilize the teardown. If you implement one, it’s called after every single test case. This also ensures that if everything is deleted, strange memory bugs are more easily found. Some classes require more than just nilling out their variables, in the case of Corona SDK Display Objects and Widgets which must have their removeSelf called, THEN you can nil out the variable reference. You can also ensure all of the benefits of setup, just in a function called teardown where you clean up your mess.

Finally, there is a suite_setup and a suite_teardown. Unlike setup and teardown, these are only called once for the entire suite; suite_setup before any tests are run, and suite_teardown after everything, including the last teardown, is called. Sometimes you don’t need unique setup code for your tests so you can put it here instead. This is also a good place for class imports.

Here’s my StateMachine example. Notice it has a suite_setup (but no suite_teardown). The setup instantiates the StateMachine and the teardown ensures it’s deleted so each test function gets a brand new SuiteMachine instance.

module(..., package.seeall)

function suite_setup()
	require "com.jxl.core.statemachine.State"
	require "com.jxl.core.statemachine.StateMachine"
	require "com.jxl.core.statemachine.BaseState"
end

function setup()
	machine = StateMachine:new()
end

function teardown()
	machine = nil
end

function test_classWorks()
	assert_not_nil(machine)
end

function test_verifyInitialStateIsNil()
	assert_nil(machine.state)
end

function test_verifyInitialStateIsNullWithStates()
	local initial = "playing"
	machine:addState(initial)
	machine:addState("stopped")
	assert_nil(machine.state)
end

function test_verifyInitialStateIsNotNil()
	local initial = "playing"
	machine:addState(initial)
	machine:addState("stopped")
	machine:setInitialState(initial)
	assert_equal(initial, machine.state)
end

function test_enter()
	local t = {}
	local hitCallback = false
	function t.onPlayingEnter(event)
		assert_equal(event.toState, "playing")
		assert_equal(event.fromState, "idle")
		hitCallback = true
	end
	machine:addState("idle")
	machine:addState("playing", { enter = t.onPlayingEnter, from="*"})
	machine:setInitialState("idle")
	assert_true(machine:canChangeStateTo("playing"), "Not alowed to change to state playing.")
	assert_true(machine:changeState("playing"))
	assert_equal("playing", machine.state)
	assert_true(hitCallback, "Didn't hit the onPlayingEnter callback.")
end

function test_preventInitialOnEnterEvent()
	local t = {}
	local hitCallback = false
	function t.onPlayingEnter(event)
		hitCallback = true
	end
	machine:addState("idle")
	machine:addState("playing", { enter = t.onPlayingEnter, from="*"})
	machine:setInitialState("idle")
	assert_false(hitCallback, "Hit the callback when I had no initial state set.")
end

function test_exit()
	local t = {}
	local hitCallback = false
	function t.onPlayingExit(event)
		hitCallback = true
	end
	machine:addState("idle", {exit = t.onPlayingExit})
	machine:addState("playing", {from="*"})
	machine:setInitialState("idle")
	machine:changeState("playing")
	assert_true(hitCallback, "Never called onPlayingExit.")
end

function test_ensurePathAcceptable()
	machine:addState("prone")
	machine:addState("standing", {from="*"})
	machine:addState("running", {from={"standing"}})
	machine:setInitialState("standing")
	assert_true(machine:changeState("running"), "Failed to ensure correct path.")
end

function test_ensurePathUnacceptable()
	machine:addState("prone")
	machine:addState("standing", {from="*"})
	machine:addState("running", {from={"standing"}})
	machine:setInitialState("prone")
	assert_false(machine:changeState("running"), "Failed to ensure correct path.")
end

function test_hierarchical()
	local t = {}
	local calledonAttack = false
	local calledOnMeleeAttack = false
	function t.onAttack(event)
		calledonAttack = true
	end

	function t.onMeleeAttack(event)
		calledOnMeleeAttack = true
	end

	machine:addState("idle", {from="*"})
	machine:addState("attack",{from = "idle", enter = t.onAttack})
	machine:addState("melee attack", {parent = "attack", from = "attack", enter = t.onMeleeAttack})
	machine:addState("smash",{parent = "melee attack", enter = t.onSmash})
	machine:addState("missle attack",{parent = "attack", enter = onMissle})

	machine:setInitialState("idle")

	assert_true(machine:canChangeStateTo("attack"), "Cannot change to state attack from idle!?")
	assert_false(machine:canChangeStateTo("melee attack"), "Somehow we're allowed to change to melee attack even though we're not in the attack base state.")
	assert_false(machine:changeState("melee attack"), "We're somehow allowed to bypass the attack state and go straigt into the melee attack state.")
	assert_true(machine:changeState("attack"), "We're not allowed to go to the attack state from the idle state?")
	assert_false(machine:canChangeStateTo("attack"), "We're allowed to change to a state we're already in?")
	assert_true(machine:canChangeStateTo("melee attack"), "We're not allowed to go to our child state melee attack from attack?")
	assert_true(machine:changeState("melee attack"), "I don't get it, we're in the parent attack state, why can't we change?")
	assert_true(machine:canChangeStateTo("smash"), "We're not allowed to go to our smash child state from our parent melee attack state?")
	
	assert_true(machine:canChangeStateTo("attack"), "We're not allowed to go back to our parent attack state?")
	assert_true(machine:changeState("smash"), "We're not allowed to actually change state to our smash child state.")
	assert_false(machine:changeState("attack"))
	assert_true(machine:changeState("melee attack"))
	assert_true(machine:canChangeStateTo("attack"))
	assert_true(machine:canChangeStateTo("smash"))
	assert_true(machine:changeState("attack"))
end

Pay attention to that last part, it has some good and bad practices. The whole point of unit tests is to test 1 thing or unit. You shouldn’t have 50 billion asserts like I do. Worse, you’ll notice some of the asserts at the bottom do not have custom error messages in the 3rd parameter. If any 1 of those fail, there is no way to tell which one failed. Bad, Jesse, BAD!

Run

Setting up all of the above is a snap. Ready?

In your main.lua file, import Lunatest:

require "lunatest"

Next, add your suites:

lunatest.suite("tests.com.jxl.core.ServicesSuite")
lunatest.suite("tests.com.jxl.core.services.ReadFileContentsServiceSuite")
lunatest.suite("tests.com.jxl.core.statemachine.StateMachineSuite")
lunatest.suite("tests.com.jxl.zombiestick.services.LoadLevelServiceTest")

Finally, call run:

lunatest.run()

That’s it! You’ll see the report of stuff in the console/terminal.

Conclusions

The problem with unit testing is that you need to have units to actually test. A lot of code that wasn’t written to be unit tested needs to be refactored first which can be a lot of work. If it’s also entangled with other code that’s also not easily testable… you see where this is going. That’s ok. You can start in one section, or test high level results. Yes, your setup code may be burdensome, but it’s something. Automated tests, even if a certain section, can help you a lot. Additionally, when writing new code, think about writing it in a way that’s testable, even if you never actually test it. This often results in less coupled code anyway.

Additionally, while testing is in fact creating more code, you find more bugs, early, and faster, proactively. For games that require quick iterations… this can be tricky. If the game needs to be built to verify it’s even fun, is it worth it to unit test? As you make a lot of games in the same tech (whether Lua, C#, ActionScript, Java, whatever…), you’ll start to gather are a large set of functions, libraries, and classes that are reusable from project to project. THOSE are worth unit testing.

Unit tests are just a bunch of functions that test expectations, and are organized in suites. You run a bunch of test suites inside of Lunatest. Unit tests help ensure your function types are correct since Lua has no helpful compiler to help on this front as well as spelling mistakes. It also verifies your code works as you expect at runtime. Finally, for larger code bases, you can add new code without worry of breaking old code, including if you modify old code. The more unit tests you write, the more coverage you have. The more coverage, the easier it is to manage a larger code base.

Remember, unit tests are code too. You can still follow all the best practices your used to such as OOP, DRY, and being clean. MOAR UNIT TESTS!

Title image “Moon” by S4cr4m3nt.

Also posted on iOS Gaming.

4 Replies to “Unit Testing in Corona SDK Using Lunatest”

  1. thanks for the article – can I ask a few questions:

    Q1 – So do you just effectively run the tests every time you run up the app during development then? (noting you run it via main.lua) Wondering if there are some addition seconds/minutes of lag time each time you want to rerun your app during development after making a change

    Q2 – Does Lunatest have an autorun / continuous integration tool, so it just runs every time you save a file (like Ruby autotest) and then you can get a growl notification if there is a fail? Or perhaps the issue is then the tool would need to have it’s own lua engine and not use the Corona SDK??? Would be nice though

    Q3 – Did you look at any other libraries out there re unit testing for Corona? i.e. if so and you’ve already gone through this we might then jump straight to Lunatest too

    Q4 – So you recommend to get the modified version of Lunatest you did correct? Is there any issues in terms of Lunatest updates here? Not sure if you’re considering considering keeping your fork up to date for this? (or how much hacking was involved to change)

    thanks again

  2. PS. Can I another one :)

    Q5 – Is there a coverage report you can get? like RCov for ruby?

  3. 1. No, but I should. I don’t currently because I haven’t setup a clean way to remove the classes once required. It’s not a clean test if I have classes hanging around, or even globals from bad code. But yes, I should.

    2. Geez, that would be nice. As far as I know there isn’t. Corona SDK has a desktop emulator, and this emulator is a command line tool on top of their stuff. It does have a step debugger, but that’s about it. I’m sure there is something out there, though, given Lua’s variety of implementations (embedded testing, etc). I just haven’t looked myself. That, plus, I haven’t done that large of implementations on Lua, heh; usually CI is for larger language frameworks… however, now that we have a Node.js version called Luvit, maybe that’ll change?

    3. Yes, they didn’t appear to have the bredth of asserts, and everyone I found online and in forums seemed to say Lunatest was the most legit. That, and I don’t like things mucking around with assert/metatables; that was a selling point of a few of them.

    4. You don’t need it if you’re not using Corona SDK. If you are, you need it, else it’ll below up. Search for “[jwarden” in the code, you’ll see my 3 minor changes.

    No, there’s nothing to keep up to date, really. That is, unless a newer build of Corona SDK breaks it, in which case I’ll fix it.

    5. I know there is luacov, but I haven’t been able to get it to work yet. Not sure if it’s my class-as-closure based system, or the way I do require or what yet…

Comments are closed.