I recently did a test for a game company. It was an assignment sent to me by email, and I was given a time limit to get it done. I did it, tested it, and it worked fine, so I sent it back to them. They came back to me saying that it failed their "testing harness". So I must not have tested it thoroughly enough. Now I'm in the same situation with another company, and I really want to pass their testing harness. I find it hard to test my own code; it's easy to come up with tests that it will pass, and hard to come up with tests that it will fail. I want to believe that it'll work, and so that works against me. I was wondering if anyone had any advice for properly testing your own code, what do you usually do to make sure your code is flawless?

It depends on the code in question.

Much code is transformitive. You get some input values, you run some operations, and you transform it to a result.

A test harness is another program that is used to test the code. Usually that means giving specific inputs and testing the results, or passing in proxy objects and verifying the operations done on the objects are correct. It can be something custom written for their code, or larger existing test frameworks like xUnit (JUnit, NUnit, CppUnit, ...) or similar. These test systems usually have you write up a large number of tests, one test per thing you are testing, and run automatically as part of the build system. Other test frameworks do some work to test whatever it is they are interested in, and ensure the thing hasn't been broken. Most testing tools are designed to test permanence, to ensure the behavior has not changed unexpectedly.


When building tests, look at boundaries and the operations involved.

Does it do what the design says it is supposed to do? Does it do anything extra that it shouldn't? Many defects are caused by missing items on the list. An action is supposed to do five steps, but only four were implemented. Or it is supposed to do one thing, but has some extra side effects.

Does it operate at the full range of boundaries? If you've decided a function can take a range of numbers, does it work with the full range? Generally you can test this with endpoints and a few mid-point values. If there is a specified result outside the range, such as returning a code or throwing an exception, verify that on both ends of the range it generates those results. If the range includes key values like null inputs, test those too. If it requires non-null parameters, make that assert noisily on debug builds, and follow your project's practice on release builds, like returning a failsafe result or log and crash. Those can be done black-box, you don't need to see what is inside the code to test it.

If you have access to the code you can do some white box tests on that, too. If your transformation involves other operations, does it operate within those ranges? For example, many implementations require input to sin/cos operations be within a specific range, does your call fit the range? How about square roots of negative numbers? If any library you are using has limits, can you verify that the full range of your boundaries also fits the range of those libraries? Review it for the full range and boundaries.

Next, look for key values and error propagation. Look at the operations involved. If it involves division, is there a chance for division by zero? Are there any operations you use that can fail? How do you handle any calls that can potentially return null? Do you test for nulls for everything before dereferencing any pointer? Are you testing return codes for all functions you call that potentially return an error result?

Those are the basics I usually look at for all code I write. The pattern is so automatic I have to stop and think carefully about what I typically search for.

There are many more tests you can build, all are depending on what the system is supposed to be doing. There are also many good books on the subject and hundreds of test patterns, much like design patterns for code, but patterns in making sure software is supposed to do what it is supposed to do.

Thanks for the responses! I tried Visual Leak detector, and It's detecting leaks for things I know I de-allocated. It's also detecting leaks for simple things like:

class foo
{
    std::string name;
    void foo::func(const char* str)
    {
        name = str;
    }
}

apparently that calls 'new' inside of std::string somewhere? Super strange. Is it normal for something like visual leak detector to give a lot of false positives?

PS: If the program's closing anyway, why does anyone care if there's leaks?

I tried Visual Leak detector, and It's ... Is it normal for something like visual leak detector to give a lot of false positives?

That is more technical than job-advice.

It might not be a false positive. It might indicate you are leaking foo objects. Assignment to a string can allocate memory, but the snippet you posted does not leak itself. Assuming you properly clean up your foo object, the string inside it should release the memory it allocated. Leak detection tools indicate that something was allocated but not cleaned up yet. Maybe you really did leak a foo object, or maybe it will still get cleaned up at some point after the tool stops detecting. That would require looking at more code.

If you want help with finding memory leaks, post the relevant information in a discussion topic in another area, perhaps in 'general programming'.

PS: If the program's closing anyway, why does anyone care if there's leaks?

Several reasons, but again that is a technical question rather than job advice.

Since this was for a coding test for a company, it is an example of your work. You're submitting sloppy work. Leaks are always problem. They accumulate and grow and consume additional system resources until eventually your program dies. Employers will wonder if all your work is that sloppy.

You are somewhat right about program termination. Many libraries have 'fast teardown' options that do not call destructors or otherwise does not do cleanup. Sometimes this happens when the program is shutting down and the objects can be dumped without consequence. Sometimes code works with pool allocators and the pool will be recycled. If the blocks are being destroyed and recycled and there is no additional cleanup to do, having a fast teardown that doesn't clean up is fine. Note that this is not accidentally leaked, this is typically an explicit action.

Sometimes there is stuff that needs to be handled. Sometimes there are buffers that need to be flushed, sent to disk or across a network. Sometimes there are stats that need to be computed and stored. Sometimes there are resources that the system cannot easily reclaim and should be returned to a proper state. If those are what you are leaking, you end up losing vital information or leaving the system unstable. If someone wrongly used a fast teardown on those objects that is a bug caused by being too aggressive. If someone just leaks the objects, that is probably a bug from lazy or not fully understanding the system.

Bringing both technical questions back to job advice...

Understanding object lifetimes is a vital thing in programming. If your code test shows you don't know how to manage object lifetimes, that's a good warning sign that you may not be the best programmer for the job. Maybe if they are looking for programmers in languages less sensitive to object lifetimes you might work out with some training, but that is a bad candidate for a C++ programming job.

During interviews if one candidate writes code with memory or resource leaks and another candidate writes clean code, the second is far more likely to get the job.