Unit testing means isolation?

I've been tasked with heading up the design and specification of a testing framework for my team at work. At first, I was looking at comprehensive testing frameworks like Tapper and Jenkins, but I realized we really need to think about the way our unit tests are run and then how those tests are automated and results tracked later.

Because we're using Perl, it's a no-brainer that our tests will generate TAP, so we just need to make sure whatever encompassing frontend automation we add to the mix later on understands TAP.

Attributes of unit tests

Unit tests are a somewhat nebulous concept. A lot of it has to do with how pure you want to be. The purist's unit test has the following attributes:

• Can be be fully automated
• Operates in complete isolation from other systems/subsystems.
  • Use mocks, stubs, fakes and test harnesses may help accomplish this isolation when necessary.
  • Doesn't depend on precedence or order of other tests run.
• Runs in memory. Doesn't touch databases, remote data stores (e.g. HTTP, etc.)
• Always returns predictable results.
• Runs "fast."
• Tests a single "unit" or logical concept.
• Readable.
• Maintainable.
• Trustworthy.

(A tip of the hat to the author of The Art of Unit Testing in Ruby, Java, and .NET because I based the above list on one that appears on their website and probably in the book as well.)

How pure?

One of our projects is an API layer that acts as middleware between remote libraries and a billing system backend. Each of these API method invocations results in a call to billing system methods that, in turn, touch a relational database and may also touch yet-another API.

Creating unit tests for this middleware API in complete isolation would mean mocking logic provided by the billing system code. Some of that code is, well, pretty intense.

So, my question is this: Can we bend the rules on total and complete isolation in exchange if we make sure other rules/guidelines are adhered to?

The billing system can't really run without a relational database backend. Some of the articles I've read about writing unit tests for systems that depend on a database backend is to run tests instead on a lightweight in-memory database or something like a temporary SQLite instance.

Because the billing system is not operative at all without certain data in the database at a minimum, we pretty much have to have a database and it has to have a minimal amount of data in it. My current thought is that we load that minimum dataset and then create a dump or snapshot of the database and use that as the basis for testing.

Doing this, unit tests would go through startup and teardown similar to the following:

• Create temporary database
• Load snapshot/dump data into temporary database
• Start temporary instance of billing system which associates with the temporary database
• Run API unit tests
• Drop temporary database

I'm not sure yet if this process of loading a snapshot of billing system data would qualify as "fast," but it could guarantee consistent, predictable test results. We may also be able to load the snapshot once for many tests run in a group, thereby minimizing the overhead of instantiating the test database.