New Site

My new site can be found here: http://www.softwarepurist.com

All new blog posts will be at this location.

Software Testing Follow-Up

As a follow-up to some of the software testing types I posted in an earlier blog, I came across this, which is a great description of various different types of software testing:

http://bugsniffer.blogspot.com/2007/11/software-testing-types.html

Timers During Daylight Savings Time

With Daylight Savings Time just passing early this morning, it occurred me that timers could have a subtle bug. This occurred to me today because I was running some overnight tests last night. While the tests exhibited no issues with the extra hour, I can foresee scenarios where it could have.

For instance, let's say you have a C++ class that looks something like this (Pseudo-code):

class Timer : public Thread
{
public:
    virtual void run()
    {
           while (running)
           {
                get current time
                while (current time < time value at top of timer queue)
                {
                      pop
                      notify timer listener
                }
                wait for top of timer queue to expire
           }
    }
};

This code is relatively flawless, except, what if the current time drops back an hour? This can occur, depending on how you retrieve your time. Now, in the worst case, all of the timers in your system won't trigger for an extra hour. This basically means your code is mostly nonfunctional for an hour after Daylight Savings Time occurs. In an application, you can just restart the application and the problem vanishes. On server code, this can be a bigger issue at this peculiar time.

Here's one solution I thought of to correct this potential problem:

class Timer : public Thread
{
public:
    virtual void run()
    {
           while (running)
           {
                get current time
                if (current time < last known time)
                {
                       create temporary queue
                       pop each element of queue, subtract difference, push to new queue
                       pop each element off temporary queue, push back to original queue
                }

                while (current time < time value at top of timer queue)
                {
                      pop
                      notify timer listener
                 }
                set last known time equal to current time
                wait for top of timer queue to expire
            }
    }
};

If you do this, you basically have this level of correction code to handle this peculiar scenario. The solution is that every timer is adjusted to handle the problem. Of course, this isn't a perfect solution, but it's one way I can think to mitigate the issue. I may write a test later to see if this solution is good enough or if there are better ones. I suspect many well-known timer libraries handle this already, but in C++, you can sometimes justify rolling your own depending on your needs.

Software Testing

So, it's been a long while since I've made a blog post. I decided to revive the blog this weekend. I was thinking a good topic would be to talk about a few different types of tests a software developer can do and what each provide.

Unit Tests

The purpose of unit tests is to test the functionality of a generally lower-level software module. External dependencies are stubbed out and mocked, so that you can test components in isolation. Checks are very granular and the test will generally give a pass/fail result, which makes it easily automatable. In my experience, there is often some confusion as to what a unit test can and cannot do well. For example, you might write a unit test that tests a class called String. It might find that you have two issues with this class that you fix. However, this won't necessarily have a noticeable effect on your product. Unit tests are to make the developer's life easier, because they take really difficult-to-fix bugs that wouldn't be found until later stages and make them into generally easy-to-fix bugs at earlier stages. This flows in with a process known as risk mitigation.

However, they don't necessarily make the development cycle take less time. When unit testing is done elaborately, including using a process like Test-Driven Development and automating these tests when making a build, they can be extremely beneficial. But, where time may be saved is often the deployment and QA teams, not necessarily the developers. Companies have to be willing to pay the extra cost for these tests in the name of higher quality, a smoother process, and to ease refactoring. Finally, unit tests are typically terrible at testing any non-deterministic process, especially when threads are involved. It is also generally difficult to test GUI functionality.

Integration Tests

I find these generally less useful, because this is the middle ground, as has enough overlap with unit tests and broader tests that they sometimes can be excluded. Still, in some scenarios they are quite useful. Just like in unit tests, integration tests tend to be automated, but this is not always the case. Where integration tests differ is that you generally don't stub out interfaces, unless it's an interface that is totally superfluous to the test or would prevent the test from running in an automated way. An example of a integration test is it might run the core engine of your application along with another component of the application, such as a logic module. The integration test would start by running a database query to reset the database to a known state, send login requests for 100 users, and verify that the 100 users logged in. It would then check the database and make sure the 100 records were properly set into the database. Then it would clean up the database.

Load Testing

I find that these are the tests I write towards the end of the cycle. You generally can get away without these if you're making a standalone application, but it is essential for a server application that expects a high user load. The load testing script, often written in a language simpler than your application, such as Python, will simply simulate a bunch of users repeatedly trying to simulate a variety of scenarios during the test. They often all try to simulate the same scenario, but sometimes they try to stimulate different areas of the system at the same time. Verification at this stage is generally minimal, because in most situations, the users can interfere with each other, creating false failures. What you are trying to verify here is longevity test to prove that the server can withstand a constant barrage for n hours with m users. Failure on this test will often be discovered manually when you come in to check it the next morning, and will be something of the nature of a crash, deadlock, excessive memory growth, performance degredation, etc... Smaller issues, such as 1 out of 1000 users not having their record stored in the database are unlikely to be discovered.

Conclusion

While this isn't all of the types of tests a developer will use, I find these to be the most frequent in my travels. One thing to keep in mind is setting expectations at the appropriate level. Developer-level tests are extremely valuable in making the developer's life easier and making life easier for your coworkers. The thing I want to keep emphasizing is that they don't necessarily save time and money, though. I give this caution, even though I'm a huge fan of software-level testing. You write tests for quality, period. Some hidden benefits are that they also provide additional "documentation", example usage and generally make refactoring easier, because if you have to fit your code into a test, it's more difficult to take shortcuts.

Overall, I have worked at both companies where quality was held highly and others where it wasn't the most important factor in the success of a product. You have to decide for yourself what's most appropriate for what you're doing. One thing that's important is setting expectations correctly. Sometimes management will think that if you write a test, the software will be bug-free. There isn't a single piece of non-trivial software that is bug-free, anywhere in the world.

Ada-like Range Validations

This post has been migrated to my new blog. Check it out here: Ada-Like Range Validations

What Is Up With Google Search?

I am getting seriously annoyed with Google search. When I first made this blog, for the first few weeks I varied between #1, #3 and #5 in Google's search if I searched for a term such as "C++ purist". Then, for no obvious reason, I completely dropped off the search. This includes if I googled the exact URL on this site. Nothing. That lasted for weeks upon weeks. Then, my blog miraculously returned and was consistently #3 or #5 in search. Now, when I check this week, it is again gone from search completely, and search terms which can make it arise seem random. For example, "The C++ Purist" only has three hits, none of them mine. However, ["The C++ Purist" Rydinare] (minus the []) will bring my blog up directly.

I can't tell if this is a serious flaw in Google's search algorithm or this is simply Google having some very obscure rules about search. Nonetheless, it's clear to me that a search for "C++ purist" and a search for "The C++ Purist" should have me in the list.

Anyway, for your entertainment, here's a few other interesting links I came across when searching for "C++ purist":

"I think moderation is a necessity also for those trying to protect it.It's basically this that I'm advocating actually. The hatred of C++/CLI comes from religious ferver of the C++ purist. Not to mention inborn hatred of microsoft"


"Any C++ purist who looks at a message map has an immediate question: Why didn't Microsoft use virtual functions instead? Virtual functions are the standard C++ way to handle what mesage maps are doing in MFC, so the use of rather bizarre macros like DECLARE_MESSAGE_MAP and BEGIN_MESSAGE_MAP seems like a hack."


">>OK, in this case it's more a matter of taste.
>>Simply, I'd choose Boost as I'm STL/C++ purist.
>>
>>But I don't agree that Boost would make code inconsistent.
>>If you think so, why don't you replace all STL's containers
>>with QVector and QList?"


"This is a good example of where a C++ purist will always use <<>>, but a pragmatist will just use printf()"


Hmm, interesting how the term "C++ purist" is thrown around. As a C++ purist, I agree with the purist views in the links I checked out. Except for the hatred of Microsoft comment. Nobody really hates Microsoft, they hate the fact that Microsoft always chooses to go its own way, instead of sharing to the benefit of everyone. If they shared, they would allow themselves to become the "standard" across the industry, instead of forcing others to come up with a different implementation. When they make improvements over what's generally out there, they should share, instead of boxing it in Windows. Furthermore, Microsoft has encouraged a lot of bad programming practices over the years, which were against some of the better judgment of the more "purist" community. Those are the reasons why purists are disenchanted with Microsoft.

OpenMP, Threading-Building Blocks, Others?

This post has been migrated to my new blog. Please check it out here: http://www.softwarepurist.com/blog/index.php/openmp-threading-building-blocks-others/