Tuesday, May 29, 2012

Automated tests, Type II errors, and the Null Hypothesis

In Part 1 I described how I use automated tests to fire up Selenium, hit a webpage, and log errors to ELMAH via jQuery. But with an elaborate test harness scenario like this, I argue that the onus should be squarely on the JS side to prove itself. I'll show you what I mean.
"Extraordinary claims require extraordinary evidence" - Carl Sagan
Flickr photo
FlickrCarl Sagan (Cosmos), by trackrecord. Carl Sagan was the man.

One of the advantages (as I see it) of the co-opting of ELMAH error logging into my automated tests is that errors end up where they should, in ELMAH. As in, in the "ELMAH_Error" table in your database, or whatever back-end storage option you choose. Because you are using ELMAH, right? (At this stage it probably looks as if I'm keyword spamming for "ELMAH". Oops, said ELMAH again.) Some might see this as noisy, an adulteration of the logging stream for your application's unhandled exceptions, but if I'm going to use my database to coordinate my automated test sessions, then the ELMAH table is the most natural place to put it. And if you don't want automated test kruft there, you can easily identify it and filter it out later. Or at the end of the test. Or hire a cleaner if you're too busy.


Automated test and JS errors go to ELMAH to die. Might wanna fix up those Google+ errors!

Pass the parcel

So how does the error get in there in the first place? Well, as I describe in part 1, the session id gets generated during the test, slapped on the end of the URL of the Frankenpage:

http://localhost/Services/Test?sessionId=11c4a798-1f75-4189-88a5-a08780a49487

The Services controller Test() method creates a ViewBag.SessionId from the modelbound querystring sessionId value. The Razor engine generates a meta tag
<meta name="sessionId" content="@ViewBag.SessionId">
which the jQuery can get:
var sessionId = $("meta[name=sessionId]").attr("content");
Of course, there's a few different ways you could do all this. Maybe a cookie. But I think this is the easiest.

The cons

A couple of times on outings with Tina, my wife, I've said something like "You go to that shop, I'll go to this one, and I'll text you if there's a change of plan, otherwise I'll give you a missed call, and we'll meet at such-and-such a place, and if that's closed, I'll text you where to meet." Too many moving parts. And we've come unstuck one or two times because of it. Phones and batteries don't always work. Same for (my) short-term memory.

This testing technique has, as it stands, too many moving parts. It relies on a few things working. For instance, it relies on the fact that the JS file is correct, that the controller and Razor between them are serving up the test page with the meta tag containing the session id, etc. To that end, you could assert that the page is serving in the first place by adding in a HttpClient check to see if all is OK, that is that the Http Response Code is 200, because if it doesn't respond with 200, I don't have a page.
void GivenIHaveAFrankenpageWithAllMyJavascriptServices()
{
    _sessionId = Guid.NewGuid().ToString();
    _servicesTestUrl = string.Format("http://localhost/Services/Test/{0}?sessionId={1}", environment, _sessionId);

    // serves OK in the first place?
    var client = new Microsoft.Http.HttpClient();
    var response = client.Get(_servicesTestUrl);
    Assert.AreEqual(HttpStatusCode.OK, response.StatusCode);
}
Fine, the page serves OK, but there's still plenty that could have gone wrong. Like the JS file didn't load. Or it did, but some bug happened and the testing didn't happen. So you get a false negative, a Type II error. Something went wrong, but you didn't spot it, and everything looked like it worked. Even though the thing that went wrong might have had nothing to do with the actual case you're trying to test, in an automated test, you need to know that the test is broken, or else it has no value whatsoever.

Switching the null hypothesis

Like I say in the first part of this two-part post, with JavaScript you want to test the crap out of it if it's critical to your app. So I need to shift the burden of proof to the JavaScript side. I need to reflect the fact that it is statistically more likely to fail than not, so it has to shoulder that burden. By engineering it so the initial arrangement of the test is now to create an ELMAH error, I am advancing a theory that the test will fail unless proven otherwise: the null hypothesis. This is completely different to the way is set up the test in the first blog post.

So, the null hypothesis (denoted H0 by statisticians, so now you know) is that the test will fail. Or rather, will stay failed, since I've changed the test to log an error as part of its setup, and the rest of the test must clear that error. This is because it is the simplest option: there are many ways for this flaky arrangement of test class/rigged-up HTML and Razor/JS/Web API and ELMAH to fail, and only one way for it to succeed. So I engineer the initial conditions to create an error, then challenge the rest of the test harness - the JS, HTML, ELMAH etc - to prove me wrong.
namespace ConnemaraComTests.Services
{
    public class BlogJavascript : Base
    {
        private string _servicesTestUrl, _sessionId;

        [Test]
        public void MyJavascriptServicesShouldBeWorkingOK()
        {
            this.Given(_ => GivenIHaveAFrankenpageWithAllMyJavascriptServices())
                    .And(_ => AndThereIsAnErrorTokenForThisTest())
                .When(_ => WhenIBrowseToThatPage())
                .Then(_ => ThenTheJavascriptShouldClearTheErrorToken())
                .Bddify();
        }

        void GivenIHaveAFrankenpageWithAllMyJavascriptServices()
        {
            _sessionId = Guid.NewGuid().ToString();
            _servicesTestUrl = string.Format("http://localhost/Services/Test/{0}?sessionId={1}", environment, _sessionId);
        }

        void AndThereIsAnErrorTokenForThisTest()
        {
            using (var context = new ConnemaraComContext())
            {
                var errorGuid = Guid.Parse(_sessionId);
                context.Errors.Add(new Error
                                       {
                                           Application = "/LM/W3SVC/1/ROOT",
                                           Host = "RAFTUS-PC",
                                           Message = _sessionId,
                                           Type = "Automated JS test",
                                           TimeUtc = DateTime.Now,
                                           ErrorId = errorGuid
                                       });
                context.SaveChanges();

                Assert.IsTrue(context.Errors.Count(x => x.ErrorId == errorGuid) == 1);;
            }
        }

        void WhenIBrowseToThatPage()
        {
            var driver = new FirefoxDriver();
            driver.Navigate().GoToUrl(_servicesTestUrl);

            // wait 5 seconds
            Thread.Sleep(5000);
        }

        void ThenTheJavascriptShouldClearTheErrorToken()
        {
            var errorId = Guid.Parse(_sessionId);
            using (var context = new ConnemaraComContext())
            {
                Assert.IsFalse(context.Errors.Any(x => x.ErrorId == errorId));
            }
        }

    }
}
Now, if it doesn't get to the JS, it can't clear the error, and the automated test fails. Or, it makes it to the JS, but something happens there. Or in the Web API controller. There is no chance of a false positive, a Type I error. There's simply no way an accidental success can fall through the cracks.

The JS file is just an instruction to run the tests, checks to see that the divs have loaded content from the public APIs they're supposed to, and then if all is well an instruction to delete the original error token.
window.onload = function () {
    ServicesLibrary.RunTests();
};

var ServicesLibrary = {

    ErrorMessage: "",

    RunTests: function () {
        var self = this;
        // wait for 5 seconds before trying anything
        setTimeout(function(){
            self.CheckContentHasLoaded({ key: "Twitter", elements: $("#Twitter span"), dataAttribute: "tweet-id" });
            
            ... and other services besides Twitter

            // if at the end of all this there's still no error message
            if (self.ErrorMessage == "")
                ServicesLibrary.DeleteError();
        }, 5000);
    },

    DeleteError: function(){
        var sessionId = $("meta[name=sessionId]").attr("content");
        $.ajax({
            type: 'DELETE',
            url: "/api/Errors/",
            data: { id: sessionId }
        });
    }
}
So easy a sea sponge could grok it. OK, so it's a little elaborate. But if there's any grokking to be done, let it be this. There's an automated test which sets up an initial failing condition which the JavaScript part of the test must pass. Because the null hypothesis must not stand.


Note the nice bddify report at the bottom of the test session window

Monday, May 21, 2012

Getting jQuery into your Automated tests

If you use Javascript in your app you want to be testing the crap out of it. But rather than relying on Selenium to implicitly assert that this jQuery method must have worked because that div is visible, for example, I prefer to invite jQuery in the front door, into the drawing room, stick a sherry in its hand and allow it to get stuck in like a man.

The problem

I have a bunch of Javascript/jQuery (I'll just say JS from now on) services that I wrote to get data from Flickr, Twitter, Google+, etc. I use them on this very blog, for example, to embed photos (like the Frankenstein picture below), tweets, even Stack Overflow questions. They work brilliantly, except in Google Reader which doesn't allow extra (to Reader's own) JS, but that's a whole 'nuther blog post.

And being composed of software, sometimes they break or stop working. The APIs from those huge web brands themselves are unlikely to be the cause of the problem. Rather, it's usually that I've got a bug in my JS. Yep. The scripts themselves go to work by finding a tag in the blog's HTML - like the tweet example below - using the tweet's id to fetch it via the Twitter API. Tweet received, the JS pushes the tweet's text into that [Tweet goes here] space. To paraphrase Cody Lindley's 'jQuery Enlightenment', jQuery is all about "Find stuff do stuff".
<span data-tweet-id="179313185281675264">[Tweet goes here]</span>
So obviously I should test this HTML/JS/API interface. The problem is that since my JS services are spread unevenly among my blog posts (some posts only need tweets embedded, some need photos and Google Books, and so on) writing a test to hit the actual posts themselves would be tricky to say the least.

Flickr photo
FlickrFrankenstein, by twm1340. 'Can you help me write some automated BDD tests, little girl?'

The test

So, like Victor Frankenstein, I decided to make something from disparate components as a sort of experiment in serving mankind. His effort was called "the monster", but mine can be called "an aggregation of my JS services into one master page". Another distinction between the two is that mine is unlikely to harm children. The idea is by loading all my scripts on this page, and reproducing my repertoire of HTML placeholders for photos, tweets, SO questions, etc. there, I can get a lot of testing bang for my buck.

That page has a custom JS file attached as part of the test harness, which is kind of the brains of the whole operation. Much like the "Frankenpage" itself, it gathers together the various JS methods required to convert the HTML to tweets, photos, etc. So, once you script up a simple test using Selenium to browse to the test page, the custom JS page executes too.

BDD - why not?

For the purposes of clarity, I've chosen to use the excellent bddify to run the actual test. I was unsure as to whether taking a behaviour driven design approach made any sense if I'm the only one working on some code: BDD is probably considered over-the-top for a one-man scenario. But as Mehdi Khalili, the creator of bddify, explained when I asked him about this, even just for your own sake BDD can be a help keeping tabs on what the purpose of your test is.

So here's my test so far: I create a session id (Guid) at the start of the test which gets passed via the URL of the test web page to the JS once the Selenium FireFox driver cranks up the browser. That JS file then does most of the actual testing - that's the crux of this blog post. If there's an error with any of the jQuery attempts to resolve a tweet, photo, whatnot, the jQuery writes the session id to an ELMAH error log to enable the assertion in ThenThereShouldNotBeAnyErrorsFromTheJavascript() via some Web API calls.
namespace ConnemaraComTests.Services
{
    [TestFixture]
    [Story(
    AsA = "As an blogger",
    IWant = "I want to embed tweets, photos, and other social media stuff",
    SoThat = "So that my blog posts come alive!")]
    public class BlogJavascriptBddify : Base
    {
        string _sessionId;

        [Test]
        public void MyJavascriptServicesShouldBeWorkingOK()
        {
            this.Given(_ => GivenIHaveAFrankenpageWithAllMyJavascriptServices())
                .When(_ => WhenIBrowseToThatPage())
                .Then(_ => ThenThereShouldNotBeAnyErrorsFromTheJavascript())
                .Bddify();
        }

        void GivenIHaveAFrankenpageWithAllMyJavascriptServices()
        {
            _sessionId = Guid.NewGuid().ToString();
        }

        void WhenIBrowseToThatPage()
        {
            var driver = new FirefoxDriver();
            var _servicesTestUrl = string.Format("http://localhost/Services/Test?sessionId={0}", _sessionId);
            driver.Navigate().GoToUrl(_servicesTestUrl);

            // wait 5 seconds for any tweets, photos, async stuff to load
            Thread.Sleep(5000);
        }

        void ThenThereShouldNotBeAnyErrorsFromTheJavascript()
        {
            using (var context = new ConnemaraComContext())
            {
                Assert.IsFalse(context.ELMAH_Errors.Any(x => x.Message.StartsWith(_sessionId)));
            }
        }
    }
}
If you're unfamiliar with BDD-style tests, the "Given" method is like the "Arrange" in a regular test, the "When" like the "Act", and the "Then" like "Assert".

As an aside, speaking of Selenium, the docs at Selenium HQ are actually written with a dose of wit. Read this, from 'Brief History of The Selenium Project':
The Beijing Olympics mark China’s arrival as a global power, massive mortgage default in the United States triggers the worst international recession since the Great Depression, The Dark Knight is viewed by every human (twice), still reeling from the untimely loss of Heath Ledger. But the most important story of that year was the merging of Selenium and WebDriver...

Pros and cons

There are definitely pros and cons in this Frankenpage approach I'm outlining here, I'm aware of that. And doubtless it's nothing new. I'm not so experienced in automated testing, but I can't see anything glaringly wrong with it so far. In any case, I'm going to continue this in a second post because there's still a bit of detail to talk about, and no-one wants to read a crazy long post.

Saturday, May 19, 2012

Developer Notes from Google Plus #1

Aggregated from my Google+ posts under the hashtag #DailyDevNotes for your viewing convenience, dear reader. And for the sake of fleshing out my blog, it should also be noted. Notes from 31st March to 19 May, 2012.

Flickr photo
FlickrHalfpenny bridge, Dublin, Ireland, by braulio.mora. See, I'm building a bridge between my Google Plus posts, and my Blogger posts. Ah, the Ha'penny Bridge: many's the use I've gotten out of it.



Tuesday, May 1, 2012

Entity Framework Migrations strategies: the Semi-Automatic

Migrations is a great complement to Code First. Its flexibility means though that you have to understand it pretty well to pick a strategy. I call the one I talk about here the semi-automatic

Other Entity Framework posts:

I recently did a 4-month contract in a government department where management had allowed a schism to develop between the "developers", of which I was one, and the "database/build" team. The secession of the database guy and the TFS build guy was sufficiently complete that despite being in the same open office as the rest of us they had better things to do every morning than attend the morning stand-up. The idea, one short year later, that our whole workflow was hostage to others' recalcitrance simply because we needed to change, for instance, the datatype of a single table column now slays me. Talk about stumbling around in the dark.


That was then, this is now

So now we have Code First. And since February we have Migrations. I've been kind of fascinated by Code First for a while now, in awe of its potential to completely change the way you work in a team where there's often a "database guy". And in my experience, there usually is. A cranky database guy. Which makes devs cranky in turn. Those guys should now be feeling the way supermarket checkout staff feel, as they glumly contemplate the self-service machines savvy shoppers use. Gaze In Terror at EF Code First, fools, especially - since EF 4.3 shipped - as used in combination with EF Migrations!

EF Migrations implies that you have an existing database. Or that you have a code-first database that has been deployed to production. In any case, it implies a scenario where you can't just use a DropCreateDatabase- or DropCreateTables-type of initialization strategy anymore.

When the first two EF team blogs on Migrations came out, it looked as if there were two distinct approaches you could take: automatic, and code-based. Choice, hurray! So, which one should you use? Why wouldn't you just go automatic? Let Migrations work out the difference between your code and the database and patch everything up?

Well, the problem with automatic migrations, as this ElegantCode blog post suggests, is that there's no record of what just happened, no way back, no direction home, like a rolling stone. Code-based Migrations gives you a way to go up and come back down again. When you look at point 8 in that post ("Tear all the way down, and then back up") where he runs through the sequence of migrations from the initial one to the most recent, and then reverses the order, you have to admit - that's a well-oiled machine right there. So I'm going with code-based.

The strategy

It means though that every time you change your entities, for instance when adding a property:
public class User : Entity
{
    public string FirstName { get; set; }
    public string LastName { get; set; }
    public string MiddleName { get; set; }
}
you have to manually create a migration:
Add-Migration AddUserMiddleName
When you do that, you get a configuration file ("/Migrations/201204280429379_AddUserMiddleName.cs" or so). When you then apply that change to your database:
Update-Database
you get a record of that in __MigrationHistory. If you don't opt for the 'add-migration' step, you still get the __MigrationHistory record, only with an effectively anonymous MigrationId, such as "201204280429379_AutomaticUpdate".

You don't have to scaffold up an explicit, code-based migration for every trifling model change you make: you could aggregate a bunch of them under a task name 'Add-Migration "ChangeLoginToAllowOpenID"' for example. But if you want to develop your app with incremental model changes, which, let's face it, is hard to do any other way, you'll find yourself doing the "Add-Migration...Update-Database" dance quite a lot. But seriously - so what? You end up doing so many menial web development housekeeping tasks all the time anyway, from periodically getting latest from TFS and merging, to cleaning up code with ReSharper, that ... you know what? Just do it.

You could always unpack the individual migration files and aggregate all the changes into one migration file, but I wouldn't really recommend that because
  1. You're not that smart
  2. You're going to have to manually trash records in __MigrationHistory
  3. Just stick them in a folder called "ChangeLoginToAllowOpenID" within Migrations. "Update-Database" ignores folders

The "__MigrationHistory" table

By the way, if you allow Code First to whip up a SQL Express database for you, and you use Visual Studio, you may be perplexed to find that you can't see a __MigrationHistory table in your database, and are wondering what everyone is talking about. As we used to say in Ireland: "I came out of the pub and there was me bike - gone!" Follow the steps in this post - Making __MigrationHistory not a system table - by Arthur Vickers, one of the EF team to fix that. (You might want to change the last statement of his SQL batch to "EXEC sp_rename [TempMigrationHistory], [__MigrationHistory]", as I mention in a comment on that post). Or you can just create a View by selecting from the table:
SELECT * 
  FROM __MigrationHistory
because it's still there, lurking, even though you can't see it. You want to be able to see this table: I found it instructive to refresh the contents of it after every Migration-based activity just to see how it all wires up. If you can't see the records going in one-by-one you're missing a vital part of the Migrations picture.

Getting it into production, automatically

So far I've looked at precisely half of the picture: the local half. The near side of the moon. A chap's local dev environment. I found there to be a bit of a dearth of supporting information advising one how to propagate local changes to a live database, which is after all the whole point - to get them where people can use them. One way suggested in the EF team blog is to generate a script:
If another developer wants these changes on their machine they can just sync once we check our changes into source control. Once they have our new migrations they can just run the Update-Database command to have the changes applied locally. However if we want to push these changes out to a test server, and eventually production, we probably want a SQL script we can hand off to our DBA.
Yeah, I don't know that that's that efficient. Surely a better way is to have the code-based migrations run automatically, if that's not too much of a contradiction, when the app runs. By using a DbMigrator. Of course, you have to not have someone exercising a database veto, like a hypothetical cranky DBA as mentioned above.
public class Migrator
{
    public static void RunMigrations()
    {
        var migrator = new Configuration();
        var dbMigrator = new System.Data.Entity.Migrations.DbMigrator(migrator);
        dbMigrator.Update();
    }
}
I call this from the Global.asax
protected void Application_Start()
{
    ...other stuff

    // update the database with any pending migrations
    Migrator.RunMigrations();
}

It doesn't really matter where you put this Migrator class...except I wouldn't put it in the Configurations folder, which is a generated folder (generated when you 'Enable-Migrations') and therefore you may regret placing any custom code there. As a general rule of thumb, generated folders - like the novels of Jeffrey Archer - should be left well enough alone.

Backlinks

Thanks to Tony Pierascenzi for the http://www.scoop.it link to this post!