Thursday 24 May 2012

How Google Tests Software

 Google's approach to testing software







Google, who's user base has been estimated as high as 1.5 billion accross it's suite of products by some, and who's products dominate the internet, recently released a new book "How Google Tests Software" revealing exactly how Google tests software, offering brand-new best practices.

Authors James Whittaker, Jason Arbon and Jeff Carollo offer the folowing insights on testing at Google:

On how testing at Google is different from Microsoft:

"At Google, testing is owned by everyone on the development team. Developers have to take responsibility for their unit tests and testers take responsibility for making developers productive testers. It’s not a dev-test model; everyone is a tester at Google."

On testing Web based Software Vs. All Other Types the Authors point out that all software is created equal, accepting input, producint output, storing data, and doing computation. Google utiliazes crowd source testing to great effect, at the end of the day the same problems are universal in testing.

According to the book Google beleives that testers are much more effective when also being able to code, expanding their impact. Another huge part of testing at Google involves having the developers perform self-service testing (described in the book).

As far as security is concerned, the Authors reflect that  horizontals like security apply everywhere, but require special knowledge and skillset.

Google Tests Software is great book which will benifit Automated Testers, Dev, and Program Management, contains real code, and talks about real automation efforts. The book is available from Amazon Here.



Tuesday 6 December 2011

Running Tests from Visual Studio vs. A Remote Computer

 Here’s a tip for running tests that will free up your machine from running tests out of Visual Studio.

Test Controller / Agent Setup

First you’ll need a free computer or virtual machine to install the MS Test Controller and Test Agent software on.

Here are the settings I’ve used for the Test Controller and Test Agent(I’ve registered the Test Agent against the Test Controller which is on the same machine):




 

 
















Visual Studio Setup

Now in Visual Studio you will need to configure alternative test settings to use for running tests remotely. To edit test settings go to Test >> Edit Test Settings >> Trace and Test Impact. You can toggle between test settings used by going to Test >> Select Active Test Settings in Visual Studio.  Under Roles add the name of the machine you’ve installed the Test Controller on, in my example I’ve used my QA-Test machine:

 

And that’s it, you’re now free of sitting and watching tests run.

Friday 2 September 2011

CodedUI Testing Automation - Assertions for Values that Change

I've found a few situations during testing where I have the need to place assertions on values that are always changing, for example during one of my tests I sell inventory products and need to verify that the quantity in the inventory reduces after the sale is processed. This is pretty easy to do, first I open my inventory item and get the quantity value from a display text property, then convert it to an integer.

UIMap.OpenInventoryModule();
UIMap.OpenInventoryItem();
int origSkuQty = Convert.ToInt32(UIMap.UIWindow.UIInStockCustom.UIItemText.DisplayText);

For this to work i've dropped the CodedUI assertion picker on the DisplayText property of my inventory item and added this to my UIMap to call during the test.

After my sale I perform the same inventory check and verify that the DisplayText int is now the origSkuQty - 1

Thursday 12 May 2011

Tool - Inspect Objects

I've been using UISpy for some time to check the properties of my app under test, Inspect Objects is a newer tool and is able to find some controls not available in UISpy.

Inspect Objects is part of the Windows SDK and available here:



More on Inspect Objects from Microsoft:

http://msdn.microsoft.com/en-us/library/ms696079%28v=vs.85%29.aspx

Monday 9 May 2011

Microsoft CodedUI Tip - Kill Unwanted Processes During MyTestInitialize

Here's a quick way to kill any unwanted processes that may be running before your test starts, in my case I'm targeting any notepad applications that may be open.

MyTestInitialize allows you to run some code before the test starts, from MyTestInitialize i call my process kill.

     public void MyTestInitialize()  
     {  
       // To generate code for this test, select "Generate Code for Coded UI Test" from the shortcut menu and select one of the menu items.  
       // For more information on generated code, see http://go.microsoft.com/fwlink/?LinkId=179463  
       TerminateProcess.YerOuttaHere();  
     }  

The kill process uses System.Diagnostics and kills any notepad process.


 using System;  
 using System.Threading;  
 using System.Diagnostics;  
 public class TerminateProcess  
 {  
   public static void YerOuttaHere()  
   {  
     Process[] processes = Process.GetProcessesByName("notepad");  
     foreach (Process process in processes)  
     {  
       process.Kill();  
       process.WaitForExit();  
     }  
   }  
 }  

Friday 6 May 2011

The Perfect UI Automation Testing Software Testing Process

 

Recording a test

I setup my test recorder and open my application, for my test i click through my application, open a report and view some data, I make a change and close out, I'm prompted for a name for the test and save.


Tweaking the Recording

The video step editor window opens, i set a few quick ignore zones in my app for dynamic items and save. I check the performance baselines and set my thresholds for when the test should fail. I set global change ignore zones for my application. I can also reverse my assertions and only select certain places in the video i would like to monitor through the test runs. The Software learns what to focus on.


Test Steps

A list of test steps are auto generated based on previous test steps I've setup using the smart match test step feature and my test case is complete.


Running Tests

I select my tests and hit run on new build, with every new build my app auto installs and test cases run.


Error Reporting

When a mismatch is found in my test the test fails generating a report containing the video section that no longer matches, I can choose to update my test to the new visual or report the bug.


Future Test Creation is Automated

As the testing software learns my testing habits it develops new tests and new combinations of tests based on what I’ve set it to focus on, tests are automatically adapted as the software changes based on my approval. I sit back and monitor, all of the new testing, test runs, and issue tracking are handled by the intelligent software. I send emails from the beach.

Assert two screenshots hack in Visual Studio Test

Here’s a workaround I came up with when trying to find a way to have visual comparisons during some of my tests. I’m using the free screenshot program Greenshot available from here to take care of my screenshots.

I set up Greenshot to dump a full screenshot automatically to a folder on each of my test machines without opening the image, here’s the settings I used:

GreenShot Settings1GreenShot Settings2

During the test run Greenshot will snap a full desktop image and dump it to a directory on the test machine, I’ll later crop the image during the test and compare. 


Comparing screenshots

Inside of the test file I have 2 Bitmap images, 1.bmp which is my baseline image and the dynamic screenshot(2.bmp) of my application taken during the test(this screenshot is also taken during the test using the Greenshot hotkey control+Print Screen in my app window).


CheckImages.cs file:

 //images to compare after taking screenshot  
 //baseline  
 Bitmap bmp1 = new Bitmap(@"C:\Dev\1.bmp");  
 //new screenshot  
 Bitmap bmp2 = new Bitmap(@"C:\Dev\2.bmp");  
 //crop the image x, y, width, height and then compare  
 Bitmap cropbmp1 = ComparingImages.CropBitmap(bmp1, 220, 200, 600, 200);  
 Bitmap cropbmp2 = ComparingImages.CropBitmap(bmp2, 220, 200, 600, 200);  
 ComparingImages.Compare(cropbmp1, cropbmp2);  



Cropping The Image

I found a simple function to crop the image, In my case I’m interested in a screenshot of a grid full of data.



 public Bitmap CropBitmap(Bitmap bitmap, int cropX, int cropY, int cropWidth, int cropHeight)  
 {  
 Rectangle rect = new Rectangle(cropX, cropY, cropWidth, cropHeight);  
 Bitmap cropped = bitmap.Clone(rect, bitmap.PixelFormat);  
 return cropped;  
 }  


Comparing The Hash

I found some code here which works perfectly for taking the two images and comparing the Hash of each, you can read more on the details of how it works here, I’m just happy that it works.


CompareHash.cs file:
 using System;  
 using System.Collections.Generic;  
 using System.Linq;  
 using System.Text;  
 using System.Drawing;  
 using System.Security.Cryptography;  
 using Microsoft.VisualStudio.TestTools.UITesting;  
 using Microsoft.VisualStudio.TestTools.UnitTesting;  
 using Microsoft.VisualStudio.TestTools.UITest.Extension;  
 using Keyboard = Microsoft.VisualStudio.TestTools.UITesting.Keyboard;  
 namespace CoreTestAutomation.Extended_Tests.Screen_Checks  
 {  
   public class ComparingImages  
   {  
     public static Bitmap CropBitmap(Bitmap bitmap, int cropX, int cropY, int cropWidth, int cropHeight)  
     {  
       Rectangle rect = new Rectangle(cropX, cropY, cropWidth, cropHeight);  
       Bitmap cropped = bitmap.Clone(rect, bitmap.PixelFormat);  
       return cropped;  
     }  
     public enum CompareResult  
     {  
       ciCompareOk,  
       ciPixelMismatch,  
       ciSizeMismatch  
     };  
     public static CompareResult Compare(Bitmap bmp1, Bitmap bmp2)  
     {  
       CompareResult cr = CompareResult.ciCompareOk;  
       CompareResult crmatch = CompareResult.ciCompareOk;  
         //Convert eachciCompareOk image to a byte array  
         System.Drawing.ImageConverter ic = new System.Drawing.ImageConverter();  
         byte[] btImage1 = new byte[1];  
         btImage1 = (byte[])ic.ConvertTo(bmp1, btImage1.GetType());  
         byte[] btImage2 = new byte[1];  
         btImage2 = (byte[])ic.ConvertTo(bmp2, btImage2.GetType());  
         //Compute a hash for each image  
         SHA256Managed shaM = new SHA256Managed();  
         byte[] hash1 = shaM.ComputeHash(btImage1);  
         byte[] hash2 = shaM.ComputeHash(btImage2);  
         //Compare the hash values  
         for (int i = 0; i < hash1.Length && i < hash2.Length  
                  && cr == CompareResult.ciCompareOk; i++)  
         {  
           if (hash1[i] != hash2[i])  
             cr = CompareResult.ciPixelMismatch;  
         }  
         Assert.AreEqual(crmatch, cr);  
         return cr;  
     }  
   }  
 }  


That's it, now i have a quick workaround to comparing some visual aspects of my application under test.