Aug 112015
 

If there is one place where developers feel least guilty about allocating large amounts of memory, it’s in the local variables of the method. After all, local variables are short lived and as the method execution is over, the call stack is wound down and the return value is popped. This frees all of them for garbage  collection. This developer assumption might hold true for most methods, but not for all.

Let’s consider a simple method first. It allocates a large  1 million element long integer array and  returns the length of the array.

    class Program
    {
        static void Main(string[] args)
        {
            var a = new Foo();
            Console.WriteLine(a.Bar());
        }
    }
    class Foo
    {
        public int Bar()
        {
            var arr = Enumerable.Repeat(1, 1000000).ToArray();
            return arr.Length;
        }
    }

Let’s compile the program and open it in WinDbg. The commands for doing that are

  • .symfix (fixes the symbol path)
  • sxe ld:clrjit.dll (telling the code to break when clrjit is loaded)
  • g (continuing execution till the clrdll is loaded)
  • .loadby sos clr (This loads the SOS managed debugging extension)
  • !bpmd (Break when Program.main is executed)

Looking at the IL code of the Foo.Bar method, it’s pretty straightforward. A 1 million element array is created and then the ldloc.0 instruction loads the local variable on the stack. After the method returns, the local variable pointer no longer exists and the garbage collector is free to reclaim the memory for other objects.

Foo.Bar

This works quite well, but imagine a scenario where you might need access to the local variable even after the method execution is over.  One scenario is when the method returns  a Func delegate instead of an integer.

class Foo
{
    public Func<int> BarFunc()
    {
        var arr = Enumerable.Repeat(1, 1000000).ToArray();
        return () => arr.Length;
    }
}

Though this method will always return the same result as the previous method, the CLR cannot make that assumption and mark the integer array for collection. Because there is no guarantee that the returned delegate will be executed immediately or even just once, the CLR has to maintain a reference to the local variable even after the method execution is completed. The compiler resolves this dilemma by promoting the local variable on to the heap as a field of an autogenerated Type. Let’s see the IL generated when this new method is called.

Foo.BarFunc

 

This IL is considerably different. The newobj instruction creates an object of a new Type c__DisplayClass1 which we never created. That is the type which the compiler autogenerated and used for storing the local variable. Since the new type lives on the heap it’s lifetime is guaranteed till the return delegate’s reference is held on by the calling method.  We can verify this by examining the managed Heap

NewAutogeneratedType

 

…and the object of the autogenerated type shows our local variable now as a field.

DumpObject

 

If we modify our main method a bit and store the resulting delegate into a class level field , we can see that the GC maintains an explicit root to the object. In essence the object lives till the application execution is completed. This is unnecessary memory usage by the application.

class Program
{
    private static Func<int> classLevelVariable;

    static void Main(string[] args)
    {
        var a = new Foo();
        classLevelVariable = a.BarFunc();
        Console.ReadLine();
        classLevelVariable();
    }
}

Finding GCRoots for the object, we see that Garbage collector can never collect this object.

gcroots

This particular scenario might seem trivial, but in a LINQ-heavy production application it is very easy to lose track of the methods that are creating closures.  Awareness about the promotion of local variables can help prevent memory leaks and improve application performance.

 

Aug 072015
 

Microsoft recently open sourced the CLR and the framework libraries and published them on Github. Though a non production version has been open sourced for a long time under the name Rotor or SSCLI, this time there were no half measures. It gives the community the opportunity to raise issues and also fix them by creating pull requests.

The journey from source to executable code has two phases – first the compiler compiles the source code into the Intermediate Language (MSIL) and then the execution engine (CLR) converts the IL to machine specific assembly instructions. This allows .NET code to be executable across platforms and also be language agnostic as the runtime only understands MSIL.

When the program is executed, the CLR reads the type information from the assembly and creates in-memory structures to represent them. The main structures that represent a type at runtime are the MethodTable and the EEClass. The MethodTable contains “hot” data which is frequently accessed by the runtime to resolve method calls and for garbage collection. The EEClass on the other hand is a cold structure which has detailed structural information about the type including its fields and methods. This is used in Reflection. The main reason for splitting these structures is to optimize performance and keep the frequently accessed fields in as small a data structure  as possible. Every non-generic type has its own copy of the MethodTable and the EEClass, and the pointer to the MethodTable is stored in the first memory address location of each object. We can observe this by loading the SOS managed debugging extension in WinDbg

methodtablelayout

 

The DumpHeap command gives us the information of our type along with the the addresses of all the objects for the type. Using the WinDbg command dq to read the address at the memory address we see that the first memory address points to its MethodTable. There is another structure called the SyncBlock which exists at a negative offset to the MethodTable in the memory. This structure handles the thread synchronization information for the object.

This diagram from the SSCLI Essentials Book explains the relationship between various data structures very clearly.

objectlayout

As you can see the object header points to the MethodTable which in turns point to the EEClassSince the EEClass is not frequently used during runtime, this extra level of indirection doesn’t hurt performance. The MethodTable itself is followed by a call table – a table which contains the addresses of the virtual and non virtual methods to be executed for the type. Since the dispatch table is laid out at a fixed offset from the MethodTablethere is no pointer indirection to access the right method to call. One more thing to be noted about the CLR is that everything is loaded only when it’s needed to be executed. This holds true for both types and methods. When the CLR executes a method which creates another type, it creates the memory structures for the new type. However, even then the methods themselves are not compiled till the absolute last moment when they are needed to be executed.

In the above diagram, you can see the MethodTable vtable pointing to a thunk, which is called a prestub in .NET. When the method is first called, the prestub calls the JIT compiler. The JIT compiler is responsible for reading the MSIL opcode and generating the processor specific assembly code. Once the JIT Compilation is done, the address at which the compiled code resides is backpatched on to the call table. Subsequent calls to the method are directly executed without having to go through the compilation phase

Loading the MethodTable for our calculator type using the command DumpMT with the MD switch which also loads the MethodDescriptors.

methodtable

At this stage in the application execution, the object for Calculator class has been created but the AddTwoNumbers method hasn’t been executed yet. So the MethodDesc table shows that only the constructor method has been jitted but not the AddTwoNumbers method.  Seeing the MethodDescriptors for both the methods using the command !DumpMD

MethodDescriptors

 

The Constructor method now contains a code address, but the AddTwoNumbers doesn’t have code yet. Let’s step forward and see what happens after the method is jitted. Now the code address is replaced by an actual memory address which contains our machine specific assembly code. The next time this method is called, this assembly code will be directly executed.

afterjitting

To view the assembly, use the !u command followed by the code address.  Like in most languages, there are two registers ebp and esp to keep track of each stackframe. During a method call a new stackframe is created and the ebp maintains a pointer to the base of the stack. As code executes the esp register keeps track of how the stack grows and once execution completes, the stack is cleared and the epb value is popped.

assemblyccode

 

Now lets look at this from a code level. Detailed building and debugging instructions are given at the coreclr repo. The MethodTableBuilder class contains the method which loads the types. You could put a breakpoint here but it will keep breaking when system types are loading. To avoid this , put a breakpoint in the RunMain method in assembly.cpp class, and once it breaks then put the breakpoint in the CreateTypeHandle method. This will start breaking on your custom type creation.

createtypehandle

Below is the simple Calculator class code that we are calling. I just used the name of the executable as a Command Argument to run CoreRun.exe in the coreclr solution (Detailed instructions given in Github repo)

DebugCode

 

Now for the fun part – we start debugging the solution. The first step (after loading allocators) is to make sure all parent types are loaded. Since our type doesn’t inherit any class, its parent is System.Object. Once the Parent type is found (can’t be an interface, only a concrete type), it’s method table is returned to the MethodTableBuilder

loadparenttype

 

Then there are some additional checks to handle cases like enums, Generic method, excplicit layouts etc. I’ll skip over them for brevity. At this time we have started to build the MethodTable but not the EEClass. That is done in the next step.

eeclass

 

At this  stage, the CLR checks if the type implements any interfaces. Since interface calls are a bit more complex, there needs to be a relationship from the interface vtable to the implementing type, the calls are mapped using a slot map maintained on the implementing type’s MethodTable which maps it to the vtable slot on the interface. Since our Calculator Class doesn’t inherit interfaces, it will totally skip this block.

interfaces

Now we go into the final and most crucial method which will finally return the TypeHandle. If this method succeeds, then our type has been successfully loaded into memory.

bmtfinalmethod

The first thing the BuildMethodTableThrowing class does is to walk up the inheritance hierarchy and load the parent type. This holds for all methods except interfaces. An interface’s vtable will not contain the System.Object’s method calls. So the method builder will simply set the parent Type to null if the type being loaded is an interface.

interfaceInBuilder

After this, the method makes sure the type in question is not a value type, enum, remoting type, or being called via COM Interop. All this would be loaded differently then simple reference types deriving directly from System.Object. Then the MethodImpl attributes are checked since they impact how a type a loaded. Our Calculator class just skips over these checks. The next method is EnumerateClassMethods which iterates through all the methods and adds them to the MethodTable.

Now that the implemented methods are added to the MethodTable, we need to also add the parent type’s method calls to the current vtable. this is done by the methods ImportParentMethods, AllocateWorkingSlotTables and CopyParentVtable in the MethodBuilder class. Here virtual methods have to be handled differently since they can be overridden by the current type. For non virtual methods, a direct entry to the methods implemented by the Parent type should suffice.

First the maximum possible vtable size is computed. Next a temporary table is allocated for the maximum possible size of the table

maxVTableSize

Then the parent vTable methods are loaded to the Calculator type.

CopyParentVtable

After the Parent methods are added, the current type’s methods are added. We just have two methods – the Constructor and the AddTwoNumbers method. Here first the Virtual Methods are added and then the Non-Virtual ones. Since we didn’t define a custom constructor, it will just inherit the Default constructor and add it in the vtable. Once all virtual methods are added, the remaining methods will get the non vtable slots.

constructorCurrentType

Now that the type’s methods have been completely been loaded, the MethodDescriptors are  created. However the code for the methods is not called even once so it will simply be pointing to a stub waiting to be JIT compiled on execution. After this stage the remaining fields are placed in the MethodTable and some additional integrity checks are done. Finally the Type is loaded and is ready to be published

finalMethodTable

 

Jul 122012
 

Oh how I wish there was a out of the box solution to do this where you could just enter the source and destination urls and voila! the entire blog was migrated. Unfortunately there isnt 🙁 (or atleast I am not aware of one).

So when I set out to migrate around a 1000 posts from 16 different Sharepoint blogs to WordPress, there were quite a few challenges on the way. The first step is to read the Sharepoint data.This would have been far easier with the proper database access to sharepoint db but unfortunately there was way too much red tape wrapped around it to get the access quick enough to start immediately, hence I took the dirty way out – to scrape the HTML pages and retrieving the data out of it. Scraping HTML is the usually the worst possible way to extract data from any source since the code is rarely ever reusable, bloated to handle nested tags and often filled with branching statements to handle special cases. Only go for the scraping approach if you don’t have any other way to access the data. What makes scraping bearable is the wonderful HTML Agility pack which I had blogged about earlier. Its XMLish approach to traverse HTML makes this activity quite easy.

This is the object that I used to represent each blog to be imported. It contains the source URL, destination blog, author Id. Each object represents the source site as well as the destination information needed to write to WordPress.

   class BlogsToBeMigrated
    {
        public string Url { get; set; }
        public int SiteID { get; set; }
        public int AuthorId { get; set; }
        public string DestinationBlogName { get; set; }
        public BlogsToBeMigrated(string url, int siteId, int authorID,string destination)
        {
            Url = url;
            SiteID = siteId;
            AuthorId = authorID;
            DestinationBlogName = destination;
        }
    }

This is the code to read each blog entry. Note that it takes in each blog object and extracts information out of it. The concatenation step would depend on how your sharepoint site is structured. Just make sure it points to the list view of the blog where all posts are listed in a tabular form. This helps us to take out each link and get the content to add in WordPress. To get the various xpath to navigate to the nodes, I used the chrome extension Xpath helper. This could be different for every sharepoint site. Just play around with the xpath till you get the required information

private static void MigrateSingleBlog(BlogsToBeMigrated blog)
{
    var siteUrl = String.Concat("YOUR SITE URL HERE", blog.Url, "/Lists/Posts/AllPosts.aspx");
    string siteList = ReadWebPage(siteUrl);
    DeletedBlogData(blog.SiteID);
    var listDoc = new HtmlDocument();
    listDoc.LoadHtml(siteList);
    var siteListNodes = listDoc.DocumentNode.SelectNodes("//td[@class='ms-vb-title']/table/tr/td/a");

    foreach (var site in siteListNodes)
    {
        var postUrl = String.Concat("YOUR SITE URL HERE", site.GetAttributeValue("href", "href"));
        var PageDump = ReadWebPage(postUrl);
        var postDoc = new HtmlDocument();
        postDoc.LoadHtml(PageDump);
        var siteContent = postDoc.DocumentNode.SelectSingleNode("//div[@class='ms-PostWrapper']");

        string postDate = siteContent.ChildNodes[0].InnerText;
        string postTitle = siteContent.ChildNodes[1].ChildNodes[0].ChildNodes[0].InnerText.Trim();
        var postContent = postDoc.DocumentNode.SelectSingleNode("//div[@class='ms-PostWrapper']/div[@class='ms-PostBody']/div");

        var postHtml = postContent.InnerHtml;

        MoveToWordPress(postDate, postTitle, postHtml, blog.Url,blog.SiteID,blog.AuthorId,blog.DestinationBlogName);
        Console.WriteLine(postDate + postTitle);
    }

}

Also remember to put this line in the constructor of your class before any of the HTML agility pack code is executed. This is needed because forms can be tricky elements in HTML due to their overlapping between tags which makes it difficult to parse the markup. This makes HTML Agility pack parse form tags as empty elements and the below line allows you to look inside them.

HtmlNode.ElementsFlags.Remove("form");

Now that I had access to all the sharepoint data the challenge was to enter this is in the right WordPress blogs. I took a look at CSV importer which allows bulk import of posts from CSV files. This step didnt work properly at all since the post content was way too large for a CSV file to be parsed properly. After numerous attempts to sanitize the CSV and escape each linebreak and comma, this step still ignored many valid posts and also filled gibberish in others. Then I thought of directly inserting the data in the WordPress database. Initially I was skeptical since wordpress might fill some related tables when a post was published, but found that there were no such problems. Directly inserting the data worked like a charm. It also allowed me to migrate the data multiple times each time I noticed an issue with improperly rendered markup

This is the method that enters in the WordPress blog table directly. Note that the multisite installation means different post tables which have a number in the table name. e.g. WP_2_posts, WP_3_posts etc. The index for the table name was in the BlogToBeMigrated object as I had manually created each wordpress blog which corresponded to a sharepoint blog. This step is fairly simple. It just creates a connection to the MySql database using the connector dlls, gets the maximum ID, increments it and uses that to insert a new post. The code isnt production standard but this isnt really something that I am looking to maintain for a long time. Till the migration is done right, we can just keep repeating with the required fixes and once its finished – you have a functioning site with no need to migrate anymore. Pragmatism wins.

private static void MoveToWordPress(string postDate,string postTitle,string postContent,string postUrl, int blogID, int authorID,string destinationBlogName)
{
   //Remember to include ConvertZeroDateTime=true in the connection string
    MySqlConnection wordpressConn = new MySqlConnection("DATABASE_CONNECTION STRING; ConvertZeroDateTime=true");

    using (wordpressConn)
    {
        wordpressConn.Open();
        int maxID = 0;
        var id = new MySqlCommand(String.Format("Select Max(ID) from WP_{0}_Posts", blogID), wordpressConn).ExecuteScalar();
        if (id.GetType() == typeof(System.DBNull))
            maxID = 1;
        else
            maxID = Convert.ToInt32(id);

        string SQLCommandText = "Insert into wp_{0}_posts (id,post_author,post_date,post_content,post_title,post_excerpt,post_status,comment_status,ping_status,post_name,to_ping,pinged,post_modified,post_modified_gmt,post_content_filtered,post_parent,guid,menu_order,post_type,comment_count)";
        SQLCommandText += " values (?ID,{1},?postDate,?postBody,?postTitle,'','publish','open','open',?postName,'','',?postDate,?postDate,'',0,?postGuid,0,'post',0)";
        HtmlDocument span = new HtmlDocument();

        MySqlCommand insertPost = new MySqlCommand(String.Format(SQLCommandText, blogID, authorID), wordpressConn);
        span.LoadHtml(postContent);

        var siteName = "YOUR_BLOG_URL_HERE/{0}/files/{1}";
        var imagenodes = span.DocumentNode.SelectNodes("//a/img");
        if (imagenodes != null)
        {
            foreach (var image in imagenodes)
            {
                var imageUrl = image.ParentNode.GetAttributeValue("href", "href");
                var imageThumbUrl = image.GetAttributeValue("src", "src");
                if (imageUrl.Contains("/sites"))
                {
                    var migratedFileName = imageUrl.Replace(String.Concat(postUrl, "/Lists/Posts/Attachments/"), string.Empty);
                    postContent = postContent.Replace(string.Format("href=\"{0}", imageUrl), String.Format("href=\"{0}", String.Format(siteName, destinationBlogName, migratedFileName)));
                           
                    postContent = postContent.Replace(string.Format("src=\"{0}", imageThumbUrl), String.Format("src=\"{0}", String.Format(siteName, destinationBlogName, migratedFileName)));
                }
                else
                    postContent = postContent.Replace(string.Format("src=\"{0}", imageThumbUrl), String.Format("src=\"{0}", imageUrl));
            }
        }
            span = new HtmlDocument();
            span.LoadHtml(postContent);
            var htmlNode = span.DocumentNode.SelectSingleNode("//span[@class='erte_embed']");
            if (htmlNode != null)
            {
                string urlValue = htmlNode.GetAttributeValue("id", "id");
                urlValue = HttpUtility.UrlDecode(urlValue);
                //   urlValue = urlValue;
                postContent = postContent.Replace(htmlNode.OuterHtml, urlValue);
            }

            maxID++;
            insertPost.Parameters.AddWithValue("?ID", maxID);
            insertPost.Parameters.AddWithValue("?postDate", DateTime.Parse(postDate));
            insertPost.Parameters.AddWithValue("?postBody", postContent);
            insertPost.Parameters.AddWithValue("?postTitle", postTitle);
            insertPost.Parameters.AddWithValue("?postName", postTitle.Replace('#', '-').Replace(' ', '-'));
            insertPost.Parameters.AddWithValue("?postGuid", "YOUR_BLOG_URL_HERE/?p=" + maxID);
            insertPost.ExecuteNonQuery();

        }
            
}

Note the additional processing around the image tags. This was because I had migrated the image files separately and wanted to update the image tag’s src attributes to reflect to the new path. If you plan on keeping your previous sharepoint installations up, then this step is optional since the attachments will be loaded from the sharepoint site anyway. But I would recommend migrating the images as well just for easier maintenance of the content.

The code to migrate a single image is below. The way to get all the image tags is very similar to how each blog content was retrieved. The only difference is that instead of entering in the database, we just use Agility pack to extract all image tags in the post contennt and call the below method to download it. The files are then saved in the wp-content directory and the path is updated in the migration logic.

  private static void DownloadImage(string url,string postUrl,string destination)
        {
            string saveDir = String.Format("C:\\Wordpress_Images\\{0}\\", destination);
            string filename = string.Concat(saveDir, url.Replace(string.Concat("YOUR_SHAREPOINT_URL", postUrl, "/Lists/Posts/Attachments/"), string.Empty)).Replace("/","\\");
            FileInfo fi = new FileInfo(filename);
            if (!fi.Directory.Exists)
                Directory.CreateDirectory(fi.Directory.FullName);
            //DirectoryInfo info = new DirectoryInfo(
            HttpWebRequest request = (HttpWebRequest)WebRequest.Create(url);
            request.UseDefaultCredentials = true;
            HttpWebResponse response = (HttpWebResponse)request.GetResponse();

            if ((response.StatusCode == HttpStatusCode.OK ||
                response.StatusCode == HttpStatusCode.Moved ||
                response.StatusCode == HttpStatusCode.Redirect) &&
                response.ContentType.StartsWith("image", StringComparison.OrdinalIgnoreCase))
            {

                using (Stream inputStream = response.GetResponseStream())
                using (Stream outputStream = File.OpenWrite(filename))
                {
                    byte[] buffer = new byte[4096];
                    int bytesRead;
                    do
                    {
                        bytesRead = inputStream.Read(buffer, 0, buffer.Length);
                        outputStream.Write(buffer, 0, bytesRead);
                    } while (bytesRead != 0);
                }
            }
        }
Jan 242012
 

Reusability is one of the most underrated aspects of software development. The focus on meeting functional requirements in a product as quickly as possible often overshadows the reusability and maintainability part. Code duplication, poor adherence to design are often symptoms of such development which increase costs in the long term in favour of short term benefits.

WPF encourages creation of reusable controls, primarily by its support for creation of “lookless controls” where the client of the control is not only able to reuse the control, but also completely alter its appearance. The control in essence is just a bundle of logic with a default template. Its upto the client whether to accept this default skin or override it with one of his own. Though this concept looks similar to themes and skins in other technologies, its extremely powerful that you can alter the visual appearance of the control at a granular level. If the default template defines a button and a textbox, that can easily be changed to an filled rectangle with a Linear Gradient and a TextBlock to display the data.

Difference between User Controls and Custom Controls.

Custom controls arent the only type of control library that WPF offers. There is also the User Control library. The difference between the two lies in our end purpose. If we are simply looking to bundle a few controls together and provide basic customization to the end user, then User Controls are the way to go. They are also much more easier to use than Custom Controls. However, if your aim is to provide full customization capability to the developer who consumes your control, then Custom Controls are much better.

Dependency Properties

Dependency Properties are quite different from the conventional properties which we use in C#, and are exclusive to WPF. They do not belong to any particular class and their value can be set sources other than just the class itself. The values could come from their default values, styles, themes, callbacks etc. They also support Data binding so your UI elements can directly bind to them and update whenever the value changes, like how properties in classes implementing INotifyPropertyChanged events behave.

Custom Control Example

The custom control example I build is a filled rectangle with a slider. As you increase the slider the fill percentage of the rectangle increases as well. Here is an illustration

The control is quite simple. There are two main properties here which are exposed to the outer world – The FillColor and the EmptyColor denoting the colors of the rectangle. The third property is the Value of slider which is used within the control, but that too could be exposed out of the control. Lets see the code for the control. ( Notice the absence of any UI stuff)

public class FilledBarControl : Control
    {
        public FilledBarControl()
        {
            DataContext = this;
        }
        
        static FilledBarControl()
        {
            DefaultStyleKeyProperty.OverrideMetadata(typeof(FilledBarControl), new FrameworkPropertyMetadata(typeof(FilledBarControl)));
          
        }

        public static readonly DependencyProperty EmptyColorProperty = DependencyProperty.Register("EmptyColor", typeof(Color), typeof(FilledBarControl), new UIPropertyMetadata((Color)Colors.Transparent));

        public Color EmptyColor
        {
            // IMPORTANT: To maintain parity between setting a property in XAML and procedural code, do not touch the getter and setter inside this dependency property!
            get
            {
                return (Color)GetValue(EmptyColorProperty);
            }
            set
            {
                SetValue(EmptyColorProperty, value);
            }
        }
        

        public static readonly DependencyProperty FillColorProperty = DependencyProperty.Register("FillColor", typeof(Color), typeof(FilledBarControl), new UIPropertyMetadata((Color)Colors.Red));

        public Color FillColor
        {
            // IMPORTANT: To maintain parity between setting a property in XAML and procedural code, do not touch the getter and setter inside this dependency property!
            get
            {
                return (Color)GetValue(FillColorProperty);
            }
            set
            {
                SetValue(FillColorProperty, value);
            }
        }
    }

As you can see, there are two steps to declaring a dependency property – First is the registering of the property using the DependencyProperty.Register method and the second is the definition of the getter and setter methods. The naming convention for the DependencyObject is to append “Property” after the name of the Dependecy Property. Hence here the object becomes FillColorProperty. You can also define the default values for the Property in the Register method. It needs to be passed inside the constructor of the PropertyMetadata object. I used Red Color here, so if the developer doesnt pass any value for the FillColor, the control automatically chooses Red.

Like written before, a custom control is simply defined in code, it doesn’t necessarily need a UI to exist – the UI can be supplied by the developer using the control. The default look and feel for the control is defined in a separate file – Generic.xaml inside the themes folder. Here is the xaml for the filled Bar control.

<Style TargetType="{x:Type local:FilledBarControl}">
        <Setter Property="Template">
            <Setter.Value>
                <ControlTemplate TargetType="{x:Type local:FilledBarControl}">
                    <UniformGrid Columns="1">
                        <Border BorderThickness="{TemplateBinding BorderThickness}" BorderBrush="{TemplateBinding BorderBrush}" >                           
                            <Rectangle Height="{TemplateBinding Height}"
                            Width="{TemplateBinding Width}">
                            <Rectangle.Fill>
                                <LinearGradientBrush StartPoint="0,0" EndPoint="1,0">
                                    <GradientStop Color="{Binding Path=FillColor}"
                        Offset="0"/>
                                    <GradientStop Color="{Binding Path=FillColor}"
                        Offset="{Binding ElementName=slider, Path=Value}"/>
                                    <GradientStop Color="{Binding Path=EmptyColor}" 
                        Offset="{Binding ElementName=slider, Path=Value}"/>
                                </LinearGradientBrush>
                            </Rectangle.Fill>
                        </Rectangle>
                        </Border>

                        <Slider x:Name="slider" Width="200" Height="50" 
            Minimum="0" Maximum="1" Value="0.2"/>
                    </UniformGrid>
                </ControlTemplate>
            </Setter.Value>
        </Setter>
    </Style>

The XAML is just a control template containing a Uniform Grid. One of the rows is the Rectangle and the other a Slider for the value. The rectangle’s fill tag is a LinearGradientBrush with two phases. One from the Start Point (0,0) to the Value (bound to the Slider’s value) and another which starts from the Value to the end point. This gradient gives the impression of a filled rectangle whose fill percentage changes as the slider is dragged.

Now, how can a host application change the appearance of the control? There are two ways – one using the Dependency Properties and other completely overriding the Control Template itself. As you can recall, there were two dependency Properties defined in the FilledBar.cs – The FillColor and the EmptyColor. Both these properties appear in the Intellisense while defining the control in the XAML. An example of such customization would be

         <FilledBar:FilledBarControl HorizontalAlignment="Center" 
                                    VerticalAlignment="Center"  
                                    Height="100" Width="200" 
                                    BorderThickness="2" BorderBrush="#003300" 
                                    FillColor="Maroon" EmptyColor="LightGreen" />

This is how the control now looks. Note that both colours have changed as per our definition

The second form of customization is what makes Custom Controls so much more powerful than ordinary User Controls. Lets assume that a rectangular fill bar doesnt suit my requirement and my application would look better with a FilledCircle rather than a FilledBar. Rather than writing the entire control again just for one change, I could just swap out the Rectangle and substitute it with an Ellipse whose Fill is done by a RadialGradientBrush. These changes do not require any changes to the original control itself – a style can be created in the Resource Dictionary and referred to by the code.

            <Style TargetType="{x:Type FilledBar:FilledBarControl}" x:Key="FilledCircle">
                <Setter Property="Template">
                    <Setter.Value>
                        <ControlTemplate TargetType="{x:Type FilledBar:FilledBarControl}">
                            <UniformGrid Columns="1" Width="{TemplateBinding Width}" Height="{TemplateBinding Height}" >

                                <Ellipse Height="100" Width="100" Stroke="{TemplateBinding BorderBrush}" StrokeThickness="{TemplateBinding BorderThickness}"
                            >
                                        <Ellipse.Fill>
                                            <RadialGradientBrush >
                                                <GradientStop Color="{Binding Path=FillColor}"
                        Offset="0"/>
                                                <GradientStop Color="{Binding Path=FillColor}"
                        Offset="{Binding ElementName=slider, Path=Value}"/>
                                                <GradientStop Color="{Binding Path=EmptyColor}" 
                        Offset="{Binding ElementName=slider, Path=Value}"/>
                                            </RadialGradientBrush>
                                        </Ellipse.Fill>
                                    </Ellipse>
                              

                            <Slider x:Name="slider" Width="200" Height="50" 
            Minimum="0" Maximum="1" Value="0.2"/>
                        </UniformGrid>
                        </ControlTemplate>

                    </Setter.Value>
                </Setter>
                
            </Style>

The only changes in the above style tag are the changing to ellipse and RadialGradientbrush. Note that instead of using the Border, we just bind the BorderBrush and BorderThickness properties to the Stroke and StrokeThickness properties of the Ellipse. This allows setting of the properties just as we did with the unchanged controls and is much more easier to read. The control is declared in xaml with the style tag referring to our user dictionary which overrides the default style written in the Generic.xaml

        <FilledBar:FilledBarControl HorizontalAlignment="Center" VerticalAlignment="Center" 
                                    Grid.Row="3" Grid.Column="1"  
                                    Style="{StaticResource ResourceKey=FilledCircle}"  
                                    Height="200" Width="200" 
                                    BorderThickness="2" BorderBrush="Brown" 
                                    FillColor="Blue" EmptyColor="Transparent">

Here is how the control looks now.

We can have multiple instance of the same control using different styles and values for Dependency Property. An example of the Host Application

Jan 202012
 

I wrote a small application to extract post and user data for groups in the popular enterprise microblogging portal, Socialcast. It lets users access data instantly about the activity going on in their groups and the number of likes/comments being posted by group members.

The details about the API can be found here and their side demo.socialcast.com can be used for testing. There are three main calls in the application

  • Message Data for the group (Using the parameters Group_Id and since to filter by the data)
  • To find the group Id using the group name using the Groups.xml API call.
  • Then finding the group members using groupname/members.xml.

The method to retrieve group_id from the groupname needed improvement. Most organizations contain upwards of a 1000 groups. Making two heavy API calls for finding out the group id from the name is a bit of a drag on the performance of the application. To avoid this, I just cached all the group Ids and names in a Dictionary<string, int> object and used that to lookup the id. For groups that were created after the tool was distributed, the API call was a fallback.

The UI of the tool is fairly simple. Just a textbox to enter the group URL and two datagridviews to display the data along with two listboxes to display the stats. Since data retrieval is a very slow process, a backgroundworker with a progress bar make sure the application doesnt freeze and the user is shown some progress.  There is also a dropdown to select the time for which the user wants to retrieve messages and whether the member information is required or not. The quickest options are selected by default.


The API access code is in a library which hides the webservice calls from the client application. Each method contains the standard page and number of records per page argument. Apart from them, there are other parameter which are needed to make the call e.g GroupName, member id etc. To simply the URL construction for the web service call a class contains the skeleton urls for each call like given below

public static class ObjectType
 {
 static string defaultFormat = ".xml";
 public static string Users = "users" + defaultFormat;
 public static string Streams = "streams" + defaultFormat;
 public static string Messages = "messages" + defaultFormat;
 public static string StreamMessages = "streams/{0}/messages" + defaultFormat;
 public static string MessagesById = "messages/{0}" + defaultFormat;
 public static string SearchUsers = "users/search" + defaultFormat;
 public static string Groups = "groups" + defaultFormat;
 public static string GroupMembers = "groups/{0}/members" + defaultFormat;
 }

public class SocialCastData
 {
 ///
<summary> /// These are the private variables which are configured
 /// as per your socialcast site
 /// </summary>
 string skeletonURL = "https://{0}.socialcast.com/api/{1}";
//Rest of code here
}

The method to construct the service url takes in the Object Type member, and appends it to the skeleton url along with filling in the domain of the socialcast site and any additional parameter (e.g. group id). This approach allows me to add new service calls easily with just one more variable in the ObjectType class. Any additional query string parameters are supplied using the serviceParams which is a list of keyvaluepairs. The list is iterated through and appended after the url. The SocialcastAuthDetails object is a must in every calls since it contains the domain, the username and password all of which are required to be supplied for getting the response.

Here is the method to get the Group ID from the groupName. The GroupQuery.QueryForGroupId accesses the cache and returns immediately if found. If not then another API call is made to get all the groups, iterate through them and identify the id. If the group id is still not found, an exception is thrown.

        private int GetGroupIdByGroupName(string groupName,SocialCastAuthDetails auth)
        {

            int _groupId = 0;
            int _pageNumber =1;
            bool moreItems = true;

            _groupId = GroupsQuery.QueryForGroupID(groupName);
            if (_groupId != 0)
                return _groupId;
            else
            {

                while (moreItems)
                {
                    XmlDocument group = new XmlDocument();
                    var serviceParams = new List>();
                    serviceParams.Add(new KeyValuePair("page", _pageNumber.ToString()));
                    serviceParams.Add(new KeyValuePair("per_page", "500"));
                    group.LoadXml(base.MakeServiceCalls(helper.GetSocialcastURL(ObjectType.Groups, auth.DomainName, null, serviceParams),GetCredentials(auth.Username,auth.Password)));
                    if (group.SelectNodes("//groups/group") == null || group.SelectNodes("//groups/group").Count == 0)
                        moreItems = false;
                    else
                    {
                        foreach (XmlNode groupNode in group.SelectNodes("//groups/group"))
                        {
                            if (String.Compare(GetNodeInnerText(groupNode, "groupname"), groupName, true) == 0)
                            {
                                string id = GetNodeInnerText(groupNode, "id");
                                _groupId = Convert.ToInt32(id);
                                GroupsQuery.AddToDictionary(groupName, _groupId);
                                moreItems = false;
                                break;
                            }
                        }
                    }
                    _pageNumber++;
                }
                return _groupId;
            }
        }

In the front end, a background worker is used to perform the time consuming operation asynchronously. Here is the code (Rough, not refactored)

    class GetStatistics
    {
        APIAccessor _api = new APIAccessor();

        public AppData GetData(string urlEnteredbyUser, string sinceWhatPeriod, bool loadUserData)
        {
            var serviceUrl = UrlReplace(urlEnteredbyUser.Trim());
            List<SocialcastMessage> messages = null;
            List<UserProfile> users = null;
            if (serviceUrl == string.Empty)
            {
                throw new GroupNotFoundException(null, "The group Url you have entered is not valid");
            }
            else
            {
                messages = GetMessageStats(serviceUrl, sinceWhatPeriod);
                if(loadUserData)
                        users = GetAllGroupUsers(serviceUrl);
                return new AppData(messages, users);
            }
        }

        private List<UserProfile> GetAllGroupUsers(string groupName)
        {
            List<UserProfile> Users = new List<UserProfile>();
            int userCount = 500;
            int pageNumber = 1;
            bool moreItems = true;
            while (moreItems)
            {
                var xdoc = _api.GetGroupMembers(groupName, pageNumber.ToString(), userCount.ToString(), GetCredentials());
                if (xdoc.SelectNodes("//users/user") == null || xdoc.SelectNodes("//users/user").Count == 0)
                    moreItems = false;
                else
                {
                    foreach (XmlNode userNode in xdoc.SelectNodes("//users/user"))
                    {
                        string name = GetNodeInnerText(userNode, "name");
                        string userName = GetNodeInnerText(userNode, "username");
                        string email = GetNodeInnerText(userNode, "contact-info/email");
                        int followingCount = Convert.ToInt32(GetNodeInnerText(userNode, "following-count"));
                        int followersCount = Convert.ToInt32(GetNodeInnerText(userNode, "followers-count"));

                        Users.Add(new UserProfile()
                        {
                            Name = name,
                            userName = userName,
                            Email = email,
                            FollowerCount = followersCount,
                            FollowingCount = followingCount
                        });
                       
                    }
                    pageNumber++;
                }
            }

            return Users;
        }

        private List<SocialcastMessage> GetMessageStats(string serviceUrl, string sinceWhatPeriod)
        {
            List<SocialcastMessage> Messages = new List<SocialcastMessage>();
            int messageCount = 500;
            int pageNumber = 1;
            bool moreItems = true;
            long sinceWhen = GetSinceWhenValue(sinceWhatPeriod);
            while (moreItems)
            {
                var xdoc = GetMessagesForTheGroup(serviceUrl, messageCount, pageNumber, sinceWhen);
                if (xdoc.SelectNodes("//messages/message") == null || xdoc.SelectNodes("//messages/message").Count == 0)
                    moreItems = false;
                else
                {
                    foreach (XmlNode node in xdoc.SelectNodes("//messages/message"))
                    {
                        DateTime createdDate;
                        DateTime.TryParse(GetNodeInnerText(node, "created-at"), out createdDate);
                        string ticksLastUpdatedDate = GetNodeInnerText(node, "last-interacted-at");
                        var epoch = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);
                        epoch = epoch.AddSeconds(Int64.Parse(ticksLastUpdatedDate));
                        string permaUrl = GetNodeInnerText(node, "permalink-url");
                        string commentsCount = GetNodeInnerText(node, "comments-count");
                        string likesCount = GetNodeInnerText(node, "likes-count");
                        string title = GetNodeInnerText(node, "title");
                        string user = GetNodeInnerText(node, "user/name");

                        Messages.Add(new SocialcastMessage()
                        {
                            Title = title,
                            Url = permaUrl,
                            CreatedDate = createdDate,
                            UpdatedDate = epoch,
                            Comments = int.Parse(commentsCount),
                            Likes = int.Parse(likesCount),
                            UserName = user
                        });

                    }
                    pageNumber++;
                }
            }


            return Messages;

        }


        private long GetSinceWhenValue(string sinceWhatPeriod)
        {
            switch (sinceWhatPeriod)
            {
                case "All Messages":
                    return 0;
                case "Since Last week":
                    return GetTicksSince1970(7);
                case "Since Last Month":
                    return GetTicksSince1970(30);
                default:
                    return 0;
            }
        }

        private static long GetTicksSince1970(int days)
        {
            DateTime sinceLastWeek = DateTime.Now.Subtract(new TimeSpan(days, 0, 0, 0, 0));
            DateTime epoch = new DateTime(1970, 1, 1);
            return Convert.ToInt64(sinceLastWeek.Subtract(epoch).TotalSeconds);
        }


        private XmlDocument GetMessagesForTheGroup(string streamName, int numberOfPosts, int page, long sinceWhen)
        {
            string sinceWhenString = null;
            if (sinceWhen != 0)
                sinceWhenString = sinceWhen.ToString();
            var xdoc = _api.GetStreamMessages(streamName, numberOfPosts.ToString(), page.ToString(), sinceWhenString, GetCredentials());
           return xdoc;
        }


        private string UrlReplace(string enteredByUser)
        {
        
            string regexPattern = @"https://demo.socialcast.com/groups/([A-Za-z0-9\-]+)";
            Match match = Regex.Match(enteredByUser, regexPattern,
                RegexOptions.IgnoreCase);

            // Here we check the Match instance.
            if (match.Success)
            {
                // Finally, we get the Group value and return it
                string key = match.Groups[1].Value;
                return key;
            }
            return string.Empty;
        }

        private SocialCastAuthDetails GetCredentials()
        {
            return new SocialCastAuthDetails()
            {
                Password = "demo",
                Username = "emily@socialcast.com",
                DomainName = "demo"
            };

        }

        private string GetNodeInnerText(XmlNode node, string xpath)
        {
            try
            {
                if (node.SelectSingleNode(xpath) != null)
                {
                    return node.SelectSingleNode(xpath).InnerText.Trim();
                }
                return String.Empty;
            }
            catch
            {
                return string.Empty;
            }
        }
    }
Dec 242011
 

I have been spending some of my free time trying to build a complete cricket statistics database by parsing records from Cricinfo. However scraping HTML pages is an ardous task. There is simply no standard way of achieving it and often becomes a struggle with regular expressions. A good solution to this problem is the Html Agility Pack. Its a library which standardizes parsing of HTML pages and converts them into a XML style DOM object that you can extract data from. There are a good number of options for error checking (for HTML which is not XHTML compliant)

The API is very similar to the XmlDocument class in System.Xml namespace and hence there is hardly any learning curve. You can search for nodes based on the Xpath expression of the element you want to search. Now getting the xpath can be a bit tricky, so an easier way would be to use a chrome extension called XPath Helper. Once this extension is installed and activated, press Ctrl+Shift+X to activate and then shift to give the xpath of any particular element on which the mouse is hovering. The given XPath can be easily tailored to get the whole set of data which we need to extract.

Now, its time to start scraping. Download Html Agility pack from Codeplex and add a reference to the dll. Its a pretty simple code to get the webpage as a string , then load it in the HTML Agility pack and let it create the DOM structure. Then the XPath is used to get the list of rows in the table and each row is translated into an innings object and added to a collection. At the end its written to a csv file that can be converted to an excel spreadsheet. The code is pretty rough and I did it more for a trial. When the complete database will be built it will become much more difficult since it would involve parsing of different kind of pages and ensure integrity of data.

class Program
    {
        static void Main(string[] args)
        {
            new ReadText().StartParsing();
        }

        
    }

    class ReadText
    {
        public void StartParsing()
        { 
            string TestUrl = "http://stats.espncricinfo.com/ci/engine/stats/index.html?class=1;filter=advanced;page={0};orderby=start;size=200;template=results;type=batting;view=innings;wrappertype=print";
            Console.WriteLine("Extracting Tests\n\n");
            ExtractInningsView(TestUrl, "..\\..\\AllTestInnings.csv",404);
            Console.WriteLine("Extracting ODIs\n\n");
            string ODIUrl = "http://stats.espncricinfo.com/ci/engine/stats/index.html?class=2;filter=advanced;page={0};orderby=start;size=200;template=results;type=batting;view=innings;wrappertype=print";
            ExtractInningsView(ODIUrl,"..\\..\\AllODIInnings.csv",356);

        }

        private void ExtractInningsView(string statUrl,string fileName,int pageCount)
        {
            List<InningsPlayed> AllInnings = new List<InningsPlayed>();
            for (int j = 1; j < pageCount; j++)
            {
                Console.WriteLine("Reading Page: " + j.ToString());
                string pageText = ReadWebPage(String.Format(statUrl, j));
                var htmlDoc = new HtmlDocument();
                htmlDoc.LoadHtml(pageText);

                for (int i = 1; i < 200; i++)
                {
                    string inningsXpath = "//tbody/tr[@class='data1'][{0}]/td";
                    var nodeList = htmlDoc.DocumentNode.SelectNodes(String.Format(inningsXpath, i));

                    if (nodeList != null)
                    {
                        AllInnings.Add(new InningsPlayed()
                        {
                            Name = nodeList[0].InnerText,
                            Runs = nodeList[1].InnerText,
                            Minutes = nodeList[2].InnerText,
                            BallsFaced = nodeList[3].InnerText,
                            Fours = nodeList[4].InnerText,
                            Sixes = nodeList[5].InnerText,
                            StrikeRate = nodeList[6].InnerText,
                            Innings = nodeList[7].InnerText,
                            Opposition = nodeList[9].InnerText,
                            Ground = nodeList[10].InnerText,
                            StartDate = nodeList[11].InnerText
                        });
                    }
                }
            }
            DumpToFile(AllInnings,fileName);
        }

        private void DumpToFile(List<InningsPlayed> AllInnings,string fileName)
        {
            StreamWriter writer = new StreamWriter(fileName);
            
            StringBuilder builder = new StringBuilder();
            writer.WriteLine("Name,Runs,Minutes,Balls_Faced,Fours,Sixes,StrikeRate,Innings,Opposition,Ground,StartDate");
            int iterations = 0;
            foreach (var inning in AllInnings)
            {
                writer.WriteLine(string.Format("{0},{1},{2},{3},{4},{5},{6},{7},{8},{9},{10}", inning.Name, inning.Runs, inning.Minutes, inning.BallsFaced, inning.Fours, inning.Sixes, inning.StrikeRate, inning.Innings, inning.Opposition, inning.Ground, inning.StartDate));
                iterations++;
                if (iterations % 10 == 0)
                    writer.Flush();
            }
            
        }

        private string ReadWebPage(string Url)
        {
            // Reading Web page content in c# program
            //Specify the Web page to read
            WebRequest request = WebRequest.Create(Url);
            //Get the response
            WebResponse response = request.GetResponse();
            //Read the stream from the response
            StreamReader reader = new StreamReader(response.GetResponseStream());
            return reader.ReadToEnd();
        }

        
    }

    class InningsPlayed
    {
        public string Name { get; set; }
        public string Runs { get; set; }
        public string BallsFaced { get; set; }
        public string Minutes { get; set; }
        public string Fours { get; set; }
        public string Sixes { get; set; }
        public string StrikeRate { get; set; }
        public string Innings { get; set; }
        public string Opposition { get; set; }
        public string Ground { get; set; }
        public string StartDate { get; set; }
    }
Aug 142011
 

Testing the user interface has to be the most boring part of testing software. Not only is it not fun, it can be horribly repetitive as well with you having to check the same functionality over and over again. The time spent on it could be put to use to far better purposes. Automated Testing tools like NUnit and MS Test were developed in order to counter this. Along came processes like Test Driven development that emphasized the need to write test cases and write testable software. Though all this helped a lot, its benefits were confined mostly to the developer himself, it wasn’t of any use to the testers who again put in redundant effort in testing the UI.

It was then that the idea of automating UI testing starting getting thrown about and many such tools and frameworks appeared in the market. These frameworks essentially provide an API to programmatically access the user interface of Windows applications and gives the developer to simulate the end user’s way of accessing our software. One such framework which I found extremely useful was the UI Automation library from Microsoft. It works pretty well with any UI technology including relatively recent ones such as WPF.

At the core of the UI Automation library is the type AutomationElement – All UI elements in the application get wrapped into this AutomationElement and provide a common way of interacting with the UI elements. For e.g. The application window will be an AutomationElement just like a small button inside the window. Both support different set of Actions but share the commmon types, this makes writing code for UI automation extremely simple.

I wrote a simple calculator application and a console application which tests the calculator giving random values and verifying the output. The XAML Code for the WPF application is:-

    <grid x:Name="LayoutRoot">
        </grid><grid .RowDefinitions>
            <rowdefinition Height="1*" />
            <rowdefinition Height="1*" />
            <rowdefinition Height="1*" />
            <rowdefinition Height="4*" />
        </grid>
        <grid .ColumnDefinitions>
            <columndefinition Width="2*" />
            <columndefinition Width="4*" />
        </grid>
        <label Grid.Row="0" Grid.Column="0" Content="First Number" />
        <textbox Grid.Row="0" Grid.Column="1" AutomationProperties.AutomationId="txtFirstNumber" Text="{Binding FirstNumber,Mode=TwoWay,UpdateSourceTrigger=PropertyChanged}" />
        <label Grid.Row="1" Grid.Column="0" Content="Second Number" />
        <textbox Grid.Row="1" Grid.Column="1" AutomationProperties.AutomationId="txtSecondNumber" Text="{Binding SecondNumber,Mode=TwoWay,UpdateSourceTrigger=PropertyChanged}" />
        <label Grid.Row="2" Grid.Column="0" Content="Result" />
        <textbox Grid.Row="2" Grid.Column="1" AutomationProperties.AutomationId="txtResult" Text="{Binding Result,Mode=TwoWay,UpdateSourceTrigger=PropertyChanged}" />
        
        <grid Grid.Row="3" Grid.Column="0" Grid.ColumnSpan="2">
            </grid><grid .ColumnDefinitions>
                <columndefinition Width="1*" />
                <columndefinition Width="1*" />
                <columndefinition Width="1*" />
                <columndefinition Width="1*" />
            </grid>
            <button Grid.Column="0" AutomationProperties.AutomationId="btnAdd" Content="Add" Command="{Binding Add}" />
            <button Grid.Column="1" AutomationProperties.AutomationId="btnSubtract" Content="Subtract" Command="{Binding Subtract}" />
            <button Grid.Column="2" AutomationProperties.AutomationId="btnMultiply" Content="Multiply" Command="{Binding Multiply}" />
            <button Grid.Column="3" AutomationProperties.AutomationId="btnDivide" Content="Divide" Command="{Binding Divide}" />
   

This creates a simple calculator app with four functions Add, Subtract, Multiply and Divide. Here is a screenshot of the app

The viewmodel is pretty simple as well

    public class MainViewModel : ViewModelBase
    {
       
        private double _firstNumber;
        private double _secondNumber;
        private double _result;
        private RelayCommand _add;
        private RelayCommand _multiply;
        private RelayCommand _subtract;
        private RelayCommand _divide;

        public double FirstNumber
        {
            get { return _firstNumber; }
            set
            {
                _firstNumber = value;
                RaisePropertyChanged("FirstNumber");
            }
        }

        public double SecondNumber
        {
            get { return _secondNumber; }
            set
            {
                _secondNumber = value;
                RaisePropertyChanged("SecondNumber");
            }
        }

        public double Result
        {
            get { return _result; }
            set
            {
                _result = value;
                RaisePropertyChanged("Result");
            }
        }

        public ICommand Add
        {
            get { _add = _add ?? new RelayCommand(() => Result = FirstNumber + SecondNumber); return _add; }
        }

        public ICommand Subtract
        {
            get { _subtract = _subtract ?? new RelayCommand(() => Result = FirstNumber - SecondNumber); return _subtract; }
        }

        public ICommand Multiply
        {
            get { _multiply = _multiply ?? new RelayCommand(() => Result = FirstNumber * SecondNumber); return _multiply; }
        }

        public ICommand Divide
        {
            get { _divide = _divide ?? new RelayCommand(() => Result = FirstNumber / SecondNumber); return _divide; }
        }
  }

The project also contains a simple Console application which launches along with the project and automates the testing of the application by entering values in the textboxes and pressing the four buttons and verifying the result values.

namespace CalculatorTest
{
    class TestCalc
    {
        AutomationLibrary library = new AutomationLibrary();

        AutomationElement desktopElement;
        AutomationElement mainWindow;

        public TestCalc()
        {
            desktopElement = GetDesktop();
        mainWindow = library.GetElement(desktopElement, AutomationElement.NameProperty, "Simple Calculator");
        }
        internal void StartTest()
        {
            for (int i = 0; i < 25; i++)
            {
                Random r = new Random();
                double firstNumber = r.Next(1300);
                Thread.Sleep(30);
                double secondNumber = r.Next(1000);
                if (VerifyResult(firstNumber, secondNumber))
                    Console.WriteLine(" All Tests Have Passed for {0} & {1}",firstNumber,secondNumber);
                else
                    Console.WriteLine("Tests have failed for {0} & {1}", firstNumber, secondNumber);
            }

        }

        public bool VerifyResult(double firstNumber,double secondNumber)
        {
            bool allTestPass = true; 
            library.SetValueOnTextBox(mainWindow, AutomationElement.AutomationIdProperty, "txtFirstNumber", TreeScope.Descendants, firstNumber.ToString());
            library.SetValueOnTextBox(mainWindow, AutomationElement.AutomationIdProperty, "txtSecondNumber", TreeScope.Descendants, secondNumber.ToString());
            double addResult = PressOpButtonandGetValue("btnAdd");
            double subResult = PressOpButtonandGetValue("btnSubtract");
            double mulResult = PressOpButtonandGetValue("btnMultiply");
            double divResult = PressOpButtonandGetValue("btnDivide");

            if (addResult != (firstNumber + secondNumber))
                allTestPass = false;
            if (subResult != (firstNumber - secondNumber))
                allTestPass = false;
            if (mulResult != (firstNumber * secondNumber))
                allTestPass = false;
            if (Math.Round(divResult,5) != Math.Round((firstNumber / secondNumber),5))
                allTestPass = false;
            return allTestPass;
        }

        public double PressOpButtonandGetValue(string buttonID)
        {
            library.PressButtonOnWindow(mainWindow, AutomationElement.AutomationIdProperty, buttonID, TreeScope.Descendants);
            Thread.Sleep(200);
            string result = library.GetTextFromTextElement(mainWindow, AutomationElement.AutomationIdProperty, "txtResult", TreeScope.Descendants);
            double actualResult;
            if (double.TryParse(result, out actualResult))
                return actualResult;
            throw new ArgumentException("Wrong Value detected");
        }

        private AutomationElement GetDesktop()
        {
            Console.WriteLine("Getting Desktop");
            var desktopElement = library.GetRootElement();
            if (desktopElement == null)
            {
                Console.WriteLine("Unable to get the desktop Element, Exiting the application");
                throw new ElementNotAvailableException("Unable to get the desktop Element");
            }
            return desktopElement;
        }
    }
}

Directly interacting with the Microsoft UI Automation API can lead to extremely repetitive code since most of the functions are used again and again. So I wrote a small library to simply this. Here is the code. It is still largely uncommented and not refactored yet, will improve it in future blog posts.

class AutomationLibrary
    {
        public AutomationElement GetRootElement()
        {
            return AutomationElement.RootElement;
        }

        public AutomationElement GetElement(AutomationElement rootElement, AutomationProperty property, object value)
        {
            return GetElement(rootElement, property, value, TreeScope.Children);
        }

        public AutomationElement GetElement(AutomationElement rootElement, AutomationProperty property, object value, TreeScope searchScope)
        {

            AutomationElement aeMainWindow = null;

            int numWaits = 0;
            do
            {
                aeMainWindow = rootElement.FindFirst(searchScope, new PropertyCondition(property, value));
                ++numWaits;
                Thread.Sleep(200);
            } while (aeMainWindow == null && numWaits < 50);
            return aeMainWindow;
        }

        public AutomationElement GetElementWithoutWait(AutomationElement rootElement, AutomationProperty property, object value)
        {
            return GetElementWithoutWait(rootElement, property, value, TreeScope.Children);
        }

        public AutomationElement GetElementWithoutWait(AutomationElement rootElement, AutomationProperty property, object value, TreeScope searchScope)
        {
            AutomationElement aeMainWindow = rootElement.FindFirst(searchScope, new PropertyCondition(property, value));
            return aeMainWindow;
        }


        public bool PressButtonOnWindow(AutomationElement element, AutomationProperty property, object value)
        {
            return PressButtonOnWindow(element, property, value, TreeScope.Children);
        }

        public bool PressButtonOnWindow(AutomationElement element, AutomationProperty property, object value, TreeScope treeScope)
        {
            try
            {
                //var window = GetElementWithoutWait(element, property, value,treeScope);
                //if (window == null)
                //    window = GetElement(element,property,value,treeScope);
                //if(window==null)
                //    return false;
                //else
                //{
                var buttonInvoke = GetInvokePattern(element, property, value, TreeScope.Descendants);
                buttonInvoke.Invoke();
                return true;
                // }
            }
            catch
            {
                return false;
            }
        }

        public InvokePattern GetInvokePattern(AutomationElement window, AutomationProperty property, object value, TreeScope treeScope)
        {
            AutomationElement aeButtonElement;
            int numWaits = 0;
            do
            {
                aeButtonElement = window.FindFirst(treeScope, this.GetPropertyCondition(property, value));
                ++numWaits;
                Thread.Sleep(500);
            } while (aeButtonElement == null && numWaits < 75);
            object objPattern;
            InvokePattern invokePatternObj;
            if (aeButtonElement.TryGetCurrentPattern(InvokePattern.Pattern, out objPattern))
            {
                invokePatternObj = objPattern as InvokePattern;
                return invokePatternObj;
            }
            return null;
        }

        public ValuePattern GetValuePatternWithoutWait(AutomationElement window, AutomationProperty property, object value, TreeScope searchScope)
        {
            AutomationElement aeTextBoxElement = window.FindFirst(searchScope, GetPropertyCondition(property, value));
            if (aeTextBoxElement == null)
                throw new ElementNotAvailableException("TextBoxElement Element not Available");
            object objPattern;
            ValuePattern valuePatternObj;
            if (aeTextBoxElement.TryGetCurrentPattern(ValuePattern.Pattern, out objPattern))
            {
                valuePatternObj = objPattern as ValuePattern;
                return valuePatternObj;
            }
            else
                throw new ElementNotEnabledException("The Value Pattern was not retrieved from the element");
        }

        public InvokePattern GetInvokePatternWithoutWait(AutomationElement window, AutomationProperty property, object value, TreeScope searchScope)
        {
            AutomationElement aeButtonElement = window.FindFirst(searchScope, GetPropertyCondition(property, value));
            if (aeButtonElement == null)
                throw new ElementNotAvailableException("Button Element not Available. Try the GetInvokePattern method which has a wait");
            object objPattern;
            InvokePattern invokePatternObj;
            if (aeButtonElement.TryGetCurrentPattern(InvokePattern.Pattern, out objPattern))
            {
                invokePatternObj = objPattern as InvokePattern;
                return invokePatternObj;
            }
            else
                throw new ElementNotEnabledException("The Invoke Pattern was not retrieved from the element");
        }


        public ExpandCollapsePattern GetExpandCollapsePattern(AutomationElement element, AutomationProperty property, object value, TreeScope searchScope)
        {
            AutomationElement aeExpanderElement;
            int numWaits = 0;
            do
            {
                aeExpanderElement = GetFirstChildNode(element, property, value, searchScope);
                ++numWaits;
                Thread.Sleep(300);
            } while (aeExpanderElement == null && numWaits < 75);
            object objPattern;
            ExpandCollapsePattern togPattern;
            if (true == aeExpanderElement.TryGetCurrentPattern(ExpandCollapsePattern.Pattern, out objPattern))
            {
                togPattern = objPattern as ExpandCollapsePattern;
                return togPattern;
            }
            else
                return null;

        }

        public void SetValueOnTextBox(AutomationElement element, AutomationProperty property, object value, TreeScope searchScope, string valueToBeSet)
        {
            var valuePattern = GetValuePatternWithoutWait(element, property, value, searchScope);
            if (valuePattern != null)
                valuePattern.SetValue(valueToBeSet);
        }

        public ExpandCollapsePattern GetExpandCollapsePatternWithoutWait(AutomationElement element, AutomationProperty property, object value, TreeScope searchScope)
        {
            try
            {
                AutomationElement aeExpanderElement = GetFirstChildNode(element, property, value, searchScope);
                if (aeExpanderElement == null)
                    throw new ElementNotAvailableException("Expander Element not available. Try the GetExpandCollapsePattern which has a wait");
                object objPattern;
                ExpandCollapsePattern togPattern;
                if (true == element.TryGetCurrentPattern(ExpandCollapsePattern.Pattern, out objPattern))
                {
                    togPattern = objPattern as ExpandCollapsePattern;
                    return togPattern;
                }
                else
                    return null;
            }
            catch
            {
                return null;
            }
        }

        public SelectionItemPattern GetSelectionItemPattern(AutomationElement element, AutomationProperty property, object value, TreeScope searchScope)
        {
            AutomationElement aeSelectionPattern;
            int numWaits = 0;
            do
            {
                aeSelectionPattern = GetFirstChildNode(element, property, value, searchScope);
                ++numWaits;
                Thread.Sleep(300);
            } while (aeSelectionPattern == null && numWaits < 75);
            object objPattern;
            SelectionItemPattern selectionItemPattern;
            if (true == element.TryGetCurrentPattern(SelectionItemPattern.Pattern, out objPattern))
            {
                selectionItemPattern = objPattern as SelectionItemPattern;
                return selectionItemPattern;
            }
            else
                return null;
        }

        public SelectionItemPattern GetSelectionItemWithoutWait(AutomationElement element, AutomationProperty property, object value, TreeScope searchScope)
        {
            try
            {
                AutomationElement aeExpanderElement = GetFirstChildNode(element, property, value, searchScope);
                if (aeExpanderElement == null)
                    throw new ElementNotAvailableException("Expander Element not available. Try the GetSelectionItemPattern which has a wait");
                object objPattern;
                SelectionItemPattern togPattern;
                if (true == element.TryGetCurrentPattern(SelectionItemPattern.Pattern, out objPattern))
                {
                    togPattern = objPattern as SelectionItemPattern;
                    return togPattern;
                }
                else
                    return null;
            }
            catch
            {
                return null;
            }
        }

        public TextPattern GetTextPatternWithoutWait(AutomationElement element, AutomationProperty property, object value, TreeScope searchScope)
        {
            try
            {
                AutomationElement aeTextElement = GetFirstChildNode(element, property, value, searchScope);
                if (aeTextElement == null)
                    throw new ElementNotAvailableException("Text Element not available");
                object objPattern;
                TextPattern txtPattern;
                if (true == aeTextElement.TryGetCurrentPattern(TextPattern.Pattern, out objPattern))
                {
                    txtPattern = objPattern as TextPattern;
                    return txtPattern;
                }
                else
                    return null;
            }
            catch
            {
                return null;
            }
        }

        public string GetTextFromTextElement(AutomationElement element, AutomationProperty property, object value, TreeScope searchScope)
        {
            var textPattern = GetTextPatternWithoutWait(element, property, value, searchScope);
            if (textPattern != null)
                return textPattern.DocumentRange.GetText(-1);
            return null;

        }



        public PropertyCondition GetPropertyCondition(AutomationProperty property, object value)
        {
            return new PropertyCondition(property, value);
        }

        public AutomationElementCollection GetAllChildNodes(AutomationElement element, AutomationProperty automationProperty, object value, TreeScope treeScope)
        {
            var allChildNodes = element.FindAll(treeScope, GetPropertyCondition(automationProperty, value));
            if (allChildNodes == null)
                throw new ElementNotAvailableException("Not able to find the child nodes of the element");
            return allChildNodes;
        }

        public AutomationElement GetFirstChildNode(AutomationElement element, AutomationProperty property, object value, TreeScope searchScope)
        {
            var firstchildNode = element.FindFirst(searchScope, GetPropertyCondition(property, value));
            if (firstchildNode == null)
                throw new ElementNotAvailableException("Not able to find the first child node of the element");
            return firstchildNode;
        }

        public bool FirstChildTextNodeContains(AutomationElement element, string toCheck)
        {
            var firstTextNode = this.GetFirstChildNode(element, AutomationElement.ControlTypeProperty, ControlType.Text, TreeScope.Children);
            if (firstTextNode.Current.Name == toCheck)
                return true;
            else
                return false;
        }

        public void SelectComboBoxItem(AutomationElement element, int indexToBeSelected, AutomationProperty property, object value, TreeScope searchScope)
        {
            var allChildren = this.ExpandComboBoxViewAndReturnChildren(element, property, value, searchScope);
            if (allChildren.Count < indexToBeSelected)
                throw new Exception("The combobox has fewer items than those that need to be selected");
            var itemToBeSelected = allChildren[indexToBeSelected];
            var selectPattern = this.GetSelectionItemPattern(itemToBeSelected, AutomationElement.ControlTypeProperty, ControlType.ListItem, TreeScope.Element);
            if (selectPattern != null)
                selectPattern.Select();
            var togglePattern = this.GetExpandCollapsePattern(element, property, value, TreeScope.Descendants);
            if (togglePattern != null)
                togglePattern.Collapse();
        }

        public AutomationElementCollection ExpandComboBoxViewAndReturnChildren(AutomationElement element, AutomationProperty property, object value, TreeScope searchScope)
        {
            try
            {
                var comboBox = GetFirstChildNode(element, AutomationElement.AutomationIdProperty, value, searchScope);
                var expandPattern = GetExpandCollapsePattern(comboBox, AutomationElement.ControlTypeProperty, ControlType.ComboBox, TreeScope.Element);
                if (expandPattern == null)
                    throw new ElementNotAvailableException("Couldnt Find Expand Pattern in combobox");
                expandPattern.Expand();
                return this.GetAllChildNodes(comboBox, AutomationElement.ControlTypeProperty, ControlType.ListItem, TreeScope.Children);
            }
            catch (Exception e1)
            {
                throw e1;
            }
        }

        public void WorkaroundPopulateControlTree(int x1, int y1)
        {
            uint x = (uint)x1;
            uint y = (uint)y1;
            NativeMethods.SetCursorPos(x1, y1);
            Thread.Sleep(200);
            NativeMethods.mouse_event(NativeMethods.MOUSEEVENTF_LEFTDOWN | NativeMethods.MOUSEEVENTF_LEFTUP, x, y, 0, 0);
        }


        public bool WaitTillElementSelected(AutomationElement aeMainWindow, AutomationProperty automationProperty, object value, TreeScope treeScope)
        {
            var btnStartAnalysis = GetElement(aeMainWindow, automationProperty, value, treeScope);
            int numWait = 0;
            do
            {
                if (btnStartAnalysis.Current.IsEnabled == true)
                    return true;
                Thread.Sleep(2000);
            } while (btnStartAnalysis.Current.IsEnabled == false && numWait < 90);

            return false;
        }
    }

    class NativeMethods
    {

        [DllImport("user32.dll", CharSet = CharSet.Auto, CallingConvention = CallingConvention.StdCall)]
        public static extern void mouse_event(uint dwFlags, uint dx, uint dy, uint cButtons, uint dwExtraInfo);

        public const int MOUSEEVENTF_LEFTDOWN = 0x02;
        public const int MOUSEEVENTF_LEFTUP = 0x04;
        public const int MOUSEEVENTF_RIGHTDOWN = 0x08;
        public const int MOUSEEVENTF_RIGHTUP = 0x10;

        /// Return Type: BOOL->int  
        ///X: int  
        ///Y: int  
        [System.Runtime.InteropServices.DllImportAttribute("user32.dll", EntryPoint = "SetCursorPos")]
        [return: System.Runtime.InteropServices.MarshalAsAttribute(System.Runtime.InteropServices.UnmanagedType.Bool)]
        public static extern bool SetCursorPos(int X, int Y);

    }
}

Running the application launches the console application which enters random inputs and tests the result for all mathematical operations.

The code can be downloaded here.

May 232011
 

At the recently concluded I/O developer conference, Google made an much awaited announcement – The Google Places API has been opened to everyone (was in beta testing for some time). For the uninitiated, Google Places is a Google application for searching local businesses like hotels, ATMs etc. Places fits in beautifully with Google maps both on the web and the Android.

Now with the API being opened up, Its also possible for any location aware tools and websites to make uses of Places search option to add the functionality to your own sites. Check out the documentation here. I integrated the Places API search in my previous geolocation example explained in a blog post here. The application returns 20 places near the user’s current location and adds markers to the map for each.

Web Workers

Web workers are one of the most interesting concepts of HTML5. They are a standard based on running JS scripts on a background thread rather than the main UI thread. This is extremely important since the more time consuming scripts (like complex mathematical calculations) can be offloaded to a secondary thread rather than freezing up your application, having huge applications in graphics intensive work. I used Web Workers in the current application to work call a server side method which in turn calls the Google places API to search for a list of places near the user’s application.

Here are some limitations of Web Workers in their current implementation:-

  • Not supported on all browsers (Most notably Internet Explorer).
  • We cannot access any DOM object in the Web Worker script. All communication needs to be to the main thread using the postMessage function.
  • Because we cannot access DOM objects, it also doesn’t allow any script to be loaded which refers to DOM, which renders most JS libraries like JQuery and Prototype unusable.

In this example, I following components.

  • An ASP.NET MVC server side in order to call the Google places API. Its a controller method which calls the API url and a method which holds the data. On client side we have a similar method in JSON with same properties. The ASP.NET MVC Model binder converts the JSON object to a CLR object and passes it to the Action method. Client Side Javascript cannot call the Google places API directly because it would be a cross site request and not allowed by Google. Hence the Server’s broker method becomes necessary here
  • Client side main script which uses Geolocation to determine the user’s location. It then passes this location to the MVC Action. Before calling the action, it checks whether the browser supports Web Workers – If so they are offloaded to secondary thread. Else called on the main thread itself.
  • Worker script which makes the AJAX call to the Server using xmlHttpRequest object (Jquery cannot be used here) 🙁

Here is the code.

ASP.NET Server Side

        public ActionResult GoogleSearchAPI(SearchQuery query)
        {
            //Base URL for calling the Google Places API
            string BaseAPIURL = String.Format("https://maps.googleapis.com/maps/api/place/search/xml?location={0},{1}&radius={2}", query.Latitude, query.Longitude, query.Radius);
            if (!string.IsNullOrEmpty(query.Name))
                //Append the name parameter only if data is sent from Client side.
                BaseAPIURL = String.Concat(BaseAPIURL, String.Format("&name={0}", query.Name));
            //Include the API Key which is necessary
            BaseAPIURL = String.Concat(BaseAPIURL, String.Format("&sensor=false&key={0}", GetAPIKey()));
            //Get the XML result data from Google Places using a helper method whichc makes the call.
            string _response = MakeHttpRequestAndGetResponse(BaseAPIURL);
            //Wrap XML in a ContentResult and pass it back the Javascript
            return Content(_response);
        }

        //Helper method to call the URL and send the response back.
        private string MakeHttpRequestAndGetResponse(string BaseAPIURL)
        {
            var request = (HttpWebRequest)WebRequest.Create(BaseAPIURL);
            request.Method = WebRequestMethods.Http.Get;
            request.Accept = "application/json";
            string text;
            var response = (HttpWebResponse)request.GetResponse();

            using (var sr = new StreamReader(response.GetResponseStream()))
            {
                text = sr.ReadToEnd();
            }

            return text;
        }

    ///
    /// Our data object  which sends the object with data from client side to
    /// server. The Model binder takes care of conversion between JSON and
    /// CLR objects.
    ///
    public class SearchQuery
    {
        public string Latitude { get; set; }
        public string Longitude { get; set; }
        public string Radius { get; set; }
        public string Type { get; set; }
        public string Name { get; set; }
    }

Client Side Main script

Most of the Geolocation code is the same as my previous example. This is the additional code written after the geolocation data is found and the coordinates is passed on to another method which uses it to retrieive places data and mark it on map

var spawnWorkerThread = function (position) {
    //This is executed if the getPosition is successfull. Moving the map to the user's location
    map.panTo(new google.maps.LatLng(position.coords.latitude, position.coords.longitude));
    var coordinates = new google.maps.LatLng(position.coords.latitude, position.coords.longitude);
    //Create a JSON object with the details of search query.
    var placesQuery = { Latitude: position.coords.latitude, Longitude: position.coords.longitude, Type: "establishment", Radius: "500" };
    //Check if the browser supports WEbWorkers
    if (Modernizr.webworkers) {
        printMsg("Web Workers are supported on your browser. Searching for places nearby your location");
        //Load the Worker Script
        var myWorker = new Worker("/files/webworkersmvc/Scripts/worker.js");
        //Send the JSON object to the Worker thread after serializing it to string
        myWorker.postMessage(JSON.stringify(placesQuery));
        // receive a message from the worker
        myWorker.onmessage = function (event) {
            //Send the returned data to the processPlacesData method
            processPlacesData(event.data);
        };
    }
    else {
    //Make the call in a standard way and not using Web Workers
        printMsg("Web Workers isnt supported on your browser.Calling Places API the conventional way");
        var xhr = new XMLHttpRequest();
        //Calling the controller method.
        xhr.open("POST", "http://www.ganeshran.com/Files/webworkersmvc/Home/GoogleSearchAPI");
        xhr.setRequestHeader("Content-Type", "application/json");
        xhr.onreadystatechange = function () {
            if (xhr.readyState == 4 && (xhr.status == 200 || xhr.status == 0)) {
                processPlacesData(xhr.responseText);
            }
        }
        xhr.send(JSON.stringify(placesQuery));
    }

};

var processPlacesData = function (data) {
    //Parse the XML result into an xml obect
    var places = $.parseXML($.trim(data));
    var xmldoc = $(places);
    var resultstring = "";
    //Iterate through each result object
    $("result", places).each(function () {
        var typestring = "";
        $("type", this).each(function () {
            typestring += $(this).text() + " , ";
        });
        //Create a MapResult object for each result.
        var resObj = new mapResult($("name", this).text(),
                                      $('vicinity', this).text(),
                                      typestring,
                                      $('lat', this).text(),
                                      $('lng', this).text(),
                                      $('icon', this).text());
        //Create a Google Maps Marker and use the result object's latitude and
        //longitude.
        var marker = new google.maps.Marker({
            position: new google.maps.LatLng(resObj.latitude, resObj.longitude),
            title: resObj.name,
            animation: google.maps.Animation.DROP
        });
        //If the screen is smaller then zoom lesser else zoom more closer.
        //This is to make the markers visible
        if (screen.width < 1000) {
            map.setZoom(12);
        }
        else {
            map.setZoom(15);
        }
        //Set each marker on the map
        marker.setMap(map);
        //this is for the window to show information when the marker is clicked.
        //A single infowindow is reused in order to display only one
        google.maps.event.addListener(marker, 'click', function () {
            infowindow.setContent(resObj.getMarkerHTML());
            infowindow.open(map, marker);
        });
    });

};

//Javascript object to hold the map data and the get the HTML required for the marker.
function mapResult (name, vicinty, types, latitude, longitude, icon) {
    this.name = name;
    this.vicinity= vicinty;
    this.types = types;
    this.latitude = latitude;
    this.longitude = longitude;
    this.iconpath = icon;
}
//Prototype method to avoid creating seperate copies of the method
//for each object
mapResult.prototype.getMarkerHTML = function () {
var htmlstring = "
<div style="color: blue; font-weight: bold;">";
    htmlstring += "Name: " + this.name + "";
    htmlstring += "Types: " + this.types + "";
    htmlstring += "Location: " + this.latitude + "," + this.longitude + "";
    htmlstring += "Vicinity: " + this.vicinity +"</div>";
    return htmlstring;
};

Worker Side Script

The worker side script is pretty straightforward. Just calls the controller and passes on the data to the main thread using the PostMessage function

// receive a message from the main JavaScript thread
onmessage = function (event) {
    // do something in this worker
    var info = event.data;
    var xhr = new XMLHttpRequest();
    xhr.open("POST", "http://www.ganeshran.com/Files/webworkersmvc/Home/GoogleSearchAPI");
    xhr.setRequestHeader("Content-Type", "application/json");
    xhr.onreadystatechange = function () {
        if (xhr.readyState == 4 && (xhr.status == 200 || xhr.status == 0)) {
            //Send message back to Main thread
            postMessage(xhr.responseText);
        }
    }
    xhr.send(info);
};

Demo Page

http://www.ganeshran.com/files/webworkersmvc/

Demo Pics

On PC

On Android

Dec 272010
 

Since the last post, I changed the library/API wrapper a bit. I removed all the ugly reflection stuff to retrieve the specific API urls and substituted them with static variables in a separate class. However this does have the added disadvantage that the urls are exposed to the client, but at least it wont break any client code if Socialcast decides to change the API in the future. Also in the previous example, the username, password and subdomain are variables in the wrapper itself. In the absence of oAuth, every call needs to be authenticated with the user credentials. To avoid having to handle the responsibility of storing user information, I created a class to encapsulate this information (SocialcastAuthDetails) which is passed to the API Accessor for every call. I also added the data objects to return strongly typed responses from the API accessor instead of an XmlDocument, but havent gotten around to incorporate them yet.

Here is the code to Post a message and Get the company stream. Accessing the Company Stream requires two calls – first to get the Stream ID and the next to get the messages for the particular stream.

        public XmlDocument GetCompanyStream(SocialCastAuthDetails auth)
        {
            XmlDocument streams = new XmlDocument();
            if (companyStreamID == 0)
            {
                streams.LoadXml(base.MakeServiceCalls(helper.GetSocialcastURL(ObjectType.Streams,auth.DomainName,null), GetCredentials(auth.Username,auth.Password)));

                foreach (XmlNode node in streams.GetElementsByTagName("stream"))
                {
                    if (node.SelectSingleNode("name").InnerText.ToLower() == "company stream")
                    {
                        companyStreamID = int.Parse(node.SelectSingleNode("id").InnerText);
                        break;
                    }
                }
            }
            streams = new XmlDocument();
            streams.LoadXml(base.MakeServiceCalls(
                                 helper.GetSocialcastURL(ObjectType.StreamMessages,auth.DomainName,companyStreamID.ToString()),
                                 GetCredentials(auth.Username,auth.Password)));
            return streams;
        }

        public XmlDocument PostMessage(string title,string body,SocialCastAuthDetails auth)
        {
            string data = String.Format("message[title]={0}&message[body]={1}", HttpUtility.UrlEncode(title), HttpUtility.UrlEncode(body));
            XmlDocument update = new XmlDocument();
            update.LoadXml(base.MakeServiceCallsPOST(
                                helper.GetSocialcastURL(ObjectType.Messages, auth.DomainName, null),
                                GetCredentials(auth.Username, auth.Password), data));
            return update;
        }

Since any messages which manipulates data requires a POST call instead of a GET, the WebServiceHelper class needs a new method to make the service call using POST. Also the data which is to be posted is URL encoded before being sent to this method.

  protected string MakeServiceCallsPOST(string _requestURL, NetworkCredential credentials, string data)
        {
            // Create the web request
            HttpWebRequest request = WebRequest.Create(_requestURL) as HttpWebRequest;

            request.Credentials = credentials;
            request.ContentType = "application/x-www-form-urlencoded";
            request.Method = "POST";

            byte[] bytes = Encoding.UTF8.GetBytes(data);

            request.ContentLength = bytes.Length;
            using (Stream requestStream = request.GetRequestStream())
            {
                requestStream.Write(bytes, 0, bytes.Length);

                using (WebResponse response = request.GetResponse())
                {
                    using (StreamReader reader = new StreamReader(response.GetResponseStream()))
                    {
                        return reader.ReadToEnd();
                    }
                }
            }
        }

This is the client code to post the message. The socialcast auth details class is initialized by the client and sent, so its their headache to maintain passwords and other sensitive information.

    class Program
    {
        static SocialCastAuthDetails auth = new SocialCastAuthDetails()
        {
            DomainName = "demo",
            Username = "emily@socialcast.com",
            Password = "demo"
        };
        static void Main(string[] args)
        {
            int _messageCounter=1;
            APIAccessor api = new APIAccessor();
            api.PostMessage("Posting from API", "this is a test message posted through C#", auth);
            var xdoc = api.GetCompanyStream(auth);
            Console.WriteLine("Company Steam of demo.socialcast.com");
            Console.WriteLine("******************************************************");
            foreach(XmlNode node in xdoc.GetElementsByTagName("message"))
            {
                Console.WriteLine("Message {0} posted by {1}", _messageCounter++, node.SelectSingleNode("user/name").InnerText);
                Console.WriteLine("Message: {0} {1}", node.SelectSingleNode("title").InnerText, node.SelectSingleNode("body").InnerText);
                Console.WriteLine("====================================================");
            }
        }
    }

It works!!

Dec 252010
 

Socialcast is one of the better enterprise microblogging tools out there. I have been trying to use its API to understand better how people use microblogging in the enterprise. There is no better way to validate (or invalidate) set hypotheses than by actually mining data and identifying patterns in them. If sufficient data exists which is spread across a larger time period, fascinating patterns emerge. In this older post, I had correlated the length of each blog post (in my previous organization’s internal blogging platform) with the number of comments it received. After a senior colleague helped me make sense out of the data, a clear conclusion was that the shorter a blog post is, the more likely people will comment on it.

To attempt something similar with Socialcast, I finally got around to using their API through C# after procrastinating for a very long time. I didnt map the API responses/arguments to .NET objects yet, just wrote a few classes to make it easier to make different service calls without repeating code. In the below code, I used the API to return user details of some of the users. Similarly there are different calls to get groups, streams, followers etc. All the information which is gotten from the site needs a GET call, anything where information is changed (commenting, posting a message etc.) has to be done via POST.

Every socialcast site has a different subdomain (e.g. demo.socialcast.com) and a username/password to authorize requests. Since oAuth is not yet available, this information needs to be stored in the application itself. I saved it in a class for now but a better way would be to store it in a config file (and encrypt it for good measure). The SocialCastData class has all the client specific details like api urls, usernames , password etc. All these properties are protected and only the Helper class which inherits from the data class can access it. The helper class provides the API URL and credentials to the APIAccessor class.

    class SocialCastData
    {
        /// <summary>
        /// These are the private variables which are configured
        /// as per your socialcast site
        /// </summary>
        string domainName = "demo";
        string skeletonURL = "https://{0}.socialcast.com";
        string userName = "emily@socialcast.com";
        string passWord = "demo";
        string _usersURL = "/api/users.xml";

        //Protected properties to give access to the username/password and API
        //URL only to the helper class which inherits from this data classes
        protected string UserName { get { return userName; } }
        protected string Password { get { return passWord; } }
        protected string usersURL { get { return _usersURL; } }

        //Get the basic URL of the site, without any API call
        protected string GetSocialCastURL()
        {
            return String.Format(skeletonURL, domainName);
        }

        //This method uses reflection to provide the API url
        // value based on an agument to this method
        protected string GetSocialCastURL(string apiFilter)
        {
            try
            {
                PropertyInfo _allProperties = this.GetType().GetProperty(apiFilter + "URL", BindingFlags.NonPublic | BindingFlags.Instance);
                if (_allProperties == null)
                    throw new Exception("There was no corresponding API URL found");
                else
                {
                    string value = _allProperties.GetValue(this, null).ToString();
                    return GetSocialCastURL() + value;
                }
            }
            catch (Exception _eObj) { throw new Exception("There was no corresponding API URL found", _eObj); }
        }

    }

    /// <summary>
    /// This is the helper class which provides the URL
    /// and Credentials to the WebServiceHelper object. Only the Helper
    /// class has access to the SocialCastData members since all its
    /// members are protected.
    /// </summary>
    class SocialcastHelper:SocialCastData
    {

        //use the base class data to get the Credentials object.
        public NetworkCredential GetCredentials()
        {
            return new NetworkCredential(base.UserName, base.Password);
        }

        //Get the URL for the socialcast website. The overloaded methods are for
        //returning the appropriate URL depending on the function and if any additional
        //query parameters are to be appended to the URL.
        public string GetSocialcastURL()
        {
            return base.GetSocialCastURL();
        }

        public string GetSocialcastURL(string _apiURL)
        {
            return base.GetSocialCastURL(_apiURL);
        }

        public string GetSocialcastURL(string _apiURL, List<KeyValuePair<string, string>> _paramMessages)
        {
            //Get the URL from the base class method and append
            //the query params
            string _url = base.GetSocialCastURL(_apiURL);
            if (_paramMessages.Count > 0)
            {
                _url += "?";
                foreach (var item in _paramMessages)
                {
                    //appending each param key and value
                    _url += String.Format("{0}={1}&", item.Key, item.Value);
                }
                //String the last ampersand sign
                return _url.Substring(0, _url.Length - 1);
            }
            return _url;
        }
    }

The APIAccessor is the class which contains Business logic functions like GetUsers or GetMessages etc. It sends in its parameters as a List of Keyvalue pairs to the helper class which constructs the actual REST call URL out of the parameters.

 class WebServiceHelper
    {
        public string MakeServiceCalls(string _requestURL, NetworkCredential credential)
        {

            // Create the web request
            HttpWebRequest request = WebRequest.Create(_requestURL) as HttpWebRequest;
            request.Credentials = credential;
            // Get response
            using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
            {
                // Get the response stream
                StreamReader reader = new StreamReader(response.GetResponseStream());

                // Console application output
                return reader.ReadToEnd();
            }
        }
    }
    class APIAccessor:WebServiceHelper
    {
        //Creating the Helper class Instance
        SocialcastHelper helper = new SocialcastHelper();

        public XmlDocument GetUsers(string numberOfusers,string page)
        {
            XmlDocument Users = new XmlDocument();
            var serviceParams = new List<KeyValuePair<string,string>>();
            serviceParams.Add(new KeyValuePair<string,string>("per_page",numberOfusers));
            serviceParams.Add(new KeyValuePair<string,string>("page",page));

            Users.LoadXml(base.MakeServiceCalls(helper.GetSocialcastURL("users", serviceParams), helper.GetCredentials()));
            return Users;
        }
    }

The test code is below to display the name of users from the socialcast demo site.

 class Program
    {
        static void Main(string[] args)
        {
            var _xDoc = new APIAccessor().GetUsers("30", "1");
            foreach (XmlNode item in _xDoc.GetElementsByTagName("user"))
            {
                XmlNode name = item.SelectSingleNode("name");
                Console.WriteLine(name.InnerText);
            }

        }
   }

The Sample code can be downloaded here.