Category Archives: Uncategorized

Parsing Azure Blob Storage logs using Azure Functions

One of the ways AC and I track how many people are listening to the Microsoft Cloud Show episodes we put out is by using the logs created for Azure Storage.  These track the various requests for the mp3 files for each episode.  You have to turn on this logging for your account and once this is done then log files are written into the /$logs/blob folder of your storage account.  You can read more about Azure Storage Analytics here:


However, the way the storage logs are filed does not exactly make analyzing them easy in something like Excel.   They are logged in /year/month/day/hour/5min chunk folders/files in CSV format. (detail of what is logged is here).


Until recently our process for downloading the analyzing these was rather laborious and included:

  1. Download all the logs files.
  2. Use some PowerShell to squash all the thousands of CSV files into one big one.
  3. Use SQL Import tool to take the CSV and import each line into a temp table in SQL Azure
  4. Run a T-SQL script over the data to remove all the rows we were not interested in (we only wanted the logs for downloaded MP3s, not other assets etc.…)
  5. Mutate the data in various columns for better reporting. e.g. pulling important parts from the User Agent string and pulling an episode number out of the filenames.
  6. Finally inserting all the new rows into the final reporting Table
  7. Using Excel to report over the data

It was error prone, painfully slow at times and required all these things to be done each time we wanted new statistics.  At times we would go months without doing this and were flying blind on how many people were listening.  Not really very good.

What we really needed was an automated process!

Azure Function Apps to the rescue!

In March 2016 Microsoft announced Azure Functions.  They are lightweight bits of code that can be run in Azure when certain events occur, like on a timed basis, when an HTTP request is made and when a blob is added to storage.  What is nice about them is that you are not paying for a Virtual Machine to sit there mostly idle.  You only pay for what you use.  You can write them in a variety of languages and trigger them in various ways.  If you know how to write a console app in C# they are super easy to understand, same with a hello world app in Node.JS.

They looked perfect for what I wanted to do.  I wanted to read in log files as they were created, parse the content out of them & them log that to an Azure SQL Database.  From there we just attach Excel to the DB and go to town with graphs and pivot tables etc.…

Microsoft.Azure.WebJobs.Host: Invalid container name: $logs.


Sad smile

Problem! Currently it looks like you can’t trigger a Function off blob creation of a log file in the $logs container which is where the files are dropped 🙁  I don’t know if this is something that will change, but ill try and find out.

This means that for now we would have to do one of two things:

  1. Manually copy log files to another container periodically. Ok solution.
  2. Make another function that is triggered every 15 mins or so to copy the log files out to another container which would trigger or other function. Better solution.

I opted for copying the blobs on a schedule.  I wrote some pretty crude code that:

  1. Enumerates all the blobs in the source $logs container
  2. Checks if they exist in the destination container
  3. Copies them over if they don’t

I say this is crude because it doesn’t keep track of the last blob that it copied and therefore each time it runs it enumerates all the logs each time. This isn’t ideal and I will probably need to make it a bit more sophisticated in the future. But for now running it once a day shouldn’t be a problem.
Update:  Since I published the post I updated the copy code to check to see if it needs to sync files since the last time the function ran + 1 extra day (to be safe).  This means it wont scan all your logs every time.

So here is what we have end to end …


  1. Copy logs from $logs in one subscription to /showlogs in another subscription
  2. Process each log file as it arrives and put the data in the Database

Show me the code!

Ok so how did I do this.  Below is the two Azure Functions.  I have tried to comment them so you can follow along.

Copying log files from /$logs to another Azure Blob container.

Next, Processing those log files when they arrive.

I have put all the source for these two functions in a GitHub repo here:

There are two files for each function:

  • project.json – contains some dependency information for libraries that I used in each function
  • run.csx – the main azure function code

Hopefully someone else will find this useful!

Azure Functions provide a really handy and simple way to run code periodically.  If you are familiar with Node.JS, C#, Python or PHP you should go take a look at them.


Raspberry Pi, Flight tracking and Minecraft

Like most parents of young kids I have a son who is deeply into Minecraft. I also had a raspberry pi 2 sitting around and a friend mentioned that I could run a Minecraft server on it! Woohoo I thought. My son will think it’s the most amazing thing ever and I will be super dad! (Note: kids just like Minecraft, putting it on a raspberry pi is secondary to them it turns out and isn’t as cool a I thought it was).  Turns out its pretty simple, drop on an OS, in this case Raspbian, and then load up a minecraft server following one of the many guides available on the internet.  I got that all up running with a great sense of satisfaction and looking like superman, batman and a skylander all in one to my Son. #BadAssDad.

I’m also a bit of a space and aeronautics nut. And it got me thinking, I wonder if I can get ADS-B signals (most planes broadcast these) feeding to to help them build their coverage map. The raspberry pi doesn’t have a receiver that can do this out of the box, but you can buy USB sticks that do this on amazon pretty cheaply like this one from NooElec (the one I bought).  provides pretty simple instructions about how to load an app onto your pi that takes signals from the USB stick and feeds them to their site.  So simple i managed to follow it!  Mind you I was already getting intimate with the command line, PuTTY for the Minecraft install, so things felt a bit simpler at that point.

Once you get the whole thing up and running you get log into their site and see statistics about your feed, how many aircraft you have tracked … and generally feel good about helping to crowd source this data.

FlightRadar Screenshot

Oh … and it’s all kinda nerdy which gave me a sense of satisfaction too.

Here is a shot of the wee beast running.



Hola Microsoft

Back in November 2011 I said “Adios Microsoft” and left to start the US office for Provoke Solutions. I can’t believe it has been over two years since then.  Time has flown by!

Well all things come to and end and on the 10th of March I will rejoin Microsoft.
I have thoroughly enjoyed my time with Provoke.  On Nov 1 2011 I literally unloaded a car full of things and set up myself in an office in Bellevue as employee #1.  We have achieved some amazing things since then and I am personally very proud what we accomplished. It’s been very personally rewarding, challenging and character building. Running a small business is hard, but a lot of fun all at the same time. I was fortunate enough of having the backing from our New Zealand based team so I never felt like it was just me.

But now I am ready for my next chapter.

I am heading back to a familiar but different team from my last stint at Microsoft, the Office 365 team. I will be leading a team to help to make Office 365 a great place for developers, software vendors (ISVs), development houses (“System Integrators”) and others to customize, extend and build software for Office 365 and constituent products on-premises.


This opportunity was too good to pass up.  The chance to work on the worlds #1 leading productivity software and to help take developers and ISVs along for the ride! Couple that with a unique time in computing history as the paradigm shift of cloud computing takes hold and it makes for a very unique opportunity that I couldn’t resist.

They had me at “helping developers”.

The current story for developers building for the cloud in 365 is only in its infancy and I am very much looking forward to being involved and helping as much as I can.

I know lots of people out in the community, the SharePoint community especially, who are very passionate and vocal about the issues they are facing with development for Office 365 today.  I know fully well what many of the issues are, however, as always, I look forward to talking with as many people as I can and working on these issues together.  It needs to be a collaborative effort and it wont happen over night.  But it will happen.

I am really looking forward to the SharePoint Conference in Vegas next week!


Moving house on the internet and SEO

When I left Microsoft a couple of years back I also decided to move house on the internet. I started and made it my new location on the tubez (aka the internet).  I didn’t technical “move” house I guess as you can still find my old blog.

In fact that is the problem!

If you search for “Chris Johnson SharePoint” my new site is on the first page of Bing, but not at the top & it’s the 2nd result on Google.  However, my old blog is in the #1 spot in both search engines. Grrrr.

I didn’t want to delete my old blog, that seems like overkill and old posts still get a lot of traffic. There are 3 or 4 old posts that are knocking past 150,000 views which must mean someone finds them helpful. But given I have moved house I want people to find me at my new location. This issue has been confirmed by a couple of others who tried to find me and didn’t have good success.  Classic SEO problem some would say.

Unfortunately, I cant seem to “relocate” my old content to my new blog & put in a good redirection process that would be search engine friendly. My old blog is hosted with Microsoft and there are limited controls as far as I can tell.  So I think leaving it in place is my only option.

I have been making a concerted effort to try and get my new location indexed and ranking higher so people can find me. It has been working slowly.

One tool I heard about in the Startups for the Rest of us podcast was HitTail. (  I also saw that Andrew Connell was giving it a try too, so thought I would give it a shot.

It’s a very nifty tool.  You sign up (free trial) and then add a wee bit of JavaScript to your page and you are all set. 

What does HitTail do?

In a nutshell HitTail shows you what keywords (from your existing organic search traffic) you should target when writing content for your site that will increase your ranking.

Now this seems a bit counter intuitive at first, at least it did to me. Why would I write more content about things people are already coming to my site for that they found from a search? 

Well, the suggestions it makes are based on the fact that people are finding your site but that you could be doing a much better job of ranking higher for those terms. So it helps guide you on what to write about that will bump you up the ranking.

So I am trying it out.  I am reviewing the suggestions it makes and matching that up with topics I a) want to write about, b) have something useful to say about, and c) that I know people will find interesting.

At this stage it’s a trial to see how useful I find the suggestions, but I thought it was a handy tool to help with my internet home relocation.

If any of my readers are SEO experts I would love any other suggestions you have around my predicament.  Suggestions welcome! Get in touch on Twitter @LoungeFlyZ


New SharePoint Online / SharePoint 2013 version feature comparison matrix…

The always timely and astute Andrew Connell just alerted me to the new SharePoint Online / SharePoint 2013 feature comparison matrix that MS published.


Nothing too surprising … the E3 plan is the plan to buy still if you want a “full” online SharePoint Online experience. Nothing has changed in that regard.

Notable things you will notice missing if you don’t go with an E3 or higher:

  • Business Connectivity Services (BCS), External Lists etc…, BCS Web Parts
  • InfoPath Form Services
  • eDiscovery features, holds, exchange integration etc…
  • Office Pro Plus
  • Video Search
  • All BI features

(the list above is not exhaustive and are really the ones I noticed and felt were interesting)

What is of real note is what is missing from SharePoint Online all together. This isn’t news to anyone who has been playing around with the preview … but for a lot of people I speak to these are not well known.

The biggest thing (IMHO) missing are around WCM features. Things like Content By Search Web Part, Catalogs, Topic Pages, Faceted navigation and Image Renditions are all pretty critical new features in SharePoint 2013 that are not currently available online. (they might be turned on later on at some point no doubt). This is a bit of a disappointment for those looking to build out search driven web sites.

For on SharePoint 2013 on premises the biggest “ouch” worth mentioning in my view is again WCM. 

Pretty much all the new search driven WCM features are only included in SharePoint 2013 Enterprise.  WCM Catalog, Facted Navigation, Image Renditions, Multiple Domains, Topic Pages, Content by Search etc…

Another odd thing I noticed was that Site Definitions were marked as not being included in Standard.  This is weird and I suspect is wrong … but I have not checked it for myself yet.

Side Note: External Lists anomaly.  For those with a keen eye might have seen that External Lists are included in the Foundation edition. Where as a lot of the other BCS related features are not.  Reason: This is because of where the plumbing for External Lists sits.  When we implemented this (I was the program manager who owned the external lists feature for 2010) we changed the SPList object so that you could access data in an External List the with the same SPList object model calls as you would with a regular list. Because this is a fairly deeply rooted object in the OM it’s part of the Foundation edition … and therefore in the 2010 release of SharePoint a lot of the BCS engine and core plumbing was moved to SharePoint Foundation from SharePoint Server to enable this SPList OM over external lists scenario. Bit of background for those still reading 🙂


PS: the fact that MS have published this means General Avalability and announcements around SharePoint Online upgrades should be coming soon … hopefully Smile

Surface with Windows RT

Just pre-ordered a new Surface and really looking forward to getting it in a imageweek or so.

I am really trying to stay open minded about this device.  As you may know I am an MS kool-aid drinker for WAY back … but I have spent plenty of time with other devices too.  I used to have an iPhone 3G, 3GS and a 4. I have an iPad (v1) currently that is my couch surfing device of choice.

A lot of people are asking my why get the RT version (ARM) over the Pro (x86) in 3 months time.

I want to replace my iPad.  I am not looking for a laptop replacement right now. That is why I don’t care about running existing Windows desktop apps etc…

On top of that here are a few things why I think I will like the Surface better than my iPad:

  • Physical keyboard built into the screen cover.  I’m only one flip away from productively typing a blog post like this vs. feeling like a nonce trying to type it on the crappy on screen keyboard.
  • USB – plug in full size keyboard, mouse, printer etc…
  • Expandable storage
  • Multi-Tasking apps, snapped view of twitter while doing something else.

But who really knows right?!?! The proof will really be in the pudding and my intention is to update this post with my thoughts and views after having used it for a while.

Also ill post my “Will it blend” or whatever other iPad disintegration video I come up with.


Problems opening “Blocked” files in Word/Office 2013

Found an issue today opening word documents downloaded from the net or saved from an email attachment today.

When trying to open I would get the following error message:

“Word experienced an error trying to open the file”

8-20-2012 12-25-17 PM

Turns out Word was having issues if a file was “Blocked” i.e. downloaded from a untrusted source like the internet or email.

You can fix this by “unblocking” the file in the properties panel here:

8-20-2012 12-25-05 PM

Thrilling stuff I know … but hopefully I will save someone from the same frustration I had this morning trying to open a docx emailed to me.

I cant remember needing to do with with Office 2010.

UPDATE:  Tobias Lekman has a good post on this and how to turn it off once and for all in Office.  You can tweak the Protected Mode settings to change the behavior.  You can read about that here:


SharePoint’s new Plumbing, great for developers

As you probably have seen, yesterday saw the public unveiling of the new Office 2013 and SharePoint 2013 products and online services.  The SharePointsphere has gone nuts and everyone is blogging and tweeting as they find stuff out about the products.  Very exciting stuff!  Some people like Andrew Connell are doing a fine job of articulating some of the new stuff for developers and designers like the new App model and the new WCM capabilities.  There really is so much new stuff for us developers out there that I think it will take a good few months for it all to really sink in.

There is one part of all this that I am personally pretty excited about.  You may jump to the conclusion that its Apps … it’s not … you might think it’s the Marketplace … it’s not …

What am I excited about?


That’s right … I’m excited about plumbing … why? Well …

To enable these new capabilities like Apps and the Marketplace there has been A LOT of work done in the platform that makes up SharePoint.  Features like OAuth for integrating with APIs, new Client Side Object Model capabilities & Remote Event receivers.  Plus loads more.

These things all culminate in features like Apps … but also play a significant part on their own.  These are the platform building blocks that we as developers need to unlock a whole new slew of scenarios.

Take for example this simple scenario:
Programmatically uploading files from on prem to Office 365. 

You might want to do this for a multitude of reasons.  Possibly as part of a workflow that runs on prem that pushes documents out to partner sites you host in Office 365 for example.

Today this is possible in Office 365, but requires a bunch of extra work around authentication (like Headless Authentication)

With the addition of OAuth for in Office 365 you will be able to simply deploy an App to the 365 site that gives your code running on prem or in Azure the ability to seamlessly “log in” and use APIs like the Client Side Object Model without all the auth “goo” (see how politely I said that) you need today.

Or take for example the opposite scenario:
Programmatically pulling files down from Office 365 when they are uploaded

Today when someone/something uploads a file into Office 365 you don’t currently have a way to catch that “event” and push the document out externally to another system.  You can have event receivers, but code in the Sandbox cant talk to anything external.  So what do people do today?  They “poll”.  They have some code that sits and spins every minute or so looking for new documents … then they “pull” it when one arrives.  This is not a great solution for a variety of reasons, but its pretty much all they can do.  In 2013 you have the ability to have Remote Event recievers where SharePoint will call your remote end point when an event fires so you can respond.  No more polling required 🙂

Or take the scenario I demoed in the keynote at SharePoint Conference 2011:
Pushing content into SharePoint Online from an Azure based web application.

In that scenario we had the NetHope voting application we had developed and hosted in Azure pushing vote data into NetHope’s Office 365 based intranet.  We needed to use Headless Auth to do that.

The list goes on and on and on…

Just like how OAuth opens up authentication scenarios the Client Side Object Model (CSOM) and REST APIs open up programmatically callable functionality remotely.

In SP 2010 the CSOM only really covered Foundation data APIs for getting at things like List Data in sites. In 2013 that has been widely expanded to things like Search, Profiles & Taxonomy (personal favorite of mine).  This makes working with SP remotely so much easier than having to cobble together calls using a combination of CSOM, Web Services, FPRPC & REST.  So so much easier.  If you are building code that isnt .Net or JS then you can call the Client.svc service with REST/OData to call the endpoints manually.

Between OAuth, Remote Event Receivers and the expanded CSOM developers are unlocked to build a plethora of solutions that are either hard to do today … or impossible today.

That’s why I am personally excited about the plumbing.

Being able to build your solutions and monetize them through Apps and the Marketplace is extremely cool too … but I’ll save up another post for how I think people should be approaching how to decide what “style” off App to build (like Provider Hosted vs. SharePoint hosted vs. Auto Hosted).  Lots to think about and consider before jumping in there.

In the mean time take a look at some of the great SharePoint developer training videos available here:



Update: changed the title … i like this one better

Office 2013 graphics crash on my laptop

I have a MacBook pro that I run exclusively with Windows 7.  No boot camp, no OSX etc…

When I went to load Office 2013 today it all installed fine … but failed to load when I tried to open any apps.



I figured out (I cant say how) that its got something to do with Offices new graphics hardware acceleration support.

If you are into fiddling with the registry you can disable this via the following key:


(PS: if it doesn’t exist then create it)

Then create a DWORD value in there called “DisableHardwareAcceleration” and set it to “1” e.g.


Then Office apps would startup correctly:


Like I said above it might be something to do with my graphics driver and the card in the laptop.  Here is what I have loaded:


I hope this saves some frustration and pain for anyone else out there having this problem.

Update: @waldekm also tells me this has solved an issue he was having with Office in a Virtual Machine being sluggish or slow and sometimes crashing.  Turning off hardware acceleration seems to have fixed this for him also.