New Dropbox and Markdown Driven Blog!

This blog has actually gone through a few different transitions - starting off as a WordPress site, but quickly moving to a custom ASP.NET site back in 2012. Then I moved it over to the Ghost blogging platform, which I've been using up until just last week!

The Ghost platform is a really nice solution, especially if you're a big fan of Markdown - and who isn't?! However, for a while now I've been thinking about going back to my initial idea of a custom ASP.NET project which synchronizes Markdown posts from Dropbox. I wanted to leverage ASP.NET Core and Azure Functions, so I decided to start again rather than going back to my old code from 2012.

I want all my content in Dropbox, so I can write posts locally even without an internet connection (I commute on the train with a rather flaky connection!). In fact, even as I write this, I'm currently on a cross country train with no connection at all! I also prefer to use VSCode as my Markdown editor, because, well, it's just awesome!

Whilst I want Dropbox to be the 'source of truth' for my content - the site itself obviously can't query Dropbox directly every time a user hits a page. This would be horrendously slow! So I needed a persistent caching datastore which gets automatically updated when I make changes to my Markdown files in Dropbox. Initially I thought of Azure DocumentDB as this datastore, but when I looked, it was quite pricey - so I decided to just use an Azure SQL database, which is insanely cheap! I'm using Entity Framework Core to communicate with it. I then use Dropbox webhooks and Azure Functions to manage the synchronization from Dropbox.

Images are also managed locally in the Dropbox folder so that they are displayed in the Markdown preview pane of VSCode. As part of the synchronization, images get resized and uploaded to Azure Blog Storage for rendering on the website.

Before I continue, let's look at an overview diagram of the architecture. A picture, a thousand words, and all that! ...

Azure Functions

As you can see in the diagram above, I'm using Azure Functions for the Dropbox to Azure synchronization. Functions are a fairly new Azure service which is part of what's now becoming commonly referred to as serverless architecture. This doesn't mean that there are no servers, it just means that you're not involved with them. With Azure Functions, you're affectively just putting functions of code in the cloud, and Azure handles the rest. The functions have various different triggers and outputs. One trigger might be a webhook (in this case I'm using a Dropbox webhook). Another trigger might be a message being put on a queue, etc.

For my blog, I have four Azure Functions ...

Dropbox Webhook Endpoint

This Function gets triggered anytime I make a change to a post (ie. a Markdown file in Dropbox). Dropbox specifies that the webhook endpoint has to respond to Dropbox within 10 seconds. So rather than this Function doing any processing itself, it just puts a message onto an Azure Service Bus queue saying that there's been a change.

Synchronization Function

This Function listens for messages on the Azure Service Bus queue mentioned above. When it sees one, it then starts an incremental synchronization. This queries Dropbox for all the changes since it last ran an incremental sync. It does this by storing a Dropbox cursor in the SQL database each incremental sync. Then the next run will send the cursor from the previous run to Dropbox to query the changes since.

It then updates the Azure SQL database with any changes. As part of that, it also uploads any new images to Azure Blob Storage so the website itself can display them.

I also had to ensure that there weren't concurrent synchronizations if I made multiple saves to a Markdown file in Dropbox. To do this, I use Azure Blog Storage Leases. Thanks to Chris Anderson for this very useful tip!

Scheduled Synchronization

This Function executes a daily full synchronization in case the webhook incremental sync misses anything for any reason. Probably not required, but as it shares most of the code, it doesn't do any harm!

Keep Alive Function (obsolete!)

Well, I had originally written the description of the fourth Function as this ...

Being a cheapskate, and not wanted to pay a lot per month for an Azure Service Plan that supports 'Keep Alive', I have another Function which just pings my site on a schedule to stop it from going to sleep.

... however, that is no longer true as I've now upgraded to the Basic plan, which does support 'Keep Alive'. At the time of writing this is £42 per month, however remember that it's for the Service Plan, not just the webapp itself. A Service Plan is effectively a virtual machine where you can host multiple webapps. So I won't just use this for my blog - I'll put the .NET Oxford site and own my company's site (Everstack) on it when they're complete. Also, this plan supports other features, like HTTPS.

Speaking of HTTPS, I'm using a free Let's Encrypt certification which was very easy to install thanks to the Troy Hunt's very detailed instructional blog post.

Deployments with VSTS

For builds and deployment I'm using Microsoft Visual Studio Team Services (or VSTS for short), so that every time I push my changes to Github, VSTS will automatically build and deploy both my webapp and the Azure Functions to Azure. I'm becoming a big fan of VSTS, as it's so easy to set up, and you even get 4 hours free build/deploy time per month - so for my use-case, it's free!

I started off just using Kudo to deploy from Github when I pushed changes, however I found that VSTS gave me much more flexibility. I had lots of issues deploying my Azure Functions with just Kudo, which went away with VSTS. Also it became nicer adding things like Slack notifications when a VSTS build/deployment had completed.

The Code

The code is available freely on Github. It's worth pointing out that this is not a blogging engine / library; there are no promises of backwards compatibility as I make future changes. I wrote it for my own blog. However, I have written it so that settings are stored in environment variables or app settings; and my blog posts aren't part of the source code (as they're on my Dropbox). So feel free to fork the repository and use for your own blog however you see fit.

Dropbox Structure

I've set up Dropbox access by creating a Dropbox App. This means that the scope of access is limited to just that app's folder, not my entire Dropbox folder.

All the post metadata is stored in a Blog.json file in the root of this app folder. Below is a snippet from my version ...

[
    {
        "Title": "Fun at DDDSW in Bristol!",
        "Folder": "/2017-05-DDDSW-Bristol",
        "Route": "/dddsw-bristol-2017",
        "Status": "Published",
        "PublishDate": "2017-05-07",
        "Tags": "Conference|Meetups",
        "Featured": false
    },
    {
        "Title": "Blog Rewrite: Markdown and Dropbox Driven!",
        "Folder": "/Drafts/BlogRewrite",
        "Route": "/blog-rewrite",
        "Status": "Draft",
        "PublishDate": "",
        "Tags": "Blog",
        "Featured": true
    },
    ... etc ...

The directory structure looks like this ...

blog.json
2017-05-DDDSW-Bristol\post.md
2017-05-DDDSW-Bristol\images\myimage1.jpg
2017-05-DDDSW-Bristol\images\myimage2.jpg
AnotherPost\post.md
AnotherPost\images\someimage.jpg
... etc ...

I changed my mind a few times between doing it this way, or putting each posts' metadata inside its own folder alongside its content - but I decided in the end that I wanted all the metadata in one place so that if I change the structure, it's all in one file.

Editing on the go!

Given my posts are now in Dropbox as Markdown files, I had a quick look for a decent Markdown editor for my Android phone and tablet, and found MarkdownX. This is a really nice lightweight Markdown editor.

What sets this apart from the other choices is that you can swipe left or right to switch between editing mode and preview mode. I find this quickly becomes a subconscious flow to swipe to read, then swipe back to made edits, before swiping back to carry on reviewing.

Whilst most of the time, I write on my laptop in VSCode - this is ideal for making tweaks on the go when I'm not on my laptop.

Summary

There were many reasons I built this instead of just using an existing blogging platform. The main one though, is that it was a fun little hobby project. I now also have a platform which I feel much happier and more motivated to blog against, and have much more control than I did with Ghost.

Any thoughts or feedback is most welcome - both on the blog, or even the code itself!

Happy blogging! :)