Tomáš Kohl

How to make hourly billing work - if you must

2022-12-05T14:00:00Z

Hourly billing is a widespread pricing scheme in software development and beyond. Inasmuch as it's easy to reason about in accounting terms, it creates unhealthy incentives on both sides of the contractual relationship.

At its core, hourly billing is about buying and selling time spent doing something, not producing any specific outcome or result.

You pay your employees per unit of time, because you expect them to engage in a variety of tasks, and negotiating price over each incremental value would add too much friction.

It's different with vendors and contractors. Ideally, you would have a well-defined job for them, one that they could conceivably value-price and create a win-win scenario.

Value pricing requires that you can create surplus for the customer and still keep a healthy profit margin.

However, it is not a magic method that you could use everywhere and expect a profit for either party.

Hourly billing, if you must

Are there any scenarios where hourly billing is still preferable even with its fundamental flaws?

I would argue that large enterprise change programs would fit.

At their beginnings, there's often just a PowerPoint deck framing the CxO's vision, and a budget. No sane technology salesperson would value-price that.

If you as a vendor had an especially good relationship with the company, you could do proper discovery, break the scope down, and value-price the iterations.

Yet the budget is likely to remain the same, so why expend all the energy and not just go with the flow if there's nothing to be gained by being clever?

The contract is likely going to be signed as an hourly affair. Despite being the best of all possible wrong pricing strategies to apply, it's likely to cause you, the vendor, to under-deliver, miss the deadlines, and become a major nuisance to the CxO in no time.

How would you make hourly billing work there? Could you?

The four riders of budgetocalypse

Here are a few contributing factors that cause enterprise programs to fail when run by hourly contractors.

Incentives, incentives, incentives - rather than assume intentional wrongdoing by the vendor, simply consider the underlying motivation that the contractor has. The only way to "scale" their business and increase "profit" is to bill as many hours as the customer can reasonably be expected to sign off on. The contractor can't be clever and do the work twice as fast without losing half of their income.

Oversight, or lack thereof - a large-scale enterprise program may have dozens if not hundreds of contractors muddling through the backlog. Reviewing each timesheet individually is a non-starter. The company could do it if it had its employees at key positions throughout the teams, but do they? The only people motivated to cut vendors' wings sit in the purchasing department and are no longer involved.

Agile - Scrum is the default now instead of Waterfall, and that has further consequences. As Agile shifted the power balance to favor the developers' interests, it made it completely legitimate for them to say, "this is as much as we are able to in the next two weeks, and not a line of code more." Say what? Yes, you heard me, plus there's no appeals process. Add in hourly billing and brace yourself for timelines shifting into the future indefinitely.

Corporate inertia - it's a cliché to speak disparagingly about things taking forever in enterprise companies, and I won't. They have good reasons to be conservative: they have existing revenue to protect, quarterly earnings to report, and an army of employees to coordinate.

The problem is that these large change programs need to operate at Internet speed to have a chance of succeeding.

When they don't, decisions come late (or never) while the clock is always running. Always. And while a big company can assume a lot of extra costs and still prevail in the end, paying its contractors by the hour with all of these factors combined is guaranteed to accomplish only one thing: spend the budget initially assigned to the program.

I may accomplish a few other things too, of course. Just without any guarantees.

Traits of successful hourly billing arrangements

I've shown you a few ways in which an enterprise change program can fail. Even more so when contractors are running the show. What about possible mitigations, are there any?

It's helpful to start with the first principles. How to work with the vendor's motivation and try to align it with the desired outcome?

Since we are stuck with hourly billing, we have to change the equation to limit its downside. Here are some that I've witnessed make an impact.

Competition - a free market does not need moral values to function. If there's enough competition, it will create and enforce them as if by magic. (Yes, there's more to it, but this isn't an economics textbook.)

I would advise the CTO to be very intentional when hiring vendors to do the job and hire direct competitors. The point is not to promise the "winner" to take over the whole program. That would destroy the competitive environment and put the equation back where it was. The fact that they are not solely in charge is usually enough.

I've seen teams mixed from several vendors, and while there's some rivalry and chemistry to manage, they focus on the mission a lot more than teams manned exclusively by a single vendor.

Oversight, for real - do the reverse of the anti-pattern described in the preceding section and staff your people at key positions. There's no way around this. Nobody is as invested in the future of your company as your own employees, assuming you have a functional compensation scheme that encourages long-term thinking.

When this is not possible, e.g., because of headcount limitations, consider hiring yet another consultancy whose job will be to oversee the program. Yes, everyone will hate them. So what! The point is to have someone on your side to keep the vendors in check. Remember, they are billing by the hour!

Before you ask, no, I don't think these consultants should be billing hourly.

Lessening the Agile impedance mismatch - as demonstrated, enterprise companies typically do not move at Internet speeds. The change program could be the best operating unit in the company, but if it stalls when interacting with the others, it can't deliver on its promise.

The approach that I've seen working best is to give the leaders as much autonomy as possible so that they don't have to consult (and ask for decisions) the parent entity more than once a quarter. Even better if the unit is a standalone entity, owned by the enterprise but operated independently.

Since this topic deals more with how the program interfaces with the stakeholders rather than vendors, I will end here before going off-tangent. Solving it is crucial for the program's success regardless of how the vendors are paid. If hourly, then even more urgently so.

Conclusion

For better or worse, hourly billing is here to stay.

Even if broken, it can be the only pricing tool available, for example on large enterprise programs.

Its inherent flaws and limitations cannot be fixed. You have to tame them, and you can.

Having said all that, if you can avoid hourly billing as a client or vendor, I encourage you to do so. It's a tool of last resort that requires a lot of effort to manage. You could spend that effort creating something valuable instead.

Is Agile a welfare program for developers?

2022-12-01T12:40:00Z

How do you make developers comfortable?

There are free lunches, game rooms, and massages. Consultants can help you craft the most comprehensive perks program in town.

Or, you could implement Scrum.

In the paragraphs that follow, I will argue that Agile has become an overfunded welfare program for developers.

And since this is going to ruffle a few feathers, I will limit my indictment to Scrum projects in the context of enterprise software development. This represents the bulk of my professional experience. Please read the text below through this lens.

Also note that when I say Agile, I mean mostly Scrum.

A bit of history

Back in the Waterfall era, project managers ruled supreme.

They set deadlines, herded their flock of developers toward milestones, and had the CxOs on speed dial.

Did the projects run over time and budget? They sure did! But you know what? We got real work done!

One explanation is as follows. There's always tension between the client (the higher-ups) and the producer (developers). One side wants an X amount of work in a Y timeline while the other side can deliver a Z amount. And they struggle to assign values to these variables.

In the Waterfall environment, this power dynamics favored the client's side. Overtime used to be common as deadlines had to be met. The developers, when asked to estimate, would argue in man-days, and the project manager would inevitably challenge them. A compromise would be forged, in which the developers would often draw the shorter straw.

This was hard on developers. So, they invented Agile.

Agile: lofty and idealistic on paper, toothless in practice

Recall that Agile was invented by developers, for developers.

It was sold to the business folks differently, of course. As a way to deploy working software more quickly than the standard Waterfall allowed.

And while I know nothing about the state of mind of the Agile inventors, I don't have to. Just looking at how the power dynamics have changed tells the story.

Case in point

Consider any sufficiently ambitious program run by an enterprise so that it can finally enter the 21st century. Any "digitization" program would do.

Bonus points if it's a "startup" launched and wholly owned by the enterprise.

How is that program staffed? If the enterprise isn't a software house, chances are excellent that 90% of the staff are contractors. The company doesn't have enough developers in-house, and won't need them once the program finishes.

Which, of course, it never does. Because guess what? The methodology sold to the higher-ups as a surefire way of joining the 21st century was Scrum.

The enterprise has no agility in its DNA and the project runs as an isolated island embedded in the traditional hierarchy. The vision is handed down to contracted "product owners" who dutifully translate it to "user stories" and take it by the developers in the "refinement session."

There, the team evaluates each story and assigns it a point value on a scale between 1-13. "1" is like "yawn, I could do this now if I didn't have to be in this stupid meeting" and the "13" tells the PO, "eff you, break it down even further, there is too much unknown here."

What's the product owner to do? He could push back a little. But, the Scrum Master is there to ensure that The Process is adhered to, and in this process, the developers are always right. Always.

The result is that developers develop at their own speed, and if they so desire, they can drag the backlog items over several Sprints and nobody will punish them.

Now imagine you are paying someone to fix up your house, and they do this to you.

The counter-argument

Agile has also produced numerous benefits, and not just for developers.

It "normalized" relationships between developers and business folks. You don't need as many intermediaries as before (good-bye, business analysts!) and as developers got better at soft skills, they get to see the suits more often.

The 24/7 deployment story can also be attributed to Agile. You could argue that the technical progress in the CI/CD tooling (and the cloud) made it possible, and I won't disagree. What I see in the Agile manifestos is the desire to deploy as frequently as possible (i.e., at the end of the Sprint in Scrum), and we do that now.

Much better than once a quarter if you ask me.

The indictment

Agile has changed the power dynamics and given developers too much power.

My intent isn't to indict its inventors. I would like to inspire you to think about how we could course-correct instead.

A situation where the development team can dictate its tempo is not healthy. That they alone define the deliverables for the next iteration is not healthy. It's bad for the people paying the bills, and it creates incentives for the developers to become lazy.

Worse yet, it's a terrible fit for programs run by enterprises. Their internal processes have a huge impedance mismatch when confronted with Agile values, and so the implementation can only result in the most bastardized version possible.

How to evolve Agile

In my practice, I never use Scrum and have explained why (and what we do instead) on my agency blog. To be clear, the projects I run are nowhere near in size of the enterprise Scrum programs I talked about above.

However we address this, the key thing is to correct the power dynamics.

The Waterfall was bad because it made developers overworked cogs with no say about the process. It barely worked then and would be laughable now when you consider how long it took to ship anything.

Agile can be better but mostly isn't, because its implementation swayed the balance too much in the developers' favor.

Where is the ideal middle ground, and is there one?

I propose three areas for analysis:

adding more rigor to the planning
adding "skin in the game"
adding oversight

Planning with purpose

Agile favors working software over comprehensive documentation. And yet, you couldn't deliver anything useful if you had no inputs and no customers.

Having seen too many undercooked "user stories," I would argue that good, rigorously analyzed documentation makes developers happier. Developers hate uncertainty.

You don't have to go all the way back to the Waterfall days and try to describe the entire system up-front. I am in favor of iterative development. By all means, release something useful early and test it on customers.

Consider documenting your iteration rigorously before development, however.

By thinking through your use cases and documenting them, you can spot and resolve errors in your thinking early. Doing so on paper is cheaper than reworking already deployed code.

This creates the first condition to correct the power balance. It gives developers more clarity about what they are about to do.

Adding "skin in the game"

Agile is a little too optimistic about people's motivation and assumes everyone is equally interested in the outcome. I think adding carrots and sticks is necessary to make it so.

Whichever way you do it, aligning compensation with performance is necessary.

Before you object, yes, I am aware that measuring developer performance is not a trivial subject. It's also not specific to Agile, or any other methodology.

My point is to highlight the importance of "skin in the game" for the success of any endeavor, including software development projects.

Trading points may not suffice.

Whatever compensation scheme you have in place, I would consider not exempting your Agile teams from it. When people are incentivized to perform at their very best, you create a second condition to fix the power balance.

Oversight

This point is specific to enterprise projects that are largely staffed with contractors.

While there's no inherent guarantee that your employees have the best interests of your company at heart, contractors usually care less.

Giving contractors a blank check to run your project using Agile can lead to cost and deadline overruns that rival those of the Waterfall era.

I witnessed large programs where the organization didn't have even its own CTO! Everyone besides a few key product people was contracted. In such an environment, there is no oversight and no accountability.

Having core engineering staff is a must. They shouldn't be the "architecture astronauts" and prescribe solutions to the teams without having to deliver. On the contrary, I would put one or two internal employees on each team so that the technical decisions the team makes are aligned with the company's overall technical strategy and values.

In a broader sense, oversight and controls should be built into the process regardless of who is doing the work (employees or contractors).

Scrum has biweekly "reviews" and "retrospectives." I would argue that two weeks is too short a period - it's a microcosm of tactical decisions.

Your process should have checkpoints that review the process and outcomes on higher levels - maybe every 6 weeks to review how complete features are being (or not being) shipped, and once a quarter to check on major milestones. These are just examples - choose a frequency that best fits the project's timelines and your velocity.

Summary

I criticized Agile in light of how it's been implemented, especially in Scrum and specifically in the context of enterprise software development.

In the process of writing, I realized I chose a topic that's quite big to chew on; I admit that. Many themes covered above deserve analysis on a much deeper level. I've only touched the surface.

The first version went off the rails as I switched my focus to the evils of hourly billing and unsuccessfully tried to tie it together with the deficiencies of Agile as I see them. I removed it and will publish a rant against hourly billing separately.

All things considered, I am not against Agile in principle but think that we should evolve it pragmatically and that in the enterprise context, it needs more adult supervision than it often gets.

You should probably just use AWS. Or Azure.

2022-11-25T14:20:00Z

The cloud marketing departments spend gazillions of dollars to lure teams to 'migrate to the cloud'.

AWS is the default buying choice like IBM used to be.

A contrarian opinion arguing against the cloud then naturally generates lots of waves.

But, does it make sense to be a cloud contrarian?

The case against the cloud

Basecamp has recently announced that they'll be leaving the cloud.

Their stated reasons are monetary ("paying over half a million dollars per year for database (RDS) and search (ES) services from Amazon") and also based on values ("[it's] tragic that this decentralized wonder of the world is now largely operating on computers owned by a handful of mega corporations").

Basecamp operates on such a scale that owning a rack in a datacenter might yield some financial savings. Their devs can surely handle a few new sysadmin tasks.

My friend Karl Sutt argued against the cloud from a more general perspective. I will steelman his argument before offering my take.

He based his critique on the following observations:

Cloud is expensive to operate and maintain - not just in terms of the direct spend but also when you include the opportunity cost.
You are not Netflix, so you don't need to replicate their setup for your non-planetscale app.
There is too much choice of cloud services you can use, and we developers can easily overcomplicate the design when a simple tech stack would do.

Cloud is too expensive

How easy is it to overpay for computers rented from Amazon? Too easy:

Any sufficiently complex enterprise backend will at some point use services that will create very unpopular line items on the monthly invoice.

And your backlog likely contains user stories such as "As a CTO, I want to slash my cloud spend by 50% to get the CFO off my ass."

In fact, the unavoidability of surprise AWS bills has created a cottage industry of AWS cost-reduction consultants. Given they exist, the cloud must be expensive, at least for some.

You don't run Netflix

Your startup idea might become the new Netflix. Before your SaaS metrics prove that trajectory inevitable, you might not need microservices. Or multi-region redundancy. You'd burn through your seed round before you'd even have a chance to validate your idea.

The tragedy of the hotel breakfast buffet

One factor contributing to the overly complex (and expensive) cloud application architectures is the plethora of services you have at your disposal.

The cloud is not just VMs. You have managed databases, message queues, CDNs, and a ton of other things.

We developers are hyper-curious beings, which is a polite way of saying that we like trying out cool shit! AWS (and Azure and GCP) makes it way too easy.

These managed services do enable computing on demand, and they also enable unnecessarily complex architectures.

When there's little oversight over the cloud spend, we developers surely have a tendency to overeat!

The case for the cloud

I think Karl's arguments are worth another look, if only because the alternative (running or renting bare metal servers) can also be costly, expensive, and has a non-zero opportunity cost, too.

Since this question cannot be conclusively answered the same for all audiences, let's focus on the needs of an imaginary B2B SaaS startup that is bootstrapped.

Being bootstrapped means having to watch over your budget with hawk's eyes. It's your money, after all.

It means you are focused on finding (and proving) your product/market fit quickly, thus finding a way to profitability quickly.

Assuming that you have one or two developers who are equally skilled in both cloud-native as well as "handcrafted" architecture, which way should you go - cloud-first, or rent (or buy) your own servers?

Comparing direct costs

You could rent dedicated servers essentially for free.

For instance, have a look at the server auctions at Hetzner. At the time of writing, I could get a box with Intel Core i7-6700 and 64 GB RAM for about €40 per month.

That's less than I would spend on a company phone plan for one employee. As a business owner, such an expense is invisible to me.

And yet, one or two servers in this configuration could power my app for a year or two without breaking a sweat.

Try configuring a similarly specced machine in the cloud, and you're likely looking at a monthly bill that's upwards of $300.

However, this comparison only makes sense if you're migrating apps from your datacenter to the cloud, and utilizing VMs is the first step in the process. You would not necessarily make the same decision when starting from scratch. More on that later.

And yet, you are still probably going to run your app in the cloud for free.

That's because the cloud vendors really want your business.

Consider Microsoft for Startups. You can get up to $150,000 in Azure credits to develop and operate your app, not to mention Office and Visual Studio licenses and other incentives.

Once you outgrow the program, you won't be paying list prices, either. Big customers have enterprise agreements and don't pay anything near the retail prices.

That said, yes, you could at some point reach the same conclusions as Basecamp. I am not arguing that the cloud is always cheaper. When we limit our inquiry to the monthly cloud bills alone, it may not be. For instance, if your startup does not qualify for the incentives, your financial decisions will be different.

Comparing opportunity costs

My conclusions differ from Karl's as we consider the opportunity costs.

Karl wrote:

[T]here is productivity and opportunity cost from having to understand, deal with and work around the idiosyncracies of complex cloud platforms. This effort is much better spent on delivering value to users.

He proposes to put your customers' needs above all else and to achieve that, do the following:

Optimise for operational simplicity - by choosing a deployment target that's easiest to maintain and operate. He mentions Heroku, Render or Fly as an alternative to AWS or GCP, which is debatable - Heroku is neither cheap nor easy to operate once you outgrow the cheapest dynos. I have no experience with the other two.
Optimise for internal simplicity - I think he argues for the simplest possible architecture here, proposing to use "boring technologies" that are well understood.
Focus on delivering value to your users - having this mindset should drive all technical decisions and you should evaluate every choice not on its technical merits alone but primarily on whether it adds the most value to the people using the product.

I won't argue against any of that since I share his beliefs. I differ in how I would realize them.

Recall that in our example, we're trying to find a product/market fit for our bootstrapped startup. Delivering value to users is our job #1.

Therefore, I would rather direct my team's energy toward customer development and shipping features quickly than toward building our own infra.

Even as operating your own LAMP stack server, for example, is a well-documented proposition, there are adjacent concerns that you have to take into account, such as security, regulatory compliance (hello, GDPR!), reliability, disaster recovery, etc.

Cloud vendors do a remarkable job of taking care of these mundane details (for a fee).

You might find out, for instance, that serverless architecture is a good fit for your use case. Instead of running a full VM with all the overhead of following the security bulletins and applying patches on the regular, you could just deploy to Azure Functions or AWS Lambda, and only worry about your product's features.

Indeed, my main case in favor of deploying to the cloud by default rests primarily on the extra abstraction layers that the cloud vendors offer on top of mundane VMs.

It's not that you couldn't run your own serverless stack on your own hardware. Why would you spend your precious time on that, however, when your job #1 is finding and satisfying your first customers?

Finding the right answer

Since this polemic runs on a very high level, I won't pretend to have answers for every scenario.

There are indeed situations when running your own VM (or a bare-metal server) may be preferable to renting machines from Amazon (or Microsoft).

To avoid the "overeating trap", I would propose aligning the incentives of the technical team with the organization's goals and values.

There are such things as budgets, and the CTO should enforce a rigid discipline with regard to cloud spending.

I've seen teams run completely amok having the right to power up an arbitrary number of cloud resources at will. That's not something a bootstrapped startup can afford. And a CFO at a Fortune 500 company would also come down knocking on doors and asking tough questions, eventually.

When comparing your choices, I would propose the following metrics:

Time to market - time is your most valuable resource besides your people. How can you ship stuff most quickly?
Time to change - you'll need to make a lot of decisions and make a lot of changes. Some of them will impact your code and require new resources, e.g., a new type of database. How quickly can you provision these? And once you don't need a particular resource, how easy is it to retire it and stop paying for it?
Time to onboard - how easy is it to onboard new developers on your stack? This is where "AWS is the new IBM" comes from. Pretty much every developer knows AWS these days, at least "kinda, sorta" (AWS is huge).

Figure these out, and only then compare the monthly spend. Chances are that it won't matter much in the first few years of your startup anyway.

Your ability to attract paying customers and find a path to profitability will. Make sure your technical architecture is 100% aligned with this.

How to publish a static website to Azure Static Web Apps

2022-07-03T15:20:00Z

Introduction

With a statically-built website, you have a variety of options when it comes to hosting. You can self-host on your own VPS, or use Netlify, Cloudflare... the world is your oyster. On Azure, you can try Azure Static Web Apps. Let's see how it works!

What's Azure Static Web Apps

The primary use case is to build and deploy your app from a code repository like Github. As soon as code is merged to main, a task runs, and your changes are live. We did this in the Heroku heyday.

You can use Azure Static Web Apps (SWA from now on) in various ways.

The simplest would be to host plain HTML files with no build step at all.

Or, build your HTML using a static-site generator (SSG from now on) like Hugo or Eleventy.

Or, build a SPA app with React, Vue, etc.

If your app is not really "static" and needs dynamic content, that's where Azure Functions comes in.

We might explore that in a later article, but for today, we'll skip this. Suffice to say, Azure Functions is "married" to SWA and provides the "A" (=API) for running JAMstack apps on Azure.

Why use Azure Static Web Apps for a static site

You should consider SWA for your static site when the following features appeal to you:

automatic deployment from a source code repository
an option to add dynamic capabilities (APIs) to your site in the future

If neither is a concern and you prefer to deploy your site manually (with a CLI), you can always use Azure Blob Storage with Azure CDN and call it a day. SWA isn't set up for deployments that do not originate in a code repository (but there are workarounds, of course).

Another good argument is that for non-planet-scale blogs and websites, you can deploy to Azure Static Web Apps for free. That includes a free, self-renewing SSL certificate (by DigiCert, not Letsencrypt as is customary elsewhere.)

Should you need more than 100GB of traffic per month, you can switch to a paid plan and get a global CDN as an additional perk (currently in preview).

The setup

The official quick-start guide assumes you've got nothing: no code, no domain, and no content.

That makes it easier to explain the basics, I suppose. Most of you already have a website or a blog, however. I will tailor this guide to those migrating an existing site instead of starting from scratch.

Create the Azure resources

First, let's create the Azure resources and get your site up and running. We'll do this on the Azure Portal, but you can also use a CLI.

Create a resource group (or pick an existing one), then proceed to create the SWA. There's a simple dialog where you pick the subscription and resource group, choose a name for your SWA, the hosting plan, location, and deployment source.

You could pick Github right away and go through the auth flow. It is not necessary, however, as you will see shortly.

After Azure creates the site, navigate to it and copy the deployment token. You'll need it for the Github workflow. Also, take note of the auto-generated URL for your site. We will create a custom domain later, and we will need the URL for it.

Create the Github workflow

Following the official build configuration guide will get you 90% there. Here, I am simplifying it even more under two assumptions:

since your site is 100% static, you don't need to worry about building the APIs,
you don't need pull requests as you publish directly to main. If that's not the case, you can follow the official guide to incorporate PRs into your workflow.

We are using the official Github action for deployment to SWA.

Create an empty file in the folder .github/workflows of your repository and give it a name, e.g., azure-static-web-apps-myblog.yml. Copy and paste the following, then edit output_location and app_build_command to match your config.

name: Azure Static Web Apps CI/CD

on:
  push:
    branches:
      - main # You can change it to whatever is your deployment branch

jobs:
  build_and_deploy_job:
    runs-on: ubuntu-latest
    name: Build and Deploy Job
    steps:
      - uses: actions/checkout@v2
      - name: Build And Deploy
        id: builddeploy
        uses: Azure/static-web-apps-deploy@v1
        with:
          azure_static_web_apps_api_token: $ # This is the deployment token for your SWA site
          repo_token: $ # Used for GitHub integration
          action: "upload"
          app_location: "/" # This will typically be the repository root
          api_location: "" # No API here, therefore
          skip_api_build: true # skip API build
          output_location: "_site" # Modify this to where your SSG places the built HTML - could be `dist`, `build`... check your config
          app_build_command: 'yarn build' # Not necessary since the action will pick it up from 'package.json' but I like it explicit

Before running the workflow, make sure that you put the secret AZURE_STATIC_WEBAPP_DEPLOYMENT_TOKEN to your "Actions secrets" of your Github repository.

That's it! Push to main and observe that your site will be built and deployed to Azure.

Adding a custom domain

I am assuming you already have a custom domain for your site. If not (e.g., you run it on Github pages without a custom domain), you can buy one and use it on SWA.

You have two options: keep your existing nameservers, or use Azure DNS.

The first option is often preferable as it minimizes the transition period from your old nameservers to Azure. However, it comes with a few caveats.

When you want to run your site on an apex domain (mydomain.com), your DNS provider must support the ALIAS record since SWA doesn't give you a fixed IP address.

A CNAME would also work, but some providers won't let you point the apex domain to an URL. If you run your site on a subdomain (e.g., blog.mydomain.com), you can simply create a CNAME for blog and point it to the URL Azure generated for your site.

I wanted to use an apex domain, and my provider did not meet these requirements, so I moved my DNS records to Azure.

The following section only applies to you if you are in the same boat. If you can set up the correct DNS records at your current provider, do that instead!

Setting up Azure DNS

Create a new resource of type DNS Zone in your resource group.

Then, navigate to your current DNS provider and copy all records to your new Azure DNS zone. You will do so via the "+ Record set" menu.

You'll want to make sure (REALLY make sure) that you've copied ALL of the records you need. You don't want to leave out the MX records, for example, and stop receiving emails... I guess 🤷‍♂️ (HELLO, Inbox Zero!)

Before continuing, switch your nameservers to Azure.

I wish I could postpone this step, but Azure wouldn't let me add the custom domain until I did so. I'd imagine it could have something to do with them trying to create an SSL certificate for you, but I could be wrong. If you know, please let us know in the comments below!

Connecting the custom domain

If you're using Azure DNS, it's now simpler as Azure will do the domain validation for you. Otherwise, you'll create a few TXT and CNAME records at your DNS provider.

Navigate to your SWA and select Custom domains. Then, add one (for subdomains like blog.mydomain.com) or two (for apex domains that include the www subdomain).

With a domain hosted in Azure DNS, you'll be done sooner than if Azure has to validate your domain. When I did that (before ultimately transferring my nameservers to Azure), it took a long time - 15, maybe 20 minutes - before Azure picked up on the verification DNS records.

When all is done and green, I would pick the default domain. For a subdomain like blog.mydomain.com, that's an easy choice. When using an apex domain and www, choose one or the other depending on which one you prefer. Azure will then redirect requests to your default domain, which is what you want.

Verifying your setup

If you have migrated your nameservers to Azure, this will take hours until the DNS changes propagate.

A good way to test is to look at the SSL certificate. If you've used Letsencrypt before, your site is now protected by DigiCert. As long as you see the old certificate, you're still looking at the "old" site.

Conclusion

I have done this migration for my agency website, and I managed it in a single afternoon.

It's not difficult at all; just a few gotchas are waiting for you down the road.

For me, the main benefit is the automated deployment once I push to main. I preview all changes locally, and I don't have an editor who would go through my copy and improve it.

If that were the case (e.g., a hired copywriter would submit a PR to me), I would add the pull requests to the mix. I have not yet tested this, but Azure supposedly generates preview environments for your PRs!

That's something I might look into very soon, as well as adding the Azure Functions into the mix.

How to set up automated deployment of a Linux function app on Azure Functions Consumption plan

2022-06-08T15:15:00Z

Introduction

Azure Functions is a feature-rich platform-as-a-service (PaaS) offering of the Azure cloud, competing with AWS Lambda in the serverless space. It is particularly well suited for event-driven systems and, by its design, encourages a modular, decoupled system architecture, e.g., by way of micro-services.

About serverless

The "serverless" moniker is both appealing and misleading.

The appeal is straightforward: only care about your code. The platform takes care of the runtime and dependencies, among other things.

The misdirection lies in the moniker itself. For one, serverless should not be equated with zero thinking about infrastructure and deployment. It also presents no threat to the careers of DevOps professionals.

Think about serverless the way you would think about renting a serviced apartment. It comes with extra features you won't find in a standard apartment, e.g., there's weekly cleaning and no extra charge for a plumber fixing the pipes.

Alas, there are also things you can't do. A serverless platform puts hard limits on how you can configure the system, and when there's a feature that's missing, well, you are out of luck - no sudo apt install $package for you.

For a great many applications, the benefits outweigh the costs.

And to achieve maximum benefits, it's worth investing in your CI/CD pipeline. After setting up the automated deployment of your serverless app, you aren't really thinking about servers at all.

Why you should read this tutorial

I've gone through multiple code samples and tutorials, looking for guidance on configuring the CI/CD pipeline for an app running on Azure Functions. There are many, many out there - Microsoft Docs are excellent and are usually my first place to inquire.

That said, given the various tiers and plans Azure Functions offers, not every tutorial works out of the box on every plan.

The most prominent entrance to the Azure Functions world is through the Consumption plan since you are not being charged any flat fees regardless of usage but only for when your code actually runs. It is indeed this plan that I chose for my experiments and ended up a little bruised along the way.

This tutorial/explainer is tailored to fellow hackers setting up their apps on the Linux Consumption plan. It is in many ways generic in that the principles can be applied to the other Azure Functions plans, though. I highlight any solutions specific to the Consumption plan.

When (not) to worry about this

I would not worry about this in the initial stages of a small-ish project where you are the only developer.

You can always create resources manually in Azure or via Azure CLI, and deployment from VS Code or Visual Studio is just a few clicks away.

The time to think about automated deployment is when another developer joins the team or when the infrastructure becomes stable, and you get tired of manually deploying several times a day, whichever comes first.

By "stable infrastructure", I mean: you have figured out which database you need, you have created other supplemental resources like the Service Bus namespace, and now you spend most of your effort on writing the application code.

Pre-requisites

I assume some experience developing on the Azure Functions platform and familiarity with general Azure topics. This article covers setting up a deployment pipeline on Github; if you host your code elsewhere, e.g., on Gitlab, you'll have to adjust the Github-specific bits.

Furthermore, this article addresses a specific scenario for a Functions app running on the Consumption plan on Linux. If your app runs on Windows or on a Premium or App Service plan, then some of the quirks and limitations addressed below do not apply to you.

The take-away promise

After reading this tutorial, you will learn about:

how to understand the ARM templates and how to use them to describe the resources you want to deploy to Azure declaratively
how to deploy the resources from ARM templates with the Azure CLI
how to use Github workflows to trigger deployments based on repository events and on-demand

My sole goal is to empower you to set up your own deployment pipeline for your very own project or app in Azure Functions.

About this example

We'll go beyond "Hello world" and look at the automation pipeline of JComments, my Open Source project aimed at publishers using the JAM stack.

It provides commenting functionality to static websites. One API is used at build time to bake the existing comments into the generated HTML. The same API is then called by JavaScript to fetch any comments posted after the page was built. Another API processes incoming comments from web pages.

The project is written in TypeScript and runs on Node.js, which is one of the languages supported by Azure Functions. It uses a Postgres database, and when deployed to Azure, it can take advantage of the managed Azure Postgres service. A few event-driven use cases utilize a message queue, which in Azure means using the Service Bus.

The architecture is pretty vanilla, so the chances are that you'll be able to derive learnings from this article to apply to your own app and its setup on Azure.

The setup

The code lives on Github, which also provides the CI/CD pipeline.

You'll find the workflows under .github/workflows, and we'll take a look at them at the end of this tutorial.

The ARM templates and scripts are hosted in infra/azure. We will cover these first.

It's common to set up your resources manually at first - I did that, and likely you did, too. Creating an ARM template manually from scratch would be an exercise only a masochist would enjoy. Even taking an exported template and making a few tweaks would usually take a few commits and failed deploys to get right.

The first step is to export the ARM templates of your resources so that we can edit and generalize them.

Preparing ARM templates for automated deployment

Navigate to the resource and look up the link "Export template" under the menu item "Automation," just above "Support + troubleshooting."

Click it and wait until Azure generates the template, then save it somewhere where you can open it locally with your favorite editor.

The exported template will have a lot of specific bits you would want to parametrize if you were to use it for deployment to another resource group or subscription. In particular, you'll want to think about:

naming conventions, which are help organize resources by a common factor such as environment or location,
configuration parameters, esp. if they could be different from one environment to another
dependencies on other resources, which can go either way (consumed or provided or both)

Let's have a look at the ARM template of an App Service Plan when exported:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "serverfarms_ASP_jamcommentsstaging_9cbf_name": {
            "defaultValue": "ASP-jamcommentsstaging-9cbf",
            "type": "String"
        }
    },
    "variables": {},
    "resources": [
        {
            "type": "Microsoft.Web/serverfarms",
            "apiVersion": "2021-03-01",
            "name": "[parameters('serverfarms_ASP_jamcommentsstaging_9cbf_name')]",
            "location": "East US",
            "sku": {
                "name": "B1",
                "tier": "Basic",
                "size": "B1",
                "family": "B",
                "capacity": 1
            },
            "kind": "linux",
            "properties": {
                "perSiteScaling": false,
                "elasticScaleEnabled": false,
                "maximumElasticWorkerCount": 1,
                "isSpot": false,
                "reserved": true,
                "isXenon": false,
                "hyperV": false,
                "targetWorkerCount": 0,
                "targetWorkerSizeId": 0,
                "zoneRedundant": false
            }
        }
    ]
}

What you would want here is not to have the resource name hard-coded and (optionally) to extract the SKU and properties to parameters that can change independently of the template, e.g., from one environment to another.

Including a parameters template is a convenient way of achieving this. You can also supply parameters from the command-line for any values you don't want to be persisted in your code repository, e.g., secrets.

A brief interlude about the naming convention

You'll want to think about the naming conventions to use and take into account aspects of your infrastructure that won't change as well as those that might.

For example, you may want to make the location of the app configurable and to use multiple environments that are deployed separately and may have different sizing profiles. In our example, we'll have staging and production. As you will see, we'll be able to add another very easily, perhaps development.

The naming convention I'll use here will use jcomm- as the leading prefix for everything. The next segment will be the environment shorthand, staging- for the staging environment and prod- for production.

The final segment if the naming prefix will be the location. Here, we'll be using "East US" and shorten it to eus- for the prefix.

For each resource we'll pick a suitable abbreviation. For an App Service Plan, let's use asp. That's also the only part that the template will decide for itself, so to speak.

The big advantage of doing this is that you will immediately know what's what at first glance when you explore a resource group in the Azure portal. It's a decision you'd make at the beginning and then use consistently.

Pick whatever pattern that suits your needs best. Perhaps the location isn't important to you, or it won't ever change. Whatever. The point is to identify the commonalities as well as differences.

The set up, continued

To accommodate our requirements, the first part of the template will morph into this:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "prefix": {
            "type": "String"
        }
    },
    "variables": {
        "componentName": "[concat(parameters('prefix'), '-asp')]"
    },

What about the sku and properties? Let's extract them to the parameters file and save it to a file. Later, we may want to merge the function app and its App Service plan to a single deployment template and do so for their parameters as well.

The parameters file then reads as follows:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentParameters.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
      "servicePlanSku": {
        "value": {
            "name": "Y1",
            "tier": "Dynamic",
            "size": "Y1",
            "family": "Y",
            "capacity": 0
        }
      },
      "servicePlanProperties": {
        "value": {
            "perSiteScaling": false,
            "elasticScaleEnabled": false,
            "maximumElasticWorkerCount": 1,
            "isSpot": false,
            "reserved": true,
            "isXenon": false,
            "hyperV": false,
            "targetWorkerCount": 0,
            "targetWorkerSizeId": 0,
            "zoneRedundant": false
        }
      }
    }
}

With that set up separately, the final version of the App Service plan template will look like this:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "prefix": {
            "type": "String"
        },
        "location": {
            "type": "String"
        },
        "servicePlanSku": {
            "type": "object"
        },
        "servicePlanProperties": {
            "type": "object"
        }

    },
    "variables": {
        "componentName": "[concat(parameters('prefix'), '-asp')]"
    },
    "resources": [
        {
            "type": "Microsoft.Web/serverfarms",
            "apiVersion": "2021-03-01",
            "name": "[variables('componentName')]",
            "location": "[parameters('location')]",
            "sku": "[parameters('servicePlanSku')]",
            "kind": "linux",
            "properties": "[parameters('servicePlanProperties')]"
        }
    ]
}

As you can see, the name of the resource we're deploying has become a variable that takes the provided prefix from parameters and adds its own abbreviation to it. The resource is deployed from its ARM template and one or more parameter files together, as you will see below.

We would use the same logic for all the other parts of our stack.

The infrastructure

We will start with the infrastructure - Key Vault, Postgres, and Service Bus.

Azure Key Vault

An important part is the Azure Key Vault, which holds various secrets such as the database passwords and API keys to third-party services.

The Key Vault is both a provider and a consumer, at least in the sense that other parts of the stack will put their secrets into it when deployed.

From now on, I won't present the original ARM template and only highlight the final, edited template. I also won't include the parameters - again, the method to extract and then apply the parameters is the same. My recommendation is that your method to create resources should be consistent and follow the same logic so that adding a new resource type is as easy as possible.

The Key Vault template looks like this:

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "sku": {
            "type": "object",
            "metadata": {
                "description": "keyvault sku"
            }
        },
        "location": {
            "type": "string"
        },
        "prefix": {
            "type": "string"
        },
        "mailgunApiKey": {
            "type": "securestring"
        },
        "mailgunSender": {
            "type": "string"
        },
        "mailgunDomain": {
            "type": "string"
        }
    },
    "variables": {
        "componentName": "[concat(parameters('prefix'), '-kv')]"
    },
    "resources": [
        {
            "type": "Microsoft.KeyVault/vaults",
            "apiVersion": "2021-11-01-preview",
            "name": "[variables('componentName')]",
            "location": "[parameters('location')]",
            "properties": {
                "sku": "[parameters('sku')]",
                "tenantId": "[subscription().tenantId]",
                "accessPolicies": [],
                "enabledForDeployment": false,
                "enabledForDiskEncryption": false,
                "enabledForTemplateDeployment": true,
                "enableSoftDelete": true
            }
        },
        {
            "type": "Microsoft.KeyVault/vaults/secrets",
            "apiVersion": "2019-09-01",
            "name": "[concat(variables('componentName'), '/mailgunApiKey')]",
            "dependsOn": [
                "[resourceId('Microsoft.KeyVault/vaults', variables('componentName'))]"
            ],
            "properties": {
                "value": "[parameters('mailgunApiKey')]"
            }
        },
        {
            "type": "Microsoft.KeyVault/vaults/secrets",
            "apiVersion": "2019-09-01",
            "name": "[concat(variables('componentName'), '/mailgunSender')]",
            "dependsOn": [
                "[resourceId('Microsoft.KeyVault/vaults', variables('componentName'))]"
            ],
            "properties": {
                "value": "[parameters('mailgunSender')]"
            }
        },
        {
            "type": "Microsoft.KeyVault/vaults/secrets",
            "apiVersion": "2019-09-01",
            "name": "[concat(variables('componentName'), '/mailgunDomain')]",
            "dependsOn": [
                "[resourceId('Microsoft.KeyVault/vaults', variables('componentName'))]"
            ],
            "properties": {
                "value": "[parameters('mailgunDomain')]"
            }
        }
    ]
}

Notice the configuration property "enabledForTemplateDeployment": true - this will let other components write secrets into the Key Vault.

Also, notice that the deployment template expects three parameters specifying the "secrets" the app will consume to interact with the Mailgun API. Granted, only the API key is really a secret, and it's debatable whether the sender (an e-mail address) or the domain needs to be in the Key Vault. You'll make your own choices depending on how you want to handle similar considerations.

If you are wondering where the Mailgun secrets come from, you'll see soon enough.

The Service Bus namespace

Adding a Service Bus namespace to your project is surprisingly easy.

Pick a SKU depending on your usage - if you can get by with just Queues, you can go with the Basic pricing tier; to use Topics and Subscriptions, you'll need Standard.

We'll put the connection string to the Key Vault so that the apps can access it to send and receive messages.

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "sku": {
            "type": "object",
            "metadata": {
                "description": "Microsoft.ServiceBus/namespaces sku"
            }
        },
        "location": {
            "type": "string"
        },
        "prefix": {
            "type": "string"
        }
    },
    "variables": {
        "componentName": "[concat(parameters('prefix'), '-svcbus')]",
        "keyVaultName": "[concat(parameters('prefix'), '-kv')]"

    },
    "resources": [
        {
            "type": "Microsoft.ServiceBus/namespaces",
            "apiVersion": "2021-11-01",
            "name": "[variables('componentName')]",
            "location": "[parameters('location')]",
            "sku":  "[parameters('sku')]",
            "properties": {
                "zoneRedundant": false
            }
        },
        {
            "type": "Microsoft.KeyVault/vaults/secrets",
            "apiVersion": "2019-09-01",
            "name": "[concat(variables('keyvaultName'), '/serviceBusPrimaryConnectionString')]",
            "dependsOn": [
                "[concat('Microsoft.ServiceBus/namespaces/', variables('componentName'))]"
            ],
            "properties": {
                "value": "[listKeys(resourceId('Microsoft.ServiceBus/namespaces/authorizationRules', variables('componentName'), 'RootManageSharedAccessKey') ,'2017-04-01').primaryConnectionString]"
            }
        }
    ]
}

Postgres Flexible Server

Lastly, let's deploy the Postgres database.

There's nothing fundamentally new here. I included it in the tutorial since I've had to tackle an unexpected problem when specifying multiple configuration parameters at the same time, and I reckon you might want to do that, too.

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "type": "string"
        },
        "sku": {
            "type": "object"
        },
        "storageSizeGB": {
            "type": "int"
        },
        "administratorLoginPassword": {
            "type": "securestring"
        },
        "databases": {
            "type": "array"
        },
        "prefix": {
            "type": "string"
        }
    },
    "variables": {
        "componentName": "[concat(parameters('prefix'), '-pg')]",
        "administratorLogin": "jcstgadmin",
        "keyVaultName": "[concat(parameters('prefix'), '-kv')]"
    },
    "resources": [
        {
            "type": "Microsoft.DBforPostgreSQL/flexibleServers",
            "apiVersion": "2021-06-01",
            "name": "[variables('componentName')]",
            "location": "[parameters('location')]",
            "sku":"[parameters('sku')]",
            "properties": {
                "version": "13",
                "administratorLogin": "[variables('administratorLogin')]",
                "administratorLoginPassword": "[parameters('administratorLoginPassword')]",
                "availabilityZone": "1",
                "storage": {
                    "storageSizeGB": "[parameters('storageSizeGB')]"
                },
                "backup": {
                    "backupRetentionDays": 7,
                    "geoRedundantBackup": "Disabled"
                },
                "network": {},
                "highAvailability": {
                    "mode": "Disabled"
                },
                "maintenanceWindow": {
                    "customWindow": "Disabled",
                    "dayOfWeek": 0,
                    "startHour": 0,
                    "startMinute": 0
                }
            }
        },
        {
            "type": "Microsoft.DBforPostgreSQL/flexibleServers/firewallRules",
            "apiVersion": "2021-06-01",
            "name": "[concat(variables('componentName'), '/AllowAllAzureServicesAndResourcesWithinAzureIps')]",
            "dependsOn": [
                "[resourceId('Microsoft.DBforPostgreSQL/flexibleServers', variables('componentName'))]"
            ],
            "properties": {
                "startIpAddress": "0.0.0.0",
                "endIpAddress": "0.0.0.0"
            }
        },
        {
            "type": "Microsoft.DBforPostgreSQL/flexibleServers/configurations",
            "apiVersion": "2021-06-01",
            "name": "[concat(variables('componentName'), '/azure.extensions')]",
            "dependsOn": [
                "[resourceId('Microsoft.DBforPostgreSQL/flexibleServers', variables('componentName'))]",
                "[resourceId('Microsoft.DBforPostgreSQL/flexibleServers/firewallRules', variables('componentName'), 'AllowAllAzureServicesAndResourcesWithinAzureIps')]"
            ],
            "properties": {
                "value": "PGCRYPTO",
                "source": "user-override"
            }
        },
        {
            "copy": {
                "name": "databasecopy",
                "count": "[length(parameters('databases'))]"
            },
            "type": "Microsoft.DBforPostgreSQL/flexibleServers/databases",
            "apiversion": "2021-06-01",
            "name": "[concat(variables('componentName'), '/', parameters('databases')[copyIndex()].name)]",
            "dependsOn": [
                "[resourceId('Microsoft.DBforPostgreSQL/flexibleServers', variables('componentName'))]"
            ],
            "properties": {
                "charset": "UTF8",
                "collation": "en_US.utf8"
            }
        },
        {
            "type": "Microsoft.KeyVault/vaults/secrets",
            "apiVersion": "2019-09-01",
            "name": "[concat(variables('keyVaultName'), '/pgHost')]",
            "properties": {
                "value": "[concat(variables('componentName'), '.postgres.database.azure.com')]"
            }
        },
        {
            "type": "Microsoft.KeyVault/vaults/secrets",
            "apiVersion": "2019-09-01",
            "name": "[concat(variables('keyVaultName'), '/pgAdminUsername')]",
            "properties": {
                "value": "[variables('administratorLogin')]"
            }
        },
        {
            "type": "Microsoft.KeyVault/vaults/secrets",
            "apiVersion": "2019-09-01",
            "name": "[concat(variables('keyVaultName'), '/pgAdminPassword')]",
            "properties": {
                "value": "[parameters('administratorLoginPassword')]"
            }
        }
     ]
}

The problem I had was with the resource types Microsoft.DBforPostgreSQL/flexibleServers/firewallRules and Microsoft.DBforPostgreSQL/flexibleServers/configurations.

When both of them only had a dependency on [resourceId('Microsoft.DBforPostgreSQL/flexibleServers', variables('componentName'))], the firewall rules were deployed while the attempt to white-list the pgcrypto extension failed with a response code Conflict.

What gives?

I suspect that in that scenario, the orchestrator tries to write to the same configuration file, and either one of the attempts can fail since the other one may not be finished yet.

I solved it by making the second configuration depend on the first one being finished, but I think it's a hacky workaround at best.

If you were to add multiple custom configurations to my database server, which you very well might want to do, you would end up with a growing mess of dependencies that the orchestrator should figure out on its own, in my opinion anyway.

That, or I've overlooked a better way of handling this. Please let me know in the comments if you know it!

Another noteworthy feature I want to highlight is how the ARM template for the Postgres database puts its own secrets into the Key Vault. Unlike with the function app, which we'll cover shortly, we haven't given the Postgres any rights to read or write to the Key Vault here, and yet it can do so.

This is a little unexpected but very practical since the function app will need the Postgres credentials to talk to it.

The function app

Azure Functions lets you choose from several pricing plans that come with different performance and scaling characteristics.

The Consumption plan is great for getting started as you only pay for the actual usage, and there is a generous free monthly grant. It may also work great in production for apps that do not handle continuous sustained traffic.

The key drawback of the Consumption plan on Linux is that it does not support deployment from a ZIP package directly. Rather, you have to put the ZIP package somewhere, e.g., into a container in Azure Blob Storage, and provide the URL to the package.

This is clearly stated in the docs, but not every tutorial you will find on the Internet takes this into account. For example, if you thought you could just use the Azure Functions Github action to deploy your code, it won't work for Linux apps on the Consumption plan. The pipeline will finish green, but your code won't deploy.

The correct template to build on is this one referenced in Azure docs. But you'll need a little more code to make it work.

My solution is based on this fantastic article by Mark Heath, which got me to 80% of where I needed to be.

Below is the rather long ARM template for the function app, which also contains the App Service plan resource. You'll see that it does not specify the package URL - we don't have it yet.

{
    "$schema": "https://schema.management.azure.com/schemas/2019-04-01/deploymentTemplate.json#",
    "contentVersion": "1.0.0.0",
    "parameters": {
        "location": {
            "type": "string"
        },
        "prefix": {
            "type": "string"
        },
        "fullDeployment": {
            "type": "bool"
        },
        "servicePlanSku": {
            "type": "object",
            "metadata": {
                "description": "sku of Microsoft.Web/serverfarms"
            }
        },
        "servicePlanProperties": {
            "type": "object",
            "metadata": {
                "description": "properties of Microsoft.Web/serverfarms"
            }
        }
    },
    "variables": {
        "componentName": "[concat(parameters('prefix'), '-fapp')]",
        "servicePlanName": "[concat(parameters('prefix'), '-fasp')]",
        "keyVaultName": "[concat(parameters('prefix'), '-kv')]",
        "appSettings": {
            "APPINSIGHTS_INSTRUMENTATIONKEY": "[concat('@Microsoft.KeyVault(VaultName=', variables('keyVaultName'), ';SecretName=appInsightsInstrumentationKey)')]",
            "AzureWebJobsStorage": "[concat('@Microsoft.KeyVault(VaultName=', variables('keyVaultName'), ';SecretName=storageAccountConnectionString-jamcomments-fapp', ')')]",
            "FUNCTIONS_EXTENSION_VERSION": "~4",
            "FUNCTIONS_WORKER_RUNTIME": "node",
            "MAILGUN_API_KEY": "[concat('@Microsoft.KeyVault(VaultName=', variables('keyVaultName'), ';SecretName=mailgunApiKey)')]",
            "MAILGUN_DOMAIN": "[concat('@Microsoft.KeyVault(VaultName=', variables('keyVaultName'), ';SecretName=mailgunDomain)')]",
            "MAILGUN_SENDER": "[concat('@Microsoft.KeyVault(VaultName=', variables('keyVaultName'), ';SecretName=mailgunSender)')]",
            "PGDATABASE": "jamcomments",
            "PGSSLMODE": "require",
            "PGHOST": "[concat('@Microsoft.KeyVault(VaultName=', variables('keyVaultName'), ';SecretName=pgHost)')]",
            "PGUSER": "[concat('@Microsoft.KeyVault(VaultName=', variables('keyVaultName'), ';SecretName=pgAdminUsername)')]",
            "PGPASSWORD": "[concat('@Microsoft.KeyVault(VaultName=', variables('keyVaultName'), ';SecretName=pgAdminPassword)')]",
            "SERVICEBUS_CONNECTION": "[concat('@Microsoft.KeyVault(VaultName=', variables('keyVaultName'), ';SecretName=serviceBusPrimaryConnectionString)')]"
        }
    },
    "resources": [
        {
            "type": "Microsoft.Web/serverfarms",
            "name": "[variables('servicePlanName')]",
            "apiVersion": "2020-09-01",
            "location": "[parameters('location')]",
            "kind": "linux",
            "properties": "[parameters('servicePlanProperties')]",
            "sku": "[parameters('servicePlanSku')]"
        },
        {
            "condition": "[parameters('fullDeployment')]",
            "dependsOn": [
                "[concat('Microsoft.Web/serverfarms/', variables('servicePlanName'))]"
            ],
            "type": "Microsoft.Web/sites",
            "apiVersion": "2021-03-01",
            "name": "[variables('componentName')]",
            "location": "[parameters('location')]",
            "kind": "functionapp,linux",
            "identity": {
                "type": "SystemAssigned"
            },
            "properties": {
                "enabled": true,
                "serverFarmId": "[resourceId('Microsoft.Web/serverfarms', variables('servicePlanName'))]",
                "siteConfig": {
                    "numberOfWorkers": 1,
                    "linuxFxVersion": "Node|14",
                    "alwaysOn": false,
                    "functionAppScaleLimit": 200,
                    "minimumElasticInstanceCount": 0
                },
                "clientAffinityEnabled": false,
                "keyVaultReferenceIdentity": "SystemAssigned"
            },
            "resources": [
                {
                    "condition": "[parameters('fullDeployment')]",
                    "type": "config",
                    "name": "appSettings",
                    "apiVersion": "2020-09-01",
                    "dependsOn": [
                        "[concat(parameters('prefix'), '-keyVaultAccessPolicy')]",
                        "[resourceId('Microsoft.Web/sites', variables('componentName'))]"
                    ],
                    "properties": "[variables('appSettings')]"
                }
            ]
        },
        {
            "type": "Microsoft.Resources/deployments",
            "apiVersion": "2020-10-01",
            "name": "[concat(parameters('prefix'), '-keyVaultAccessPolicy')]",
            "dependsOn": [
                "[resourceId('Microsoft.Web/sites', variables('componentName'))]"
            ],
            "properties": {
                "mode": "Incremental",
                "template": {
                "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentTemplate.json#",
                "contentVersion": "1.0.0.0",
                "resources": [
                    {
                        "type": "Microsoft.KeyVault/vaults/accessPolicies",
                        "name": "[concat(variables('keyVaultName'), '/add')]",
                        "apiVersion": "2019-09-01",
                        "properties": {
                            "accessPolicies": [
                            {
                                "tenantId": "[reference(resourceId('Microsoft.Web/sites', variables('componentName')), '2018-02-01', 'Full').identity.tenantId]",
                                "objectId": "[reference(resourceId('Microsoft.Web/sites', variables('componentName')), '2018-02-01', 'Full').identity.principalId]",
                                "permissions": {
                                "keys": [
                                    "get"
                                ],
                                "secrets": [
                                    "get"
                                ]
                                }
                            }
                            ]
                        }
                    }
                ]
                }
            }
        }
    ]
}

On a side note, giving the function app an identity, which is this bit:

"identity": {
    "type": "SystemAssigned"
},

makes it possible to give it permissions to talk to the Key Vault and get a few secrets from it and make them available to the application code via environment variables. The permission is granted by the resource "type": "Microsoft.KeyVault/vaults/accessPolicies", and the secrets are inserted into the environment variables in the appSettings variables.

With the layout of the ARM templates being presented and explained, we can move on to the scripts that deploy them.

The deployment process

The infrastructure is deployed separately from the function app. The idea is to run the infrastructure deployment only when required so that most of the time, only the function app is deployed because only the application code changes frequently.

I won't go over the infrastructure deployment. A curious reader can peruse the infra/azure/platform/scripts/create-infra.sh Bash script.

The deployment scripts

The function app has two scripts: one that creates the function app and its related resources (storage account, app service plan, Service Bus topics), and another that deploys the application code.

First, let's have a look at how the function app and its specific resources are deployed (infra/azure/apps/api/scripts/create-funcapp.sh):

#!/bin/bash

set -e

: "${environment:?variable not set or empty}"
: "${fullDeployment:?variable not set or empty}"

if [ "$CI" == "true" ]
then
  echo "Running on CI"
  export AZURE_CORE_OUTPUT=none
else
  echo "Not running on CI"
fi

echo "Starting function app deployment"

case "$environment" in
 staging)
   paramFolder="staging"
   ;;
 prod)
   paramFolder="prod"
   ;;
 *)
   echo "unknown environment: $environment" >&2
   exit 1
esac

templates="infra/azure/apps/api/templates"
commonParams="$templates/params/common"
envParams="$templates/params/$paramFolder"

location="East US"
locationSymbol="eus"

prefix="jcomm-$environment-$locationSymbol"
rgp="$prefix-rgp"

echo "Creating resource group $rgp"
az group create --name "$rgp" --location "$location"

echo "Creating storage account"
az deployment group create \
  -n "$prefix-storage-account" \
  -g "$rgp" \
  -f "$templates"/storage.json \
  -p "$commonParams"/storage-account-parameters.json \
     "$envParams"/storage-account-parameters.json \
     prefix="$prefix" \
     environment="$environment" \
     locationSymbol="$locationSymbol" \
     location="$location"

echo "Creating service bus topics"
az deployment group create \
  -n "$prefix-sb-topics" \
  -g "$rgp" \
  -f "$templates"/svcbus-topics.json \
  -p "$commonParams"/svcbus-topics-parameters.json \
     "$envParams"/svcbus-topics-parameters.json \
     prefix="$prefix"

echo "Creating function app"
az deployment group create \
  -n "$prefix-fapp" \
  -g "$rgp" \
  -f "$templates"/funcapp.json \
  -p "$commonParams"/funcapp-parameters.json \
     "$envParams"/funcapp-parameters.json \
     prefix="$prefix" \
     location="$location" \
     fullDeployment="$fullDeployment"

echo "Finished creating function app"

As you can see, a few key parameters that the templates expect are defined here, such as the location of the resources and the prefix. The script needs two environment variables, environment and fullDeployment, supplied by the Github runtime triggering the workflow run.

Next, here is how the application code is deployed (infra/azure/apps/api/scripts/deploy-funcapp.sh):

#!/bin/bash

set -e

: "${environment:?variable not set or empty}"

echo "Get the storage connection string"
AZURE_STORAGE_CONNECTION_STRING=$(az storage account show-connection-string -g "jcomm-${environment}-eus-rgp" -n "jc${environment}eusstgacc" --query "connectionString" -o tsv)

container=jamcomments-funcapp-releases
timestamp=$(date +"%Y%M%d%H%M%N")
blobName="functionapp${timestamp}.zip"

echo "Create storage container"
az storage container create -n "$container" --public-access off --connection-string "$AZURE_STORAGE_CONNECTION_STRING"

echo "Zip the app"
zip -q -r "$blobName" . -x@.funcignore -x .funcignore

echo "Upload ${blobName} to storage"
az storage blob upload -c "$container" -f "$blobName" -n "$blobName"  --connection-string "$AZURE_STORAGE_CONNECTION_STRING"

blobUrl=$(az storage blob url -c "$container" -n "$blobName" -o tsv --connection-string "$AZURE_STORAGE_CONNECTION_STRING")
expiry=$(date --date='5 years' +"%Y-%m-%dT%H:%M:%SZ")

echo "Get the SAS for the ZIP package"
sas=$(az storage blob generate-sas -c "$container" -n "$blobName" --permissions r -o tsv --expiry "$expiry" --connection-string "$AZURE_STORAGE_CONNECTION_STRING")

packageUrl="${blobUrl}?${sas}"

echo "Set the app settings to run from package URL"
az webapp config appsettings set -n "jcomm-${environment}-eus-fapp" -g "jcomm-${environment}-eus-rgp" --settings "WEBSITE_RUN_FROM_PACKAGE=$packageUrl"

Quite a few steps to put the ones and zeroes to work, ain't it?

I think the inline comments paint a clear picture but to recap, you have to:

create a private-access container in the Blob storage
zip the application files
upload the ZIP archive to the container
ask Azure for the URL to the archive
ask Azure to give you a read-only shared access signature (SAS) to the archive
construct the full URL
add a configuration to the function app to run from the zip archive we just uploaded

To be clear, this solution is specific to Linux function apps running on the Consumption plan. If you are on any other plan, you can just use the Azure Functions Github action and let it do the ZIP deployment for you.

I don't fully understand where this limitation comes from. If pressed, I would hypothesise that the Linux function apps on the Consumption plan do not have any dedicated storage (other than the Storage Account that's required for all Functions to operate - but then, why doesn't the runtime use it for the ZIP package hosting?) unlike all the other plans. In reality, I simply don't know, and the docs do not reveal the reasons.

With that said, let's see how this all is tied together in a Github workflow.

The Github workflow

The workflow featured below (see .github/workflows/azure-jam-comments-api.yml in the JComments repo) prepares the environment variables, runs the build and test actions, then calls the scripts (described in the previous section) to create the infrastructure in Azure and deploy the code.

The workflow can be triggered manually (see workflow_dispatch) or automatically by pushes to dev and main branches.

The very first time the workflow should run, or put differently, the very first time a given resource group and its contents are created in Azure, it should run manually with the fullDeployment flag set to true otherwise not all resources will have been created.

For our use case, it's important that we have the zip binary available. The action montudor/action-zip@v1 takes care of that.

name: Deploy JComments API to Azure Functions

on:
  workflow_dispatch:
    inputs:
      environment:
        description: 'Name of environment to deploy (staging, prod)'
        required: true
        default: 'staging'
      fullDeployment:
        description: 'Whether to do a full deployment or not'
        required: true
        default: 'false'
  push:
    branches:
      - dev
      - master
    paths:
      - 'infra/azure/apps/api/**'
      - '.github/workflows/azure-jam-comments-api.yml'
      - 'AzureCommentsApiGet/**'
      - 'AzureCommentsApiPost/**'
      - 'AzureCommentsApiNotify/**'
      - 'sql/**'
      - 'src/**'
      - 'test/**'

env:
  NODE_VERSION: '14.x'                     # set this to the node version to use (supports 8.x, 10.x, 12.x)
  fullDeployment: ${{ github.event.inputs.fullDeployment || 'false' }}

defaults:
  run:
    shell: bash

jobs:
  setupEnv:
    name: Set up environments
    runs-on: ubuntu-latest
    timeout-minutes: 5
    steps:
      - name: Checkout
        uses: actions/checkout@v2.3.4
        with:
          persist-credentials: false
      - name: setup environment variables
        id: setupEnv
        run: |
          echo "Setup environment and secrets"
          if [ "${{ github.event.inputs.environment }}" = "staging" ] || [ "${{ github.ref }}" = "refs/heads/dev" ]; then
            echo "::set-output name=environment::staging"
            echo "::set-output name=githubEnvironment::Development"
          elif [ "${{ github.event.inputs.environment }}" = "prod" ] || [ "${{ github.ref }}" = "refs/heads/master" ]; then
            echo "::set-output name=environment::prod"
            echo "::set-output name=githubEnvironment::Production"
          fi
    outputs:
      environment: ${{ steps.setupEnv.outputs.environment }}
      githubEnvironment: ${{ steps.setupEnv.outputs.githubEnvironment }}
      name: jamcomments-${{ steps.setupEnv.outputs.environment }}-eus-fapp

  build-and-deploy:
    name: Build app and deploy to Azure
    environment: ${{ needs.setupEnv.outputs.githubEnvironment }}
    concurrency: ${{ needs.setupEnv.outputs.name }}
    needs:
      - setupEnv
    env:
      environment: ${{ needs.setupEnv.outputs.environment }}
    runs-on: ubuntu-latest
    steps:
    - name: 'Checkout GitHub Action'
      uses: actions/checkout@master

    - name: Setup Node ${{ env.NODE_VERSION }} Environment
      uses: actions/setup-node@v1
      with:
        node-version: ${{ env.NODE_VERSION }}
        cache: 'yarn'

    - name: 'Resolve Project Dependencies Using yarn'
      shell: bash
      run: |
        yarn
        yarn build
        yarn test

    - name: Azure Login
      timeout-minutes: 5
      uses: azure/login@v1
      with:
        creds: ${{ secrets.AZURE_CREDENTIALS }}

    - name: Create infrastructure
      run:  infra/azure/apps/api/scripts/create-funcapp.sh
      env:
        DB_PASSWORD: ${{ secrets.DB_PASSWORD }}

    - name: Install zip
      uses: montudor/action-zip@v1

    - name: 'Deploy to Azure Functions'
      run: infra/azure/apps/api/scripts/deploy-funcapp.sh

    - name: Azure Logout
      if: ${{ always() }}
      run: |
        az logout
        az cache purge
        az account clear

Final thoughts

This guide is a product of many trials and errors I have made attempting to find a working solution for automatically deploying a function app to Azure Functions, with specific issues related to the Consumption Plan on Linux. It's by no means the best possible solution. It works but should be improved.

One thing that really bugs me is that the resources are not created in a transactional fashion. Even within a single ARM template, if an error happens while provisioning a nested resource, such as a database configuration item, whatever happened before that is kept in place.

This means that a resource can exist in an inconsistent state, potentially missing vital configuration or being exposed to exploits.

This is exacerbated by the fact we need to run multi-step procedural scripts to actually deploy the application code. The definition for code deployment would ideally look just like the ARM templates that lay out the desired state of a given resource - declaratively, not procedurally.

With the caveats admitted and even emphasized, I still enjoy this way of managing and deploying code and all its dependencies way more than I do logging into a Linux server with SSH and running the usual suite of tools - apt-get, vim configfile, etc.

Once you have figured out the pipeline and all its quirks, you save tons of time worrying about infrastructure, which you can invest more productively.

Hopefully, this has been useful to you, and if you have any questions or suggestions for improving the setup, be sure to leave a comment below!

How to teach your Node.js app to speak SQL without an ORM

2021-08-18T12:00:00Z

Confession: I love SQL. I love its declarative nature: you tell the database what data you need and how it should look, not how it should go around accomplishing that.

I love SQL so much I want to use it in any programming language I happen to be using, and do so directly. No middleman, no abstraction layer, and no ORM. I could write another article discussing the pros and cons of ORMs but let's leave that aside today. Suffice to say, I find them to be a needless intermediary.

I also prefer using types, which is why Typescript is my go-to language when working with Node.js. Granted, it is not a standalone language and compiles back to Javascript, and so its type safety is limited to compile-time. Still, its impact on my developer productivity and happiness is difficult to overstate.

Lastly, my database of choice for pretty much any web app has been PostgreSQL. Let's see how one could ties these three preferences together.

SQL with Typescript

When it comes to accessing a database, a popular choice in the Node.js is Knex. It comes with the usual assortment of features such as a schema builder, a query builder, and migrations. I've used it on client projects and I've got to admit, it is really easy to use.

It includes support for raw queries, and I could almost use normal SQL that way, but looks how it looks in practice:

knex.raw('select * from users where id = ?', [1]).then(function(resp) { ... });

Indeed, I supply an array of parameters to replace the placeholders (?), and I trust Knex will take care of sanitizing the input to prevent SQL injection. What I don't have is type safety.

Introducing pgtyped

I was looking for a Typescript library for an experiment I'm running, and found this wonderful repository: pgtyped. Long story short, it has precisely the feature set I was looking for. Quoting from the README:

PgTyped makes it possible to use raw SQL in TypeScript with guaranteed type-safety.
No need to map or translate your DB schema to TypeScript, PgTyped automatically generates types and interfaces for your SQL queries by using your running Postgres database as the source of type information.

What it means in practice: I write normal SQL queries, put placeholders inside for parameters, and run pgtyped on my code-base. It will examine the database, infer the data types, map them to Javascript datatypes, and create typed "proxies" that I work with in my data-access layer.

Thus I both have and eat my cake!

Practical example

JComments is a headless commenting platform for static websites. I've built it with NestJS and Postgres. Pgtyped came on my radar when I was setting up my stack, and has been there from the very first draft of the app.

The NestJS way of organizing your code makes a heavy use of modules and I quite enjoy its powerful dependency injection container, even though I burned myself many times when getting familiar with the framework.

Inside each module, I have services that provide functionality specific to the module's problem domain, as well as a data access objects. For example, the comments module that serves out comments for a given URL over an API has these files inside:

comments.service that is used by the API controller and mediates the data access
comments.interface that contains domain interfaces
comments.sql that has raw SQL queries used by the service class

When running pgtyped in my project folder, it generates a fourth file: comments.queries. There's generated code inside that contains proxies for query parameter objects as well as proxies for the returned results. For instance, given this SQL query:

/* @name commentsForAccountPaged */
SELECT * FROM comments WHERE account_id=:accountId ORDER BY created_at DESC LIMIT :limit OFFSET :offset;

The proxy for the query parameters looks like this:

export interface ICommentsForAccountPagedParams {
  accountId: string | null | void;
  limit: string | null | void;
  offset: string | null | void;
}

And the proxy for the resultset looks like this:

export interface ICommentsForAccountPagedResult {
  id: string;
  account_id: string;
  page_url: string;
  comment: string;
  reader_name: string;
  reader_email: string | null;
  reader_website: string | null;
  created_at: Date;
  page_title: string | null;
}

The object that runs the query have a name determined by the annotation preceding the SQL statement, in this case commentsForAccountPaged. It has a method run(), which takes the ICommentsForAccountPagedParams interface as its parameter and returns an array of ICommentsForAccountPagedResult.

This is all tied together in the service method:

async commentsForAccountPaged(account: Account, batchSize?: number, page?: number): Promise<CommentWithId[]> {
  const limit = batchSize ?? 10
  const offset = ((page ?? 1) - 1) * limit
  const params = { accountId: account.id, limit: `${limit}`, offset: `${offset}` }
  const pagedComments = await commentsForAccountPaged.run(params, this.client)
  this.logger.debug(interpretedQuery(commentsForAccountPaged, params))
  return pagedComments.map(this.recordToClass)
}

I can't make a mistake by passing it parameters of an incorrect type, and I know exactly what the object coming out of the database looks like. The last statement of the method merely maps the auto-generated object into my domain object.

I find this approach refreshing as it lets me speak SQL. It's an analogy to the "exposed brick" trend in apartment construction. We speak to databases daily, and often hide this fact behind the facades of ORMs and other middlemen. Why? The database is a foundational technology. And anything beyond a simple CRUD app will eventually tie itself to the specific database chosen for the project. We can just as well speak with it using its native tongue.

There's an argument to be made that as a project grows, the number of SQL queries written grows as well. Yes, code duplication can be an issue, if you have multiple queries that kinda do the same thing but not exactly. I just don't perceive this to be a problem: just by looking into a single SQL file, I know exactly how many queries I am using, what they look like, and can make refactoring decisions based on that.

Conclusion

I've been enamored with pgtyped and recommend that you take a look at it if you are developing Typescript apps with Postgres. I found it to be highly usable for a normal web app with standard database needs.

It's a big difference if you're coming over from the ORM world, and my goal is not to talk you out of using ORMs. If you like working with plain SQL queries and statements instead, then you'll be quick to appreciate it.

How to localize text with a binding in WPF

2021-01-30T14:45:00Z

Recently, I faced a following dilemma: I was binding a TextBlock to a string property of a class instance, and when that instance was null, I wanted to show a placeholder that had to be localizable. Given this example:

<TextBlock x:Uid="Track_Title"
           TextWrapping="WrapWithOverflow" 
           FontWeight="Bold"
           Text="{Binding Meta.Title}"/> <!-- what if it's null? -->

What would be the right way to do it?

One option would be to place another TextBlock next to it, bind it to the placeholder value, and only show it when the result of the binding expression was empty:

<TextBlock FontWeight="Bold"
           Text="{loc:Loc NO_TRACK}">
  <TextBlock.Style>
    <Style TargetType="TextBlock">
      <Setter Property="Visibility" Value="Collapsed" />
      <Style.Triggers>
        <DataTrigger Binding="{Binding Meta.Artist}"
                     PresentationTraceSources.TraceLevel="High"
                     Value="">
          <Setter Property="Visibility" Value="Visible"/>
        </DataTrigger>
      </Style.Triggers>
    </Style>
  </TextBlock.Style>
</TextBlock>

If you are wondering what's the {loc: Loc} expression, have a look at my article about localizing a WPF app on .NET Core from last year.

This technique of using separate placeholder elements has the advantage of being explicit at the cost of cluttering the XAML. I wanted something simpler, perhaps using a custom IValueConverter to provide the placeholder value when the binding source was null:

<TextBlock x:Uid="Track_Title"
           TextWrapping="WrapWithOverflow" 
           FontWeight="Bold"
           Text="{Binding Meta.Title, 
              Converter={StaticResource FallbackStringConverterSimple}, 
              ConverterParameter='NO_TRACK'}"/> <!-- null averted -->

The converter would take the result of the binding expression, return it if it's a non-empty string, and when not, look up the fallback value in a dictionary (encapsulated in an abstract class from which it is derived) and provide that instead:

public class FallbackStringConverterSimple : BaseFallbackConverter, IValueConverter
{
    public object Convert(object? value, Type targetType, object? parameter, CultureInfo culture)
    {
        if (value is string desired && !string.IsNullOrEmpty(desired))
        {
            return desired;
        }

        var fallback = parameter as string;
        if (string.IsNullOrEmpty(fallback)) return Binding.DoNothing;

        var replacement = TranslationSource.Instance[$"{Dictionary}.{fallback}"];
        return replacement;
    }

    public object ConvertBack(object value, Type targetType, object parameter, CultureInfo culture)
    {
        throw new NotImplementedException();
    }
}

This works, and it also has one limitation: were the user to switch languages in the app, e.g. from English to German, the placeholder value would not be updated and would still show the English translation.

The reason for this behavior is that the values returned from an IValueConverter are "final", and since the binding source did not change while the user switched the UI language, there was no impulse to re-evaluate the binding.

The trick I originally wanted to write about today is to use a custom IMultiValueConverter and create a sneaky little binding for the placeholder value.

The IMultiValueConverter interface accepts an array of objects on which it works, and so we can easily pass it the source element itself (the TextBlock) as well as the result of the original binding. A converter parameter can be used to pass in the fallback.

public sealed class FallbackStringConverter: BaseFallbackConverter, IMultiValueConverter
{
    public object Convert(object[]? values, Type targetType, object? parameter, CultureInfo culture)
    {
        if (values == null || values.Length < 2 || 
            values[0] is not TextBlock textBlock)
        {
            return Binding.DoNothing;
        }

        var desired = values[1] as string;
        var fallback = parameter as string;

        if (!IsEmptyString(desired) || IsEmptyString(fallback))
        {
            // return the desired text if not empty or when the fallback is not available
            return desired;
        }

        var binding = new Binding
        {
            Mode = BindingMode.OneWay,
            Source = TranslationSource.Instance,
            Path = new PropertyPath($"[{Dictionary}.{fallback}]")
        };
        BindingOperations.SetBinding(textBlock, TextBlock.TextProperty, binding);

        return string.Empty; /** 🤔 **/
    }

    public object[] ConvertBack(object value, Type[] targetType, object parameter, CultureInfo culture)
    {
        throw new NotImplementedException();
    }
}

The converter will again return the binding result if non-empty. When it's time to provide the fallback instead, it creates a new binding treating the fallback passed in as the third method parameter as a key in the translation dictionary and applying the binding to the TextBlock. And, notice the empty return value, which made me scratch my head.

It works - to the extent that when the result of the binding expression is empty, the converter steps in and provides the fallback, and the fallback is localized in real-time, responding to UI language changes. Something is definitely fishy about this trick, though.

It took me about a day to realize where I was wrong: once the binding expression was empty, I replaced the original binding with the new one, binding to the fallback value, which meant the fallback would stay there forever! 🤣

Time to educate myself.

In the end, I learned about PriorityBinding and was able to get rid of the converter altogether. The final version is:

<TextBlock>
  <TextBlock.Text>
    <PriorityBinding>
      <Binding Path="SelectedTrack.Meta.Title"/>
      <Binding Source="{x:Static loc:TranslationSource.Instance}"
               Path="[Common.i18n.Strings.NO_TRACK]"/>
    </PriorityBinding>
  </TextBlock.Text>
</TextBlock>

The PriorityBinding will return the value of the first binding expression if not empty, and default to the 2nd otherwise.

Notice two things: the fallback binding uses a static access to the TranslationSource singleton instance, and this is one of a few cases when the singleton pattern helps.

Second, the value of the Path property refers to the entire path, which comprises of a dictionary for the current assembly and the key to the translatable resource. I think having the path specified like this is a definite weakness of this approach, let alone the fact that that it's a string and Intellisense will no longer help me if I change the translation key later or move it to another assembly.

Nevertheless, I think this solution is practical enough to warrant consideration.

How to migrate from Ghost to Eleventy

2021-01-30T14:45:00Z

My tango blog used to run on the Ghost publishing platform. It's fast and comfortable, but since the rest of this website is built statically on Eleventy, I wanted move away from Ghost. Here is what I learned in the process.

Reasons to switch

Let's have a look at my reasons. Yours and mine are going to be different.

Background

My tango blog is already mostly static. There was backend with the Ghost admin app that I used to compose my articles. Once published, I had no reason to go there.

Ghost has some integrations. I didn't use them. It has a membership functionality. I didn't use it.

Perhaps surprisingly, Ghost doesn't, to this day, have built-in comments. I missed them, and integrated the Isso self-hosted comments, there and here. It's a little bit of Python code with SQLite. Not that there's much commenting going on here anyway.

That one fear that made me do it

I built a solid collection of articles, and I would be supremely pissed if I lost them.

I ran Ghost on SQLite. I trust it but hey, it's a database. I wasn't going to check it in into the repository, and I set up semi-manual backups; still, I felt uneasy. Something could go wrong, the database might become corrupted.

All my uploads (mostly images accompanying my posts) are on the filesystem, not in the database. I'd have to back them up, too, if I wanted to be thorough.

And then there's the rest of this website, which lives happily in a Git repository. One that automatically takes care of all my content is backed up in several places. I knew that if I could take my blog's content and put it in my git repo, my fear would go away instantly.

Other considerations played little to no role. Ghost is pretty fast, and the blog's performance didn't go up after the initial migration. You can self-host it at no cost, so price was also not a concern.

How I did it

Before diving in, I assumed this was going to be a lengthy process. Not so much. The initial import took about a day and I went live in a week.

First, Ghost lets you export your content into JSON. Put this JSON in an empty folder and install the package ghost-to-md. The version I used was 1.3.0.

npm i ghost-to-md

Then point it to your exported JSON like this:

.\node_modules\.bin\ghost-to-md $PATH_TO_GHOST_JSON_FILE

The script will chew through your JSON and create a folder ghost-to-md-output with all your glorious articles converted to Markdown.

I was pretty impressed by this, because I checked the SQLite database and Ghost stores your articles as HTML, and so a back conversion had to take place.

It succeeded with remarkable precision with only a few misses:

YouTube embeds
Spotify embeds
Bandcamp embeds

You'll notice the pattern here.

I was going to go through all my articles with a magnifying glass anyway, so it wasn't such a big deal. Just be aware of it. Anything contained within an <iframe...> gets quietly ignored.

How much work is left for you depends on your existing Eleventy setup. I will describe what I had to do.

The basics

Cool URLs don't change, right?

As luck would have it, the ghost-to-md exported the slug property of each post's metadata, which happens to be what Ghost to used as the final part of the path in the URL.

And so, if the slug were my-awesome-cd-review, the relative URL would be /tango/my-awesome-cd-review/. Given that by default, Eleventy maps folder and file names to URL components, all it took me to keep the URLs the same was to place my posts in a folder ./$ROOT_CONTENT_DIR/tango and instruct Eleventy to use the slug metadata variable to construct the permalink.

The way to do this is to make each post use the same template, and in the template specify this format of the permalink:

permalink: /tango/{{ slug }}/

No additional configuration / programming required.

The fine-tuning

I copied the script's output under my _site folder, where all my content lives. The articles I post under /code use slightly different metadata, but I was going to build new templates for my tango content, and so that wasn't an immediate problem.

Adapting my Ghost theme to Eleventy took just a little while. I took all the styles and HTML as-is from the theme. Ghost uses Handlebars while I use mostly Nunjucks on Eleventy, so it took a bit to convert the variables, loops, etc.

The 90% of the work was going through the articles and locating the missing bits. There might've been a way to automate that but I wanted to make sure with my own eyes anyway.

There were occasional hiccups.

RSS

For instance, the RSS that Ghost generates uses the post's id from the database for the guid property on the item element. That part wasn't exported to the Markdown, and so I ran a custom script on the IDs like this, saving the output to a file:

const file = require("./tango-is-alive.ghost.2021-01-23-12-03-06.json");
console.log(JSON.stringify(file.db[0].data.posts.map(p => ({slug: p.slug, id: p.id}))))

I placed the file in the _site/data ensuring that it would be available to Eleventy during build. Then the template for my RSS feed has this bit to use the "legacy" Ghost ID for the existing content:


<guid isPermaLink="false">{% if item.data.ghost_id %}{{ item.data.ghost_id }}{% else %}{{ item.data.slug }}|{{ item.data.date_updated }}{% endif %}</guid>

Obviously my new articles aren't going to have this ID but I wanted to avoid making the RSS aggregators think ALL of my content is new / fresh / updated, and spamming my subscribers with everything I've got so far.

Responsive images

Ghost generates multiple versions of the images you upload via the Admin app so that the browser can fetch whichever version is most appropriate given the available screen width.

I had these versions already generated for the existing content. Wouldn't it be nice to have a feature like this for my future posts, however?

It turned out Ghost uses the sharp library for this task, and so could I. It took me about an hour to write a script that resizes everything that hasn't been already resized, and I hooked it up to be executed on each build like this (this is from within eleventy.js):

  eleventyConfig.on('afterBuild', () => {
    // Run me after the build ends
    // I started writing in 2018, and so that's why we start there :)
    let years = _.range(2018, new Date().getFullYear() + 1).map(_.toString);
    makeImagesAsync(years);
  });

  async function makeImagesAsync(years) {
    for (const year of years) {
      await makeResponsiveImages(year);
    }
    console.log("Generated responsive images");
  }

You'll notice there's a missing await at the end of the afterBuild hook. Last time I checked, there wasn't a way to hook up an async event handler here.

Here's the full script that does the resizing:

"use strict";

const _ = require("lodash");
const fs = require("fs");
const path = require("path");
const sharp = require("sharp");

const responsiveImageSizes = [320, 640, 960, 1920];

const imageRoot = path.join('build', 'tango', 'content', 'images');

async function makeResponsiveImages(rootDir) {
  let realRoot = path.join(imageRoot, rootDir);
  let dir;
  try {
    dir = await fs.promises.opendir(realRoot);
  } catch (oops) {
    console.error(`Could not open ${dir}`);
    console.error(oops);
    return;
  }
  for await (let dirent of dir) {
    if (dirent.isDirectory()) {
      await makeResponsiveImages(path.join(rootDir, dirent.name));
      continue;
    }
    if (dirent.isFile() && isImage(dirent.name)) {
      await resizeImage(path.join(rootDir, dirent.name));
    }
  }
}

function isImage(filename) {
  const supported = ["jpg", "png"];
  const filenameNorm = (filename || "").toLowerCase();
  return _.some(supported, s => filenameNorm.endsWith(`.${s}`));
}

async function resizeImage(filename) {
  const dirs = filename.split(path.sep);
  const file = dirs.pop();
  const dir = path.join(... dirs);
  await Promise.all(responsiveImageSizes.map(width => {
    const wDir = `w${width}`;
    const respDir = path.join(imageRoot, "size", wDir, dir);
    if (!fs.existsSync(respDir)) {
      fs.mkdirSync(respDir, { recursive: true });
    }
    if (fs.existsSync(path.join(respDir, file))) {
      return;
    }
    return sharp(path.join(imageRoot, filename)).resize(width, undefined, {
        withoutEnlargement: true
      })
      .rotate()
      .toFile(path.join(respDir, file));
  }));
}

module.exports.makeResponsiveImages = makeResponsiveImages;

The deployment

I have come across intermittent build failures, for which I submitted an issue. Those worried me enough that I introduced a two-step deployment process:

First, build everything as normal
Verify that no errors came up during build
Rename the build folder to live, from which content is actually served

One Powershell script takes care of the first point, then another does the third. It would be nice if I could fully automate this, e.g. by re-generating the website upon each git commit to master, but then I would have to trust the software again. For my purposes here, no need.

Upon the first deployment of my now fully static website, I had to change the nginx config to stop redirecting the /tango/ path to the Ghost instance, and finally stop it when I confirmed all was well.

Closing thoughts

I've been super happy with Eleventy since I first rolled out the current version of my website. Other than their pandering to BLM on their front page, without which I would be even happier, the software is solid and most importantly, my website doesn't rely on it after it's built.

It's good to know that everything is backed up in git, that there's no database to corrupt or lose, and should I get hit with a wave of visitors, the performance is only limited by nginx.

Do I miss Ghost's visual editor? Yes, it's fantastic, and so is VS Code. Whoever is managing his or her website using Eleventy is likely technical enough to handle writing Markdown (or HTML) by hand.

Go static, young man. Go static!

Localizing a WPF app running on .NET Core 3 in 2020

2020-06-24T12:00:00Z

I have recently updated my hobby desktop app, Bewitched, from .NET 4.6 to .NET Core 3.1. The process was mostly painless as I was late to the game and most Nugets were already updated for .NET Core 3 / Netstandard support. I lost no functionality - except the ability to localize the UI.

The app runs in English, Czech, German, and Spanish, and the localization was powered by LocBaml, a legacy tool from Microsoft that was never fully supported and the development of which ended sometime 2005 or so.

Good-bye, LocBaml

Ancient as it is, LocBaml had one distinct advantage over competing approaches: I could leave literal strings in the XAML files, mark the elements containing them with Uids, and the tool would extract them for me to translate. During build, the translations are then put into satellite assemblies. When the app starts, it decides which locale to use (based on user preferences) and loads the corresponding assembly.

The advantage of strings literals in XAML is hard to underrate. XAML looks and feels like HTML (for us web devs), and having human-readable strings makes it so much easier to edit, in the text mode as well as in the XAML Designer.

You see, the alternative route is to use RESX files, put your dictionaries there (originals as well as translations) and use keys instead of string literals in your XAML.

Compare this:

<Button x:Uid="ClickMeButton">Click me!</Button>

to this:

<Button Content="{ns:Loc ClickMeButtonText}" />

One disadvantage LocBaml has is, you cannot (easily) switch locales at runtime. Speculatively speaking, you might be able to re-load all of the UI but I haven't tried and the effort to make it worth would likely be disproportionate to the value delivered.

Nevertheless, LocBaml does not work with .NET Core 3 assemblies. What's a man to do? Go back to even more historical RESX files, of course!

Learning to love RESX again

My first steps were to re-learn what localization approaches exist for WPF. To my dismay, there hasn't been any ground-breaking new technique developed since I first check. I wanted to keep my string literals but that wasn't to be.

Chance led me to this article about a simple approach of localizing a WPF app. It uses RESX files but with Bindings, and that means I could potentially switch locales at runtime, could I?

Yes, yes, that's indeed what this means.

A link at the bottom of the article then led me to this Github repo, that adds support for multiple dictionaries.

Bewitched has multiple assemblies and even multiple frontends, and so this was directly applicable to my problem. I would not have to centralize all the texts but put them where they belong.

This solution is comparable to the WPF Localization Extension, which seems to be very popular. It's far lighter than that, however, just one C# file. I decided I could live without the extra complexity and features. The lighter my dependencies the better I think.

My learning steps

I have not used RESX files before, and so there was some learning curve to climb.

I created a file Strings.resx and one for each translation, i.e., Strings.cs.resx, Strings.de.resx, etc. For the dictionaries to be usable outside their assembly, I had to set the "Custom Tool" property to PublicResXFileCodeGenerator, otherwise they would only have the internal visibility.

To have a nice tree-view of related RESX files, I used the DependentUpon property of the EmbeddedResource declaration in the project file like this:

<ItemGroup>
    <EmbeddedResource Update="i18n\Strings.resx">
      <Generator>PublicResXFileCodeGenerator</Generator>
      <LastGenOutput>Strings.Designer.cs</LastGenOutput>
    </EmbeddedResource>
    <EmbeddedResource Update="i18n\Strings.en-US.resx">
      <Generator>PublicResXFileCodeGenerator</Generator>
      <DependentUpon>Strings.resx</DependentUpon>
    </EmbeddedResource>
    <EmbeddedResource Update="i18n\Strings.cs.resx">
      <Generator>PublicResXFileCodeGenerator</Generator>
      <DependentUpon>Strings.resx</DependentUpon>
    </EmbeddedResource>
    <EmbeddedResource Update="i18n\Strings.de.resx">
      <Generator>PublicResXFileCodeGenerator</Generator>
      <DependentUpon>Strings.resx</DependentUpon>
    </EmbeddedResource>
    <EmbeddedResource Update="i18n\Strings.es.resx">
      <Generator>PublicResXFileCodeGenerator</Generator>
      <DependentUpon>Strings.resx</DependentUpon>
    </EmbeddedResource>
  </ItemGroup>

Visual Studio support is there but won't overwhelm you with features. If you'd like to see all translations side-by-side, there's an extension ResXManager for this. I have a Jetbrains subscription, and ended up using its Localization Manager that does a comparable job.

Importantly, it can import / export dictionaries as CSV, which turned out to be super helpful.

One of the artifacts of the LocBaml build process is a bunch of CSV files with strings and in which usercontrol to find them. I wrote a Node.js script that extracted them from these CSVs and merged them into the format that Localization Manager could understood. It looks like this:

Path,Name,"en-US","cs","de","es"
Embrace.NetCore3/i18n/Strings,registerview.baml_YourNameLabel,Name,"Jméno:","Name:","Nombre:"

Here's the gist of my translation helper.

I imported the generated CSV into Localization Manager, and only had to add the English dictionary by sifting through my XAML files, extracting the strings and replacing them with their dictionary keys.

I encountered a few gotchas along the way.

First and foremost, I thought that the default dictionary (Strings.resx) would be used if the app did not find one for the selected locale. It turned out I was wrong. The English originals had to be put in the *.en-US.resx resource, otherwise an exception was thrown, complaining about a missing satellite assembly.

Second, the XAML designer support is not 100% reliable. Some texts only ever show up as keys, others are translated in design-time, too. I am not a big enough Visual Studio internals geek to understand what's happening there.

Usage in XAML

This toolkit is intended for use in XAML markup, and it could not be easier to use.

The setup is straightforward. At the topmost level, you import the namespace and specify the default dictionary to use like this:

  xmlns:resx="clr-namespace:Embrace.Common.i18n"
  xmlns:loc="clr-namespace:WpfLocalizationWithMultipleResourceManagers"
  loc:Translation.ResourceManager="{x:Static resx:Strings.ResourceManager}"

The first import points to the namespace where you have your resource files. The second imports the namespace of the localization helper. The third specifies the default dictionary.

The texts are then translated on-the-go like this:

<TextBlock Text="{loc:Loc trackeditor_baml_TrackTitleLabel}"
           x:Uid="TrackTitleLabel"/>

Because the resource lookup is based on a Binding, the user can select a different locale in Settings, the app informs the localization helper about the new locale, and all texts are refreshed automatically as a result of PropertyChanged event invokation. Sweet!

Usage in code

I have a few scenarios where I need to show a message to the user from code, e.g. a notification. In that case, I don't have any XAML.

Using the localization helper from code is not the primary use-case but it's possible.

In my App.xaml.cs, I add the default dictionary such that it's available downstream, and set the culture like this:

  TranslationSource.Instance.AddResourceManager(Strings.ResourceManager); // Strings.ResourceManager points to the resources
  TranslationSource.Instance.CurrentCulture = localCulture;
  // localCulture is an instance of CultureInfo here

The localization helper uses the Singleton pattern (TranslationSource.Instance), of which I am not a huge fan but it feels appropriate here.

Later, when I need to access localized strings, I access the singleton and ask for them like this:

var translator = TranslationSource.Instance;
_statusBarService.StatusBarMessage = translator["Embrace.Common.i18n.Strings.FAILED"];

You'll notice the hard-coded path to the resource key, which happens to be the fully-qualified name of the resource file (incl. the assembly name). ~~I hope to fix that at some point but nameof(...) only returns the name without the assembly qualification.~~ I'll fix that with typeof(Embrace.Common.i18n.Strings).FullName.

Future steps

I have not given up on my string literals yet. In the future, I hope to come up with an approach that would let me use them again, perhaps adding a build step extracting them into RESX and replacing them with resource keys before compilation. I'll then update my notes with the solution.

The truth about puppies

2020-05-21T19:00:00Z

I have wanted to have a dog since I was a kid. Now, at the ripe old age of 43, I've finally got one. An amstaff puppy.

Meet Becky.

It's only been two weeks but I already feel like an expert on all things puppies. And let me tell you: you've been lied to. Here, I want to set the record straight. To tell the truth about puppies.

First things first. Puppies are dumb. Almost as dumb as human babies. I should know, I made three of those. They get smart eventually. With dogs, I am not so sure. I watch the grown up dogs in the dog park and they behave just as dumb as my puppy. So there.

Doing dumb things all the time, you'll spend an inordinate amount of time preventing them from happening or dealing with the aftermath. Are you running a business, or a side hustle, or have anything substantial going on in your life? If so, be prepared to put that aside once you get a puppy. You won't have the time anymore.

Whoever's every claimed dogs are smart is either lying or has sank down to their level. You'll soon learn why. Most of it has to do with their bad eating habits and running into dangerous or gross things.

As does Becky.

When getting to know you, she might lick you. You could be forgiven for thinking it's her way of kissing - far from it. Instead, she is tasting you and calculating the risk profile of biting the flesh off your hand. You know that's what she does because eventually she'll start biting you. And those puppy bites, they hurt! You've got to slap the bitch around until she learns not to do that.

Don't let her cute facial expression mislead you. She is not a primate like us. When she looks at you, it's only to get something to eat. And if that's not forthcoming, she'll drop the cute face fast, and attack.

Speaking of eating: yes, dogs are technically carnivores, in that only meat and bones do anything worthwhile for their metabolism. That does not preventing them from munching on anything they stick their faces into, however. And more gross it is, the better!

Licking dove shit as soon as we get out of the apartment is just the entrée. Cigarette butts, stones, plain dirt, leaves, sticks, shit of other dogs, human shit, she loves this stuff. Until something get stuck in her dirty little mouth, that is. Then you've got to pry it open and extract the shit while getting bit.

Out the door you have to get otherwise she makes a bathroom out of your living room. And so you depart, five, six, seven times a day to walk your dog.

Whoever invented the phrase "walking a dog" should've also invented a more apt phrase for taking a puppy out. Because walk you will not.

More appropriately, it could be called "sitting with your dog outside" as you can't take more than two, three steps before she'll sit down to observe the goings on.

Offering a treat can help but only to make her move towards you to take it, then she'll sit down again. Soon enough you start yelling. Lately, I've also deployed corporal punishment into my vocabulary but these beasts can take a lot more abuse than I can give.

If the surroundings are familiar to her, she'll occasionally run around or even walk by your left leg like a good dog she's supposed to be. In new areas, she would mostly sit and watch or sniff around like a Mossad spy in a hostile territory.

Being a puppy, she's interested in everything. I suppose that's a good thing. I remember being interested in things.

Like other people, for instance. They make an inviting gesture, and she runs toward them, ready to cuddle, wagging her tail like it's the first time she met someone nice. You stand there, leash in your hand, a pathetic cuckold.

I don't know about other dog owners but I just hate it when random strangers pet my dog without permission. Something about the act just sets my blood to boiling temperature instantly.

Other dogs are fine, usually. They get she's just a puppy, let her sniff their asses, and move on.

The outing can take anywhere from thirty minutes to three hours. You have to wait until the poop comes out otherwise it comes out in your living room. To be honest, this type of incident has only happened a few times with her so far. That's because we are out most of the time trying to prevent that from happening again.

How the puppy behaves in your apartment really depends on your ability to tire her outside. The ideal puppy comes home and sleeps instantly. Before I learned that, I had to watch her at all times to take inappropriate objects of affection out of her mouth.

I suppose it comes down to expectations vs. reality. And whoever sets these expectations should have a class action coming to wipe them out. It's all cuddling and affection and unconditional love when it's someone else's daily problem, I suppose. In actual reality, on planet Earth, you've got to work your ass off to get an ounce of payback out of your puppy.

Feeding her, playing with her, walking (kinda) her outside, buying things for her amusement... the list is endless. Nothing unconditional about her behavior, quite the opposite. She gets what she wants, maybe you get a lick or a tail-wag. Then back to the grind. When she sleeps, you try to recover. Perhaps you also get some sleep. And then she wags her tail when she sees you after waking up in her cage, tilts her head just a little bit, and a new day begins.

This is the truth: you are not getting a dog. The dog is getting you.

PS Yeah, of course I am kidding. Sort-of. I love my Becky, the litte bitch. It gets a little overwhelming at times.

Adventures with Makeblock mBot and Typescript

2020-04-07T08:00:00Z

I have so far not been successfully in this "teach your kids programming" thing. My three sons are technical but have not yet demonstrated any inclination to code.

Thinking I could help them find the inner fire, so to speak, I bought a programmable "robot" from China a few years ago - the mBot by Makeblock. It then proceeded to sit on my shelve and collected dust.

The thing was, I couldn't get past the block-based programming environment in mBlock IDE. It felt unnatural and constrained - I've only ever programmed in a text editor / IDE, and these visual tools confuse me.

However, having had zero experience with Arduino, upon which the mBot is based, I could just effortlessly drop into the text editor and start hacking. And I suspect that even if I could, it would not be the most friction-less way of teaching my sons to code.

On the shelf the robot sat.

Come the Great Corona Lockdown of 2020, I was gifted an extra portion of free time. One day I noticed the blue robot staring at me from the bookshelf, motionless. Hey, buddy! Could we rekindle our friendship?

It turns out we could. As it happens, I can talk to my robot in Javascript. Or better yet, Typescript.

What can the mBot do?

Simply put: it moves (two wheels with motors at each side with a single front wheel for support), it sings (or rather, squeaks), it blinks (two programmable LEDs), it can sense distance, follow a line, and when equipped with additional sensors, it gains additional powers.

Importantly, it has Bluetooth, and can wander about as instructed, wirelessly.

I think that's a pretty good basis for exploration and tinkering.

The mBot Javascript SDK

Suffice to say, I am late to the party. Perhaps too late?

There's a repo by Makeblock with Javascript SDK that was last updated in December 2016. Getting to run the simplest example on my robot was not trivial.

I had to downgrade Node to 8.9 to be able to compile the native bindings of some 3rd-party dependencies, which led to me install nvm for Windows as I don't intend to use this ancient version for my daily business.

The examples then ran but looking at the code, I did not feel motivated to expand on it and work out my own examples. After having adopted Typescript a few years back, I did not want to return to plain-old untyped Javascript.

What would it take to convert this into a Typescript project?

I didn't want to rework the existing samples, of course, but rather write my own and learn how to talk to my robot in a typed fashion.

Bringing the dependencies into 2020

The first step was to update the dependencies. The repo relies on the johnny-five and node-pixel Javascript frameworks to do the heavy lifting, and I wanted to have their latest features at my disposal.

I bumped Node to 10.*, which seemed to suffice.

After managing to install everything, which on Windows is not without pitfalls, the code ran without erroring out before reaching my robot but then failed because of an incorrect, or rather unexpected, firmware version.

The firmware we're talking about is called "Firmata" and implements a protocol for talking to microcontrollers (such as Arduino).

What I had to do was add the mBotFirmata sketch to the latest version (2.5.8) of the Arduino firmata, adding the drivers for the WS2812 LED strips used on the mBot from the node-pixel repository, and - surprisingly for me - it almost worked on the first try. Due to a defect in the current version of node-pixel that apparently does not support the newest firmata, I had to patch a single line in the node-pixel library to get past the startup errors.

Lo and behold, the LEDs blinked, the the wheels moved, my robot came alive.

Typescript FTW

Starting from scratch, using Typescript is trivial. Adjust the tsconfig.json to your liking / standards, and you are good to go.

It gets trickier with your dependencies and whether they've published their types or not. It turned out that johnny-five has while node-pixel has not.

To make my IDE experience smoother, I've stubbed some interfaces for objects and their methods from node-pixel that I'm going to be using and will add to them as I progress. Perhaps I can submit a PR at the end of this exercise.

It's hard to overstate how much value Typescript adds to a Javascript project. Even if this is just a toy project, I still want the solid Typescript refactoring experience in VS Code and be able to spot stupid errors and typos in an instant.

My first mBot program

Let's not shoot for the Moon and start simple. I want the robot to wander around my sizable working desk, blink, and if it sees an object approaching, emit a panic signal and back-off.

Current status: it only drives forward and would fall off the desk if I don't stop it. When it senses my hand, it blinks rapidly and backs off for a moment.

I don't think there's a sensor that would tell the robot it's about to fall off, and so I might try to put it in a playpen or something.

You'll find the code in the toys/scared.ts file if you want to see what I've done so far. Not much to write home about yet, frankly.

It's a bunch of functions but I'll convert it to a class as I think the object-paradigm fits it like a glove, given that the mBot is an actual object in the actual universe.

Could I use this to teach my kids to code?

Yes, I think I could, now.

What I'd like to do is to put together a few useful classes, abstractions on a sufficiently high level that they would feel like those "blocks" from the MBlock IDE without having to be visual.

I think that since programming is text-based, teaching programming should also be driven by textual interaction. The right level of abstraction is crucial, though, as I'd rather not start from bytes.

Having said that, the awareness of the existence of bytes (and strings, and numbers, etc.) early on will be useful in the education of my kids, should they eventually become interested. That's why I'll continue playing with my mBot with Typescript and not just Javascript. Hopefully I'll have something interesting to show them someday soon.

LiteDB vs SQLite: what works better for .NET developers?

2020-04-07T08:00:00Z

SQLite has long been the gold standard for desktop, local, or embedded databases. It's been around forever, is unit-tested from top to bottom, and performs exceptionally well when used as intended.

C# developers have used various abstraction layers to access its APIs more in a way that would be more idiomatic. When I looked for one I could use in my desktop app for tango DJs, Bewitched, I went with SQLite-net.

I use it to this day. Anything Frank Krueger creates is beyond awesome! :)

Because SQLite is written in C, it isn't a native citizen of the .NET ecosystem. That is perfectly fine in practice, as the native library will be included in your build, and your domain code doesn't need to know it's talking to a non-dotnet code.

Still, it bugged me occasionally. Could there be a comparable database built for .NET?

Well, there is. It's called LiteDB, and it's pretty amazing.

The fundamentals of LiteDB

LiteDB is 100% C# for .NET 4.5 / NETStandard 1.3/2.0. You run Install-Package LiteDB, and you get a single DLL. No interop assemblies, no x32/x64 hassles. The advantage of this is hard to overstate.

Just like SQLite, LiteDB:

is serverless,
has ACID transactions,
uses a single file for your data and indexes

In addition, it supports an external file storage mechanism for blobs greater than 1MB.

Unlike SQLite, LiteDB is a document database.

SQLite has tables, which have rows and columns. You define the schema, decide on the data types and choose which fields are required and which can have NULLs.

LiteDB has collections, just like MongoDB. Collections of documents. Each document has to have an ID, but otherwise it's up to you to define what the document should contain.

Its mindset is closer to MongoDB, and if you've done any work on the MEAN stack, it will feel very familiar to you.

Its API is document-oriented. BUT - if you prefer SQL, you can talk to it using a very SQL-like syntax - with SELECTs, INSERTs, UPDATEs, and DELETEs - as of version 5.

The SQL capabilities of LiteDB are impressive. The SELECT keyword has WHERE, GROUP BY, HAVING clauses, and you can order your results as well as limit how many are to be returned.

First steps with LiteDB

You can start hacking in Visual Studio or VS Code by referencing the Nuget package: Install-Package LiteDB.

Don't have any code yet? No problem! Download the LiteDB Studio instead, and play with the database engine directly.

What you need to get started is just some JSON you have lying around.

I exported a sample collection from my local MongoDB instance, cleaned up the JSON to remove ObjectId(...) and ISODate(...) references, and imported it into the app. Obviously, you can construct your JSON any way you like.

When you enter the input file and choose the collection name, the program outputs the import statement for your reference.

As you will see below, it reads very much like regular SQL:

SELECT $
INTO addresses
FROM $file('C:/Users/Tomas/Downloads/addresses.json');

I tried a query on my sample data set:

SELECT $ FROM addresses WHERE city = 'Venice';

Sure enough, the output was what I expected with a data grid of relevant records.

As this is a document database, you won't get JOINs, but the documentation explains how you can include other collections in your query like this:

SELECT $ FROM addresses INCLUDE countries ORDER BY country.name;

The docs include more in-depth info on how the references between collections work. It's a bit more work than defining foreign keys in traditional SQL databases like SQLite.

From my experience with Mongo, when you need referential integrity that you can take to the bank, you need a proper SQL database. For many schemaless (or rather, schema-loose) scenarios, the document approach works just fine.

Programming your app with LiteDB

How do you code your persistence layer using LiteDB?

You don't need DTOs (Data Transfer Objects). Instead of dummy classes, you can work directly with documents like this:

var address = new BsonDocument();
address["street"] = "Palm Street";
address["houseNumber"] = 35;
address["city"] = "Venice";
address["state"] = "CA";

using(var db = new LiteDatabase("database.db"))
{
    var col = db.GetCollection("addresses");
    
    // Insert the document into the database. This will create the collection if it does not exist yet.
    col.Insert(address);
    // ... do some more stuff
    // Look up your document
    var dbAddress = col.FindOne("$.city = 'Venice'");
}

As you can see, it's super straightforward to work with "raw" instances of BsonDocument, which itself is like a Dictionary of a string and one of BSON types.

Or, you can use classes and LINQ to create and manipulate your data in a strongly typed fashion:


public class Address
{
  public string Street { get; set;}
  public string HouseNumber { get; set;}
  public string City { get; set;}
}

using(var db = new LiteDatabase(@"Addressbook.db"))
{
    var col = db.GetCollection<Address>("addresses");
    var address = new Address
    {
      Street = "Palm Street",
      HouseNumber = 35,
      City = "Venice",
      State = "CA",
    };
    col.Insert(address);

    // ... do more stuff
    // Look up your document
    var dbAddress = col.FindOne(x => x.City == "Venice");
}

Starting with version 5, you can annotate a class constructor with [BsonCtor] to tell LiteDB how to construct your class instance. Its properties then no longer need public setters:


public class Address {
    public ObjectId AddressId { get; }

    public string Street { get; }

    public string HouseNumber { get; }

    public string City { get; }

    public string State { get; }

    public DateTime CreateDate { get; }

    public string StreetAddress => $"{Street} {HouseNumber}";

    public Address(string street, string houseNumber, string city, string state) {
        AddressId = Guid.NewGuid();
        Street = street;
        HouseNumber = houseNumber;
        City = city;
        State = state;
        CreateDate = DateTime.now;
    }

    [BsonCtor] // <- here
    public Address(ObjectId addressId, string street, string houseNumber, string city, string state, DateTime createDate) {
        AddressId = addressId;
        Street = street;
        HouseNumber = houseNumber;
        City = city;
        State = state;
        CreateDate = createDate;
    }
}

This way, you tell LiteDB how to construct your domain object without having to use a DTO. That's a big win in my book.

Alternatively, there is a fluent API. This means you can avoid having this database-specific constructor, which does not belong in the domain. Instead, you tell LiteDB how to construct the domain object somewhere in your infra/persistence layer:

var mapper = BsonMapper.Global;

mapper.Entity<Address>()
    .Id(x => x.TAddressId) // set your document ID
    .Ignore(x => x.StreetAddress) // ignore this property (do not store)
    .Field(x => x.CreateDate, "CreatedAt"); // rename a field

For a complete reference, look here

Which is better for .NET developers - LiteDB or SQLite?

There is no definite answer to this question. Other than it depends, of course.

In most cases, you can't go wrong with SQLite. You may think that having a schema is too constraining, but in the end, you'll be glad you have a schema once you've figured out the right model for your domain.

Then again, LiteDB can be just as welcoming to your domain objects. And, it can be more friendly to you while you are mapping it out.

No need to worry about migrations, for example. Assuming your persistence layer is forgiving enough about missing fields and such.

I would say that in the end, it's almost a question of esthetic preferences. Both databases will do a great job where they perform best, which would primarily be a desktop or mobile app.

There, the trade-offs are very much different compared to a web app that may receive updates several times a day and is being used by dozens of hundreds or users concurrently.

And if the .NET-native API of LiteDB appeals to you, why not give it a shot? Chances are indeed very good that it will delight you!

UPDATE October 2022

Would you like to dig deeper into the subject?

I have released a follow-up e-book taking an in-depth look at LiteDB and how it performs.

It has many more code examples, more details on the API usage, and features a performance shoot-out.

Grab your copy now! It's free (or you can pay what you want).

Are NanoVMs the future?

2020-03-22T11:33:00Z

This brilliant take-down of the containers ecosystem made quite a splash yesterday, with the predicably heated discussion on HN.

The author's key point was that containers are ihnerently difficult - if not impossible - to secure - and that we need new approach, one that is breaking from the outdated concept of multi-user, multi-process operating systems we have today that do not belong in the cloud.

This reply from the commenter "api" summarizes the situation:

I've been saying for years that containers are nothing more than an ugly hack to get around the fact that OSes are broken. More specifically modern OSes still carry 1980s assumptions about multi-tenancy, system administration, application management, state management, etc.

In the 1980s a "server" was a big expensive special snowflake. Each server had a "system administrator." The system administrator would "install" software "on" the machine.

I quoted all those words to highlight no longer valid concepts.

Boxes are ephemeral now. Systems have more than one administrator, sometimes belonging to a different organization or department. Applications are not installed "on" the system in the sense that they root into it and modify its state. If they do this makes them a terrible hassle to manage. Applications should be ephemeral things that can float freely between machines and store their state in some kind of distributed or remote data store or database, or at least can be easily migrated between hosts. Everything is mobile. Everything is commodity. Everything is liquid.

OSes are just not designed this way, so we have an ugly hack called a container that basically amounts to "let's tar up images of operating systems and just treat them like giant static binaries." Onto this is bolted a whole array of other ugly hacks to make that easy to deal with, but the concept is fundamentally ugly as hell.

I'll admit my expertise in information security isn't solid enough to evaluate the argument on its technical merits. During development, containers are a God-send in that I can quickly spin-up dependencies without going through a lengthy install-and-configure process. In production, I'll take whatever approach offers the best security given the requirements.

The author is a CEO at NanoVMs, and he claims that the answer are unikernels and micro/nano VMs where your application is the only thing that's running.

I don't have a problem with the obvious bias his role brings to his argument; I'll take a report from the trenches with a lot more seriousness than one from an Ivory Tower journalist.

Interested, I looked around to see if I can try this tech at home for personal education.

First steps with Nanos

From the README

Nanos is a new kernel designed to run one and only one application in a virtualized environment. It has several constraints on it compared to a general purpose operating system such as Windows or Linux - namely it's a single process system with no support for running multiple programs nor does it have the concept of users or remote administration via ssh.

I am not going to be hacking on Nanos but instead use it for a toy project and see if I can get it running locally.

To that end, README tells me to use the Ops orchestrator to run my application.

There, I open the Getting Started guide and find out that Ops currently supports only *nix systems. Does my WSL2 count as one? I'll find out.

First, I install qemu:

sudo apt-get install qemu

Next, I trust Ops to do the right thing and install it from the web:

curl https://ops.city/get.sh -sSfL | sh

It downloads and install into my user's directory. I source the .bashrc file to put its install location into my $PATH.

Time to run the first app! I copy-and-paste the example from the homepage:

var http = require('http');
http.createServer(function (req, res) {
    res.writeHead(200, {'Content-Type': 'text/plain'});
    res.end('Hello World\n');
}).listen(8083, "0.0.0.0");
console.log('Server running at http://127.0.0.1:8083/');

Then I try running it:

ops load node_v12.13.0 -p 8083 -f -n -a hi.js

Oops:

booting /home/tomas/.ops/images/node.img ...
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
assigned: 10.0.2.15
buffer.js:573
    slice: (buf, start, end) => buf.utf8Slice(start, end),
                                    ^

RangeError: Index out of range
    at Object.slice (buffer.js:573:37)
    at Buffer.toString (buffer.js:770:14)
    at Object.readFileSync (fs.js:386:41)
    at Object.Module._extensions..js (internal/modules/cjs/loader.js:972:22)
    at Module.load (internal/modules/cjs/loader.js:812:32)
    at Function.Module._load (internal/modules/cjs/loader.js:724:14)
    at Function.Module.runMain (internal/modules/cjs/loader.js:1025:10)
    at internal/main/run_main_module.js:17:11 {
  code: 'ERR_OUT_OF_RANGE'
}
exit status 3

Not going to debug this now, I look up what other versions of Node are supported on Nanos via ops pkg list and try 11.5.0.

booting /home/tomas/.ops/images/node.img ...
qemu-system-x86_64: warning: TCG doesn't support requested feature: CPUID.01H:ECX.vmx [bit 5]
assigned: 10.0.2.15
Server running at http://127.0.0.1:8083/

I navigate to localhost:8083, and indeed, "Hello World" it says in my browser. Neat!

WSL is a marvel. I can experiment with tech that is Linux-only without having to dual-boot or fight with Virtualbox. I will now try to run a more sophisticated app on Nanos and report back what I find.

Ideal remote working setup for a mature introvert

2020-03-06T12:20:00Z

I've freelanced for over 10 years now, most of which I spent in my clients' offices.

That began to change, and I now work primarily from my home with occasional travel to in-person meetings.

The change was intentional and I prepared for it, both mentally and technically. I think you should, too, if you contemplate a move to remote working.

COVID-19 notwithstanding, it seems like the industry has been under transformation to more flexible working models anyway. Whether or not this health scare will accelerate the process, it pays to be prepared.

Here's what works for me.

Pre-requisites

I am going to assume a few things about you.

You are either freelancing or working within an organization that trusts you. While this article outlines how to create the environment for productivity at home, it does not address negotiating terms with your employer. You're going to have to resolve any trust issues first.

Second, I'm assuming you are an experienced professional able to work independently. If you are just starting out, doing it completely remotely may not be the right move for you. I, for one, have benefited immensely by being able to rely on the physical support network of peers and mentors when entered the industry.

Lastly, I'm assuming you are able to control the physical environment around you. That includes being able to declare a portion of your house or apartment as your office, furnishing it according to the needs of your work, creating and maintaining "office hours", making the necessary purchases, etc.

This is hard to overstate! Unless you live alone, you are going to negotiate with your spouse or any other humans sharing your living environment.

The importance of mindset

Let's start inside your own head.

Don't worry, I am not a "mindset consultant" :) I don't want to change anything, just let you ask yourself a couple of things. And, instead of telling you what to think, I'll just share my own response.

How comfortable are you spending hours and hours alone, in general?

My response: I am naturally introverted and like it when I am alone in a room for a few hours when I have things to do. A prolonged solitude would be uncomfortable, though. I like to space my day with pleasant interactions with people I like.

How well and for how long can you focus on the task at hand when nobody is looking?

My response: that's the only way I can focus. On the contrary, when I have a feeling that people could peer over my shoulder anytime, I feel vulnerable and disconcerted.

How well are you able to set and enforce boundaries between your work and the rest of your life?

My response: I don't have to discipline myself, because I was born lazy and never suffered from the ambition to work 80+ hours a week. When out with friends or my spouse, I don't check my work-related e-mails or Slack. And, when working, I refuse non-work interruptions to distract me. Either way, I am trying to be mindful of what I am doing at each particular moment.

Examine your responses, if you will. Are your answers compatible with being at your home office most of the time, interacting with your peers or clients over Skype only, and moving seamlessly between the "work mode" and "free-time mode" without sacrificing either your work results or the quality of your life?

Counter-indications: you like the physical company of peers, you like bouncing ideas off each other regularly or just shooting the shit at the water cooler, you get weirded out when alone for a few hours, or you need a boss around to make you lift a finger, so to speak.

Only you know.

Negotiating your living arrangement

This section applies to you if you share your living space with another human, e.g. your spouse.

You are about to transform this space into a mixed-use territory. Some of it stays dedicated to leisure but a significant portion is now your office.

In my case, about a quarter of our living room is my office. It's where the TV would be. Thankfully, my 43" monitor can also fit that need when we want to watch Netflix.

Regardless of how much space you fill, that room becomes an office when you work. You will impose limits on what others can do there while in the working mode.

Potential impacts: your spouse cannot have a friend come over and talk over coffee in this room, your children cannot play there while you work, and when you have a conference call, any other occupants must be quiet.

Then there's your presence in itself.

If your partner works outside your home and you remain home-bound, requests to "take care of stuff" might arrive. You are always at home, after all.

No, actually you are at work, it just happens that your workplace is also your home.

Let's say your partner also works at home while you used to work outside. Now you are dealing with the constant presence of each other. Hopefully your relationship is great but this is an impactful change nonetheless!

Well, I am not a "relationship consultant" either, and so you'll have to figure this one out for yourself.

For me and my partner, taking a few hours apart from each other daily works really well. I do have a need to take a walk, go to the gym, or do something outside every day in any case.

Expect this negotiation to take some time and do it in good faith.

Staying in touch

This section applies to people working inside a team. If you are a lone-wolf freelancer and you only communicate with your clients, chances are that you have already figured this out.

Let's assume the worst possible starting position: your team was sent home without any prior remote-working setup having been put in place.

You've just declared the dining table your office and put your laptop there.

You are going to have to coordinate with your team-mates daily and keep your sanity. What works, and what doesn't?

E-mail and chat - those work great when everyone is remote, and I don't get the occasional complaints about "Slack-overflow". It takes some discipline to mute Slack's incessant notifications but it can be done.

What must be done is managing the (a)synchronicity. What are the expected response times? Chat is terrible when used synchronously. That said, if you depend a lot on collaboration from others, getting to a shared agreement on each other's responsiveness is important.

This topic extends way beyond remote working as email and chat are used inside onsite teams as well.

Video conferencing - awesome invention that never works 100%. Expensive corporate setups are the worst. Try them all and decide on which one sucks the least. I hear good things about Zoom but haven't tried it yet.

Ticketing system - I like this one the best. If your team already uses something like Jira (the worst piece of garbage on planet Earth but way better than no ticketing system at all), great - use it for everything!

Need someone to review your PR? Give them a ticket.

Got stuck with a bug-fix? Create a ticket for your senior colleague.

Want a... you get the point.

A well-written ticket has a textual description of what you need as well as all the why's and when's. You leave it with the assignee who is a mature adult just like you and will handle it with the proper importance. It's asynchronicity at its best.

Do what you can to avoid synchronous tools in favor of asynchronous.

The tech

OK, now for the fun part.

Actually, it is going to be rather brief.

You probably already have a laptop for work. What you really need to add are two things: a desk and a good chair.

Any desk will do but a good chair is an investment that will repay itself many times over, measured in units of health.

Unless you work for a unicorn company (I heard many in the US are), your work chair is most likely shitty.

Don't treat yourself as your employer treats you, and buy yourself a quality ergonomic chair. Mine lets me stay put at my desk for the whole day without any back pain; its lower back support is fabulous (link in Czech only).

Considering your desk, I strongly favor motorized standing desks. Mine has just two buttons, up and down, and that's good enough, no need for presets - I just stand up, push and hold the button, place my hand on the table and wait until it raises to the comfortable position.

I can only work standing for a maximum of 20-30 minutes but doing it a few times each day works wonders. A human body did not evolve to be in one position most of the time.

Other resources

I've outlined the things that are important to me in my personal working and life situation.

What works for me may or may not be relevant to you. There are many other sources to visit.

Scott Hanselman published a thoughtful article on his remote working setup. He's got a lot more tech topics covered as well as many, many additional tips.

I heard good things about the book Remote - office not required by the makers of Basecamp.

At the end of the day, though, success in remote working does come down to two major pre-requisites: your mature mindset and the culture of your company or team that you can hopefully co-create.

These days, companies are taking a hard look at remote working. Use it to your advantage.

Managers who are used to face time will have to re-program themselves. Help them. Educate yourself and promote healthy asynchronous communication that enables remote teams to function well.

Using Nuget packages with .NET Interactive

2020-02-11T07:00:00Z

Last time I installed Jupyter with .NET and tried to run a few commands to get a feel for what it's like.

Before starting today's session and exploring how I could use Nuget packages, I wondered about the limits of Jupyter / .Net Interactive. Specifically, what kind of code I can execute.

Does it run within a sandbox?

At first glance, it does not appear to do so.

This is fine on my PC where I am the only one responsible for what code gets executed, not so much if I wanted to host a Jupyter notebook online.

Nuget packages in a notebook

The docs for .NET Interactive are sparse as of today. I'm exploring it using a few samples that are well hidden inside the repo, specifically a sample for Importing packages, libraries, and scripts.

Let's say we want to play around with taglib-sharp. First, we reference it:

#r "nuget: TagLibSharp, 2.2.0"

In the next cell, we can import its namespace, instantiate its classes and invoke their methods.

First, let's expand the original Song class to work with more metadata attributes and make TagLib a dependency.

public class Song
{
    public Song(TagLib.Tag tag)
    {
        _tag = tag;
    }

    private readonly TagLib.Tag _tag;

    public string Title => _tag.Title;
    public string Artist => string.Join(", ", _tag.Performers);
    public string Genre => string.Join(", ", _tag.Genres);
}

The rest is straightforward:

var tag = TagLib.File.Create(@"C:\Users\Tomas\Music\Library\Juan D'Arienzo\Todo de Juan 5\Ansiedad 12562-1_RP.flac").Tag;
var song = new Song(tag);
display(song);

Lo and behold, the code can access my song and read its metadata:

Very cool.

~~In the next installment, I will look at my hosting options, what the security boundaries are, and what it means for deployment.~~
PS I reviewed Jupyter docs for public server installation, which is single-user and not really what I meant by "hosting".

The docs led me to JupyterHub, which seems appropriate for class-room use. From what I was able to understand, it's still not meant for a wider public use (aka, "come to my site and play with my notebooks, anonymously"). For even more users, there's Jupyter on Kubernetes - I know Docker and k8s well enough to not want to pay for the infrastructure if all I wanted to do was to expose a little playground.

Microsoft's own docs talk about Binder, which takes the pain (and cost) of Docker deployment out of your hands. As much as I wanted to do this on my own, that's what I would probably use.

First steps with .NET Interactive

2020-02-07T16:20:00Z

Microsoft has recently announced support for C# and F# in Jupyter Notebooks. On February 6, they announced a new release, Preview 2, with some naming and organizational changes.

Having recently listened to an episode of the Merge Conflict podcast where Frank Krueger spoke quite affectionately about Notebooks, I decided to give this a try.

To be sure, I have never experimented with Jupyter and my knowledge of Python is superficial at best. That is why I've only got to it now when I can use C# with Jupyter.

Installation experience

I decided to follow the first steps for the "dotnet/interactive" experience.

The installation of Anaconda took quite some time on my Ryzen 3950X desktop. It spent most of the time initializing its package cache or something.

The installation of dotnet global tool Microsoft.dotnet-interactive and a .NET kernel for Anaconda went by in a split second.

So far so good.

Running Jupyter locally with .NET

I continued with a tutorial from Microsoft's dotnet Github repo.

I launched the Anaconda Navigator, which is a Windows app, and from there I chose to run a Jupyter Notebook.

A browser windows appeared. Bummer, a web app.

I created a new file. Can I do a Hello World here? Sure I can!

Following the next example, I define a new class, instantiate it and peek at its contents.

Nice!

Each box I create, called a "cell", can contain either code or markdown. Therefore, I can create an interactive documentation, for instance for a consumer of my C# library, can I?

I'd expect that a markdown cell will get rendered as HTML. It does, sort-of. Out of the box, it does not support smileys, for example. This SO answer, however, provided a way: just copy-and-paste it! 😉

This is how the notebook looks like now with markdown content included:

This is indeed very intriguing. Next, I am going to explore what other code I can run there. For instance, can I reference Nuget packages? Load DLLs?

Also, I'd be keen to see how I can host this notebook for the consumers of my library. I heard that I can do so directly on Github, so I am going to see about that.

To be continued!

The amazing (and dangerous!) switch expressions in C# 8

2020-02-05T15:37:00Z

I revisited my tentative understanding of the new pattern matching tools in C# 8 when it made rounds on Hacker News today.

Given that C# started out as a general purpose language within the OOP lineage, it's interesting to see it adopt some of the features made popular by the functional community.

What it also does, though, is to place seeds of doubt in the mind of a curious programmer.

What is this functional stuff? How far can I go with it until my peers will start declining my PRs because they can no longer grok my intent from the code?

When given a choice between procedural and object-oriented mindset, I choose the latter. Similarly, when comparing procedural and functional, the latter also wins in my mind. Procedural is evil, error-prone, inelegant, even vulgar.

It gets more complicated if given a choice between object-oriented and functional. In the context of C#, I am predominantly writing OOP code with some functional patterns mixed-in by way of experimentation.

Not being particularly well-versed in the functional idioms, I take the occasional learning forays into them when they make appearances in C#.

Today's post is about the switch expressions.

The new switch expression in C# 8

The switch expression is a very elegant concept that replaces a procedural statement with an expression. So far so good. Or great, actually.

Switching over simple values

Suppose I want to map a numerical score to a textual evaluation. Before, I could write this:

string evaluation;
if (score >= 0)
{
    evaluation = "top performer";
}
if (score >= 50)
{
    evaluation =  "mediocre";
}
else
{
    evaluation =  "poor" // default;
}

I don't know about you but the word if always gives me creeps. And what offends me even more is the split between the declaration of initialization of the evaluation variable. Gross.

Now, I can write this:

var evaluation = score switch
{
    var x when (score >= 80) => "top performer",
    var x when (score >= 50) => "mediocre",
    _ => "poor" // default
};

Better: no ifs and the variable is declared and initialized in a single step.

There's one limitation I do not like. Suppose I have this silly enum:

public enum Colors
{
    Red, Yellow, Orange, Blue, Black
}

Let's say I want to match a color to a string of either "warm" or "cool". I would like to be able to do this:

var mood = color switch
{
    Red, Yellow, Orange => "warm",
    Blue, Black => "cool",
    _ => "undefined" // or whatever
}

Unfortunately, that won't compile. According to this helpful SO question, the way to go is:

var mood = color switch
{
    var x when ( x is Colors.Red || x is Colors.Yellow || x is Colors.Orange) => "warm",
    var x when ( x is Colors.Blue || x is Colors.Black) => "cool",
    _ => "undefined" // or whatever
}

Notice the need for a new helper variable x in the switch expression combined with the when filter. I can live with it but it's not as elegant as it could've been.

Switching over tuples

I quite like using tuples as throw-away data carriers used in a private context, inside a method. Their support in modern C# is amazing, and yes, you can now switch over them.

This example is adapted from the Microsoft docs. Notice the dependency on LanguageExt.Core, which gives me the wonderful Option<T> struct.

using System;
using LanguageExt;

namespace PatternMatchingCs
{
    public enum Hand
    {
        Rock, Paper, Scissors
    }

    public class RockPaperScissors
    {
        public Option<Hand> Round(Hand first, Hand second)
            => (first, second) switch
            {
                (Hand.Rock, Hand.Paper) => Hand.Paper,
                (Hand.Rock, Hand.Scissors) => Hand.Rock,
                (Hand.Paper, Hand.Rock) => Hand.Paper,
                (Hand.Paper, Hand.Scissors) => Hand.Scissors,
                (Hand.Scissors, Hand.Rock) => Hand.Rock,
                (Hand.Scissors, Hand.Paper) => Hand.Scissors,
                (_, _) => Option<Hand>.None // to be interpreted as a tie
            };
    }
}

The enum Hand has three members and the method takes two arguments on this enum type, and so the number of combinations is still manageable.

I would say that with more enum members and/or more combinations, this feature would quickly cease to be elegant or maintainable. Here, I like it.

Switching over more complex objects

I'll follow with the usual examples as seen in the Microsoft docs, and introduce classes Triangle, Rectangle, and Circle.

class Triangle
{
    public double Base { get; }
    public double Height { get; }
}

class Rectangle
{
    public double Width { get; }
    public double Height { get; }
}

class Circle
{
    public double Radius {get;}
}

Notice that these are basically DTOs, "dumb" classes without any methods. Data carriers.

Suppose I want to calculate the area. Using the functional way of thinking that decouples data and functionality, the switch expression lets me do this:

double AreaOf(object param)
{
    return param switch
    {
       Triangle t => t.Base * T.Height / 2,
       Rectangle r => r.Width * r.Height,
       Circle c => c.Radius * c.Radius * Math.PI,
       _ => throw new InvalidArgumentException("Unrecognized parameter")
    }
}

So far so good? Not really.

The problems

There are two things I don't like about the examples like this one that I see around. (This one is from the official docs on Pattern Matching).

Number one, object param.

We've gone a long way from untyped variables of the yesteryear, and are they now coming back?

When would I want to write a function that can take an arbitrary object as its parameter but can only process a few specific classes? Never, that's when.

Which brings me to number two, throw new InvalidArgumentException(...).

This is a direct consequence of being flexible on the API surface but actually constrained in the implementation.

The function promises to return a double value when given an argument of an object but will mostly throw exceptions if you consider the billion possible subclasses of object that you could give it.

I don't think that throwing exceptions fits the functional paradigm (the functional folks have the concepts of Option or Either instead). It does fit the OOP mindset, but the object param doesn't. So what is this, really?

Granted, these are only examples of what you could do, not recommendations of what you should do. And I think that the other examples given therein, such as switching over enums, are sensible. If you must use enums, that is.

Third, why decouple the functionality from the object? Is a (possibly static) function operating on "dumb" data better than a "smart" class and if so, when and how?

Before, we were being taught to have an interface like this:

public interface IShape
{
    double Area();
}

And then have Circle, Triangle, and Rectangle implement this interface. By way of this useful abstraction, we then get:

public double CalculateSomething(IShape shape)
{
    var area = shape.Area();
    // do something with it and return the result
}

That way, any object with an area will implement the interface and will be guaranteed to give me its area. No having to inquire about its type at runtime and deal with exceptions when I cannot determine it.

Also, no problems in having obsolete switch expressions that do not contain logic for new types I've added since writing them.

When facing a problem like this, in that my system deals with subjects that have some things in common, my esthetic preference goes toward OOP with its polymorphism, with its interfaces, classes that implement them, decorators, the whole deal.

Elegant but dangerous

In summary, these switch expressions look very elegant when used for "primitive" values, fairly good when used for processing tuples, and somewhat questionable when used to fork behavior based on parameter type.

Their syntax is beautiful and they replace ugly procedural statements with expressions, with all the advantages this implies for code maintainability, testability, and easiness to grok at first read.

When misused, they let a naive programmer think he's using functional concepts when clearly that is not the case, such as when breaking promises of function declarations / API and throwing exceptions when not being able to deal with inputs.

As always, I could be wrong in many ways about this, and will update this post as I learn more about this functionality.

How to create reports in MongoDB with just some Javascript

2020-02-01T16:20:00Z

I didn't appreciate what I had at my disposal when first working with MongoDB.

It frustrated me by not being an SQL database, by not having joins and transactions. I came to it with the SQL mindset and it did not comply with my expectations.

It took me a long time working on Express apps in Node.js until I realized, when working on a support ticket, that I could use Javascript directly in Mongo, without having to write a Node app.

When would I take advantage of that?

Suppose the customer wants to receive a one-off report of some kind. If you have a Postgres database, you write some SQL. With Mongo, you write some Javascript.

Let's consider a simple example. Your marketing wants to receive a list of customers who have placed something in their cart but did not finish the checkout process.

Let's say you have two collections: customers and orders. The report ought to contain the user's name, e-mail, and the name of the most valuable item in the shopping cart.

Furthermore, assume that the items placed into the shopping cart are stored in the orders collection as sub-documents.

This is how I would go about it, roughly:

let unfinished = [];
let reportingPeriodStart = new Date("2020-01-01");
let orders = db.getCollection('orders').find({
  createdAt: { $gte: reportingPeriodStart },
  isPurchased: false
}).toArray();
for (let order of orders) {
  let customer = db.getCollection("customers").findOne({
    _id: order._customer
  });
  unfinished.push({
    name: user.fullName,
    email: user.email,
    item: order.items.reduce(
      (best, current) =>
        current.price > best.price
          ? current
          : best,
      { price: 0 }
      )
  })
}
print("Name;Email;Item")
print(unfinished.map(o => `"${o.name}";"${o.email}";${o.item.name}`).join('\n'))

You'll save this file somewhere, and run the mongo client to receive your report:

mongo -u username -p password --quiet mongodb://localhost:27017 yourfile.js

(COMPLAINT: even with the --quiet flag, mongo still prints out useless status information to STDOUT, necesistating further processing with your favorite Linux commandline tool)

As you can see, you can get around the lack of SQL by writing simple procedural code that can find, filter, sort, and otherwise manipulate the results.

Personally, I would still very much prefer to write SQL. It's declarative and you could get the same result as from the above example with less amount of code.

MongoDB does not have SQL but it has Javascript. It's not the next-best-thing to SQL but it's a lot better than nothing at all. I don't have to write an extra app to get my data out, and that is very good indeed.