Performing End to End Testing in Blazor with Playwright

You’re simply a test, better than all the rest.

Photo by Sigmund on Unsplash

End to end testing of front-end applications is a critical aspect of your job as a developer to show that the required work is done. Many front-end testing frameworks exist (e.g. Playwright, Cypress) with the aim to make front end testing as easy as possible. Despite this, if we’re being honest with ourselves, testing is difficult. Changing third party component libraries, unpredictable test environments and even well-meaning changes by colleagues can make testing a challenge even at the best of times.

This article provides a starting point for testing Playwright and Blazor that I wish I had starting out testing. My hope with this is that you don’t have to repeat the same mistakes I did and that you will know where to find the right resources to solve your particular problem. Note that the advice here is Blazor specific, but for the most part can be used to test any web application.

Playwright vs bUnit

To begin with, let’s clear out a misconception I had when starting out. Playwright is not a component testing library like the bUnit component testing library. For whatever reason, the first few weeks of using Playwright I was constantly looking for ways to make Playwright work like bUnit (i.e. mocking dependencies to pages etc.). At the time I had completely missed the point of end-to-end testing which is to deploy the entire system like and provide “production-like” tests the same way a user would use it.

Sample application

All the examples I refer to in this article are available on my github repo. To provide an example most Blazor devs are familiar with, I’ll be using the sample application provided by Microsoft that every Blazor dev has come to know and love. The application is a Server based application but testing this on WASM/Hybrid should be no different.

The classic Blazor template

Testing with selectors

The first (and easiest) way to do testing in Playwright is with selectors. Selectors enable you to narrow down the elements you want to test based on properties e.g. the presence or absence of a particular class. For example, we want to test that the link to the survey always contains the text “brief survey”. To select the link “brief” survey the selector below can be used.

a.font-weight-bold.link-dark

This makes a lot of sense when looking at the generated HTML.

<a target="_blank" class="font-weight-bold link-dark" href="https://go.microsoft.com/fwlink/?linkid=2186158">brief survey</a>

To test this in Playwright, the following code can be used. Note that there is a small amount of setup involved in the tests, but this can be made more compact if the same setup is widely used in your testing. Also note that I use FluentAssertions to do my testing. It’s what I prefer to use, but your team may use something different.

using FluentAssertions;
using Microsoft.Playwright;

namespace UnitTests;

[Parallelizable(ParallelScope.Self)]
[TestFixture]
public class Selectors
{
    
    [Test]
    public async Task SelectorTest()
    {
        //Initialise Playwright
         var playwright = await Playwright.CreateAsync();
        //Initialise a browser - 'Chromium' can be changed to 'Firefox' or 'Webkit'
         var browser = await playwright
             .Chromium
             .LaunchAsync();

         var browserContext = await browser.NewContextAsync();
         var page = await browserContext.NewPageAsync();
        
        await page.GotoAsync("http://localhost:5078");

        var linkText = await page.Locator("a.font-weight-bold.link-dark").TextContentAsync();
        linkText.Should().Be("brief survey");
    }
}

Everything prior to the page.GoToAsync() is setup for the test. Notice how we create a locator based on the selector, get the text and then perform an assertion on the returned text.

There are so many different ways you can use selectors for testing and I won’t even try cover that here. For more info, visit W3C Selectors page.

If, like me, you don’t want to have to memorize every type of selector out there, what you can do is use the Playwright codegen tool to help get you started with the appropriate selector. To do this, you can follow the instructions in the above link or do this within the test as follows.

using FluentAssertions;
using Microsoft.Playwright;

namespace UnitTests;

[Parallelizable(ParallelScope.Self)]
[TestFixture]
public class Selectors
{
    
    [Test]
    public async Task SelectorTest()
    {
        //Initialise Playwright
         var playwright = await Playwright.CreateAsync();
        //Initialise a browser - 'Chromium' can be changed to 'Firefox' or 'Webkit'
        
         var browser = await playwright
             .Chromium
             .LaunchAsync(
                 new BrowserTypeLaunchOptions
                 {
                     //switch to false if you want to view the test visually or use the inspector
                     Headless = false
                 }
             );
         var browserContext = await browser.NewContextAsync();
         var page = await browserContext.NewPageAsync();
        
        await page.GotoAsync("http://localhost:5078");

        await page.PauseAsync();

        var linkText = await page.Locator("a.font-weight-bold.link-dark").TextContentAsync();
        linkText.Should().Be("brief survey");
    }
}

Playwright tests can be run in a mode where you see each test being performed, it can also be run in headless mode for automated testing. To have the codegen tool pop up when running your tests you will need to see Headless = false along with using the await page.PauseAsync(); method.

When recording Playwright Inspector will assist in suggesting an selection method

Drawbacks of testing with selectors

Like everything in software there are things to consider when using selectors. The primary reason to be wary of using selectors is their potential to increase the amount of maintenance you need to do on your code base. The reason for this is simple, third party libraries that you use in your application have no obligation to keep their classes the same over time. They may change things from time to time and a small update to a class name may break all your tests which is not ideal.

Similarly, within your own application your classes may change over time, as your team finds ways to improve the front end. Additionally, if your team decide to add another object to a page which just happens to match a selector you use in a different test, now you need to go and find a way to make your selectors more specific.

Testing with data attributes

Having seen the problems that selectors pose for front end testing, this naturally led me to look for other methods to do testing. Frequently breaking and flaky tests are just a massive waste of dev time if they can be avoided. A natural progression to selectors is the use of data attributes for testing and specifically the data-testid data attribute. This allows you to attach ids onto elements, but unlike normal ids these don’t need to be unique across the DOM. For those coming from a Cypress setup, this is comparable to using data-cy in your testing.

Playwright makes our life really easy when testing using data-testid . For example, suppose that we want to ensure that the navigation link to fetching data reads “Fetch data”. One can easily add on a test id to the html element as follows:

<NavLink class="nav-link" href="fetchdata" data-testid="fetch-data-nav">
    <span class="oi oi-list-rich" aria-hidden="true"></span> Fetch data
</NavLink>

To test this, the following code can be used.

using FluentAssertions;
using Microsoft.Playwright;

namespace UnitTests;

[Parallelizable(ParallelScope.Self)]
[TestFixture]
public class TestId
{
    
    [Test]
    public async Task TestIdTest()
    {
        //Initialise Playwright
         var playwright = await Playwright.CreateAsync();
        //Initialise a browser - 'Chromium' can be changed to 'Firefox' or 'Webkit'

        var browser = await playwright
            .Chromium
            .LaunchAsync();
        
        var browserContext = await browser.NewContextAsync();
        var page = await browserContext.NewPageAsync();
        
        await page.GotoAsync("http://localhost:5078");
        var navText = await page.GetByTestId("fetch-data-nav").InnerTextAsync();

        navText.Should().Be("Fetch data");
    }
}

Using test ids is a great way to ensure that you’re selecting the correct elements and that there’s little room accidentally select the wrong element.

Testing with ARIA roles

Many websites these days have accessibility requirements. Many of these requirements rely on specific ARIA labels being present to assist those with disabilities, such as visual impairments.

Playwright makes your life easy to do testing using aria labels i.e. while setting up tests for your application you can also make them more accessible at the same time!

As an example, our home page has a total of 6 links on it. We want to test that there are 6 on the page whilst also checking that those links are accessible. The code below shows how this can be tested.

using FluentAssertions;
using Microsoft.Playwright;

namespace UnitTests;

[Parallelizable(ParallelScope.Self)]
[TestFixture]
public class AriaRoles
{
    
    [Test]
    public async Task AriaRoleTest()
    {
        //Initialise Playwright
         var playwright = await Playwright.CreateAsync();
        //Initialise a browser - 'Chromium' can be changed to 'Firefox' or 'Webkit'

        var browser = await playwright
            .Chromium
            .LaunchAsync();
        
         var browserContext = await browser.NewContextAsync();
         var page = await browserContext.NewPageAsync();
        
        await page.GotoAsync("http://localhost:5078");
        
        var navLinks = await page.GetByRole(AriaRole.Link).CountAsync();
        //should be 6 links across the home page you can click
        navLinks.Should().Be(6);
    }
}

Note that some HTML elements are automatically associated which specific ARIA roles and this can be tested for using Playwright.

Considerations when using aria-labels

Given what I’ve mentioned above, it’s also worth noting that labels are not a replacement for data test ids. The labels themselves still need to produce some kind of value for someone who is either visually impaired or struggles to see access to the website for whatever reason. There are also a set of standardized roles which you can use as a starting point. See this MDN page for more info.

Filtering locators in Playwright

When starting out with Playwright I fell into the trap of trying to make data test ids that were very specific to the item being tested in order to differentiate them from other items in the DOM. What I didn’t realize is that Playwright allows you to filter elements in the DOM. Taking the previous example one step further, suppose that I want to check that I only have 3 links within my Nav area. The code below shows how this can be done so that you don’t have to create crazy test Ids to differentiate links in the nav menu from links on the rest of the page.

using FluentAssertions;
using Microsoft.Playwright;

namespace UnitTests;

[Parallelizable(ParallelScope.Self)]
[TestFixture]
public class Filtering
{
    
    [Test]
    public async Task FilteringTest()
    {
        //Initialise Playwright
         var playwright = await Playwright.CreateAsync();
        //Initialise a browser - 'Chromium' can be changed to 'Firefox' or 'Webkit'
        
         var browser = await playwright
             .Chromium
             .LaunchAsync(
                 new BrowserTypeLaunchOptions
                 {
                     Headless = false 
                 }
             );
         var browserContext = await browser.NewContextAsync();
         var page = await browserContext.NewPageAsync();
        
        await page.GotoAsync("http://localhost:5078");
        // re-evaluated each time it's used - can be reused multiple times in the test
        var navMenuLocator = page.GetByTestId("nav-menu");
        var navLinks = await navMenuLocator.GetByRole(AriaRole.Link).CountAsync();
        //should be 3 links in nav
        navLinks.Should().Be(3);
    }
}

Note that the locator filtering can be reused multiple times throughout your tests — they are re-evaluated each time you use them. This can help reduce overly verbose testing.

Debugging failing tests

There is honestly nothing more infuriating than not being able to see why a test is failing in a pipeline don’t have direct access to. There are many reasons why a test might fail in a pipeline. For example, you are working on a machine that is typically a lot faster than the hardware running your tests on the cloud. This can result in things running a bit slower on the client side resulting in things like delays in how fast things are rendered to the DOM.

Testing your system on a variety of hardware is a great test as this replicates likely scenarios out there in the wild. Playwright provides functionality to record tests using traces. Traces record everything that is being seen by the machine doing the test. Here are some things that can be captured by Playwright

• what is it Playwright checking for (specifically when using the auto await functionality),
• what were the network calls,
• what were the logs?

Setting up a test to record a trace is very simple. The example code below shows how this can be done.

using FluentAssertions;
using Microsoft.Playwright;

namespace UnitTests;


/// <summary>
/// For more information, see https://playwright.dev/dotnet/docs/trace-viewer-intro
///
/// Also see https://trace.playwright.dev/
/// </summary>
[Parallelizable(ParallelScope.Self)]
[TestFixture]
public class CaptureTraces
{
    [Test]
    public async Task CaptureTracesTest()
    {
        //Initialise Playwright
         var playwright = await Playwright.CreateAsync();
        //Initialise a browser - 'Chromium' can be changed to 'Firefox' or 'Webkit'
        
         var browser = await playwright
             .Chromium
             .LaunchAsync(
                 new BrowserTypeLaunchOptions
                 {
                     //switch to false if you want to view the test visually or use the inspector
                     Headless = false // -> Use this option to be able to see your test running
                 }
             );
         
         
         var browserContext = await browser.NewContextAsync(new BrowserNewContextOptions(){});
         await browserContext
             .Tracing
             .StartAsync(
                 new TracingStartOptions
                 {
                     Screenshots = true,
                     Snapshots = true,
                     Sources = true
                 }
             );
         var page = await browserContext.NewPageAsync();
        
        await page.GotoAsync("http://localhost:5078");
        try
        {
            var navLinks = await page.GetByRole(AriaRole.Link).CountAsync();
            //should be 6 links across the home page you can click
            navLinks.Should().Be(1234);
        }
        catch (Exception)
        {
            var path = "C:/Temp/trace.zip";
            await browserContext.Tracing.StopAsync(new TracingStopOptions { Path = path });
            throw;
        }
    }
}

Note in the above how you are able to set the file location where the trace will be stored.

Viewing trace files

Trace files can be view locally using Playwright’s trace viewer or using a online version at https://trace.playwright.dev/. Steps on how to run the trace files locally can be found on the Playwright Trace Viewer documentation.

Reducing flaky tests in Blazor Server — Network Considerations

One of the things I realized very early on when working with Blazor server and Playwright right is that as part of the component lifecycle Blazor server loads some html as part of its OnInitialized method and updates the DOM as needed when the relevant information has been loaded via the OnAfterRenderAsync method is used. What this means for you as the dev is you need to consider that your tests might be flaky because your page isn’t actually finished loading.

One easy way to solve for this problem is to ensure that your network has been idle for long enough prior to continuing tests. This can be done by changing the load options as shown below.

await page.GotoAsync("http://localhost:5078", new PageGotoOptions(){WaitUntil = WaitUntilState.NetworkIdle});

Reducing flaky tests in Blazor Server — Asserting Page Readiness

Beyond checking if the network is done loading, what you can also do to prevent flaky tests is assert that specific elements are on the page (and wait a predetermined amount of time for it to become visible). This can be achieved using the Expect functionality which is baked into the PageTest class that Playwright provides. I’m attaching an edited example that Playwright provides along with their template.

using Microsoft.Playwright;

namespace UnitTests;

/// <summary>
/// Leaving this in here as this is what comes as part of the playwright template
/// </summary>

[Parallelizable(ParallelScope.Self)]
[TestFixture]
public class Assertions : PageTest
{
    [Test]
    public async Task HomepageHasPlaywrightInTitleAndGetStartedLinkLinkingtoTheIntroPage()
    {
        await Page.GotoAsync("https://playwright.dev");

        // Expect a title "to contain" a substring.
        await Expect(Page).ToHaveTitleAsync(new Regex("Playwright"));

        // create a locator
        var getStarted = Page.Locator("text=Get Started");

        // Expect an attribute "to be strictly equal" to the value.
        // Custom timeout of 30 seconds has been added here to show how the length of time to wait can be adjusted
        await Expect(getStarted).ToHaveAttributeAsync(
            "href",
            "/docs/intro", 
            new LocatorAssertionsToHaveAttributeOptions(){Timeout = 30_000});

        // Click the get started link.
        await getStarted.ClickAsync();

        // Expects the URL to contain intro.
        await Expect(Page).ToHaveURLAsync(new Regex(".*intro"));
    }
}

The test asserts that a element has a particular attribute, however this could be adapted to doing tests where one asserts that

• a particular locator is visible
• a specific number of elements are present
• etc.

Final thoughts

There’s quite a bit of information to chew on here — like with most things I think experience in actually going and doing a test or two is your best teacher.

It’s worth noting here that I’ve tested playwright on both Blazor server and WASM, but I have no real experience with the new hybrid model released at the end of last year. I’m assuming that most testing will be identical, some quirks may pop up and I will add them in here at a later date when I have more experience using that model.

My hope is that you will be able to use the tools mentioned above to speed up your dev process and get your entire team on board with test.

Good luck and happy testing!