Explicit browser waits followed by HTML parsing

The browser is open, the URL is correct, and the product is visible to a person, yet automation still returns an empty title. The problem is often the readiness condition rather than the speed of Selenium. A loaded document does not guarantee that JavaScript has finished populating the card. Reviewing browser automation and HTML parsing in my archive makes the boundary between those two jobs particularly useful.

Wait for the data the operation requires

Suppose a local test page inserts a .product card after two seconds. Sleeping for three seconds may pass on one machine and fail under a heavier workload. It also wastes time when the card appears immediately. An explicit wait repeatedly evaluates a condition up to a maximum duration. That duration is a failure boundary, not a delay that every successful request must pay.

The condition here is a visible card. If the actual operation requires a title or price, waiting for an empty card shell is too weak; the relevant field should reach a usable state. The following integration helper uses Selenium.WebDriver, Selenium.Support and HtmlAgilityPack with a driver supplied by its caller.

using OpenQA.Selenium;
using OpenQA.Selenium.Support.UI;
using HtmlAgilityPack;

static string ReadTitle(IWebDriver driver, Uri localPage)
{
    driver.Navigate().GoToUrl(localPage);
    var wait = new WebDriverWait(driver, TimeSpan.FromSeconds(5));
    var card = wait.Until(browser =>
        browser.FindElements(By.CssSelector(".product"))
            .FirstOrDefault(element => element.Displayed));
    var html = new HtmlDocument();
    html.LoadHtml(card.GetAttribute("outerHTML"));
    return HtmlEntity.DeEntitize(html.DocumentNode
        .SelectSingleNode("//h2")?.InnerText ?? "").Trim();
}

Move from a live DOM to a fixed input

FindElements can return an empty collection, so the initial absence of a card does not have to be treated as an exceptional event. Once a visible element is found, outerHTML captures its markup. HtmlAgilityPack then works on that string, not on the live browser. That gives the parser a repeatable input and makes it possible to test title extraction without launching a browser for every case.

The //h2 query is acceptable here because the parsed document contains only one captured card. If several cards are moved into the same document, a document-wide query can return the wrong title; the query should be scoped to the relevant node. The current helper returns an empty string when no heading exists. An import that requires a title should reject that result instead of persisting an apparently successful but incomplete record.

Decoding does one specific job

Markup containing Kitap & Defter may leave the entity representation in InnerText. DeEntitize decodes that layer, and Trim removes surrounding whitespace. Neither operation fills missing data, infers hidden business meaning or repairs an incorrect selector. Repeatedly decoding a value without understanding its encoding history is also a poor replacement for a clear text contract.

The element can become stale after the wait

A successful lookup does not freeze the DOM. A framework may replace the node between locating the card and reading its HTML, leaving a stale element reference. For a frequently rerendered page I would consider locating the element and extracting the required markup within one retry condition. Only expected transient failures should be retried. Catching every exception and returning an empty string hides a broken selector as an ordinary absence of data.

Combining large implicit waits with explicit waits can make timeout behavior difficult to reason about. I keep one consistent waiting strategy in the automation layer. The helper does not create the WebDriver and therefore should not dispose it unexpectedly; browser lifetime belongs to the caller, which must also ensure that completed jobs leave no orphaned browser processes.

Distinguish three outcomes in the test page

First insert a valid card after a delay and expect Demo kitap. Then omit the card entirely and expect a timeout. Finally display the card without an h2: the wait should succeed while extraction returns an empty result. Those tests establish whether a failure belongs to readiness or to the data contract. A screenshot and a minimal DOM fragment can help diagnosis without dumping session data or unrelated personal information into logs.

Selenium waiting strategies; Understanding stale element references.