The run this page was built from
- 24checks of the application
- 24passed
- 0failed
- 5in the smoke set
They divide into 14 against the REST API, 8 through a browser and 2 end-to-end. The end-to-end pair is one order placed through the interface from sign-in to payment, and one check that puts the same question to both doors at once.
The purchase, as it runs
The end-to-end order, recorded by the browser during the run above: a sign in with the account the suite registered through the API a moment earlier, a product opened and added to the cart, the delivery address the account was registered with, and the order placed. No narration and no cuts. The account is deleted when the test ends.
The same purchase, one action at a time
The run also recorded a Playwright trace of that order, and a trace is more than a video: it can be stepped through action by action, with the page's DOM as it was at each step, the network the page made, the timings, and the test's own source beside the line that ran. Watching it answers "did it work"; stepping through it answers "what exactly did it do".
Step through the purchase Download the trace
The first link opens the file in Playwright's own trace viewer, which is a
static page that reads the trace straight from this site. The second is
the same file to keep: playwright show-trace checkout-trace.zip
opens it locally, with no upload anywhere.
How it is put together
How the coverage was chosen
The suite starts from what costs a marketplace money when it breaks rather than from what is easy to automate. Four failures head that list: an order that cannot be completed, an account that cannot be created or cannot be deleted, a catalogue or a search that answers with nothing, and a cart that loses a line between the product page and the order. Each is covered where it shows first — the account lifecycle and the catalogue against the API, the cart and the search through the browser, the order end to end. The smoke marker then goes on the shortest set that would catch a marketplace that is broken rather than wrong in a detail: the catalogue answering, an account created and deleted again, a sign-in through the interface, the home page rendering, and the order placed front to back. 5 of the results in this run carry it.
A check then belongs at the lowest layer that can still prove the thing that matters. Product listings, search results, account creation and deletion, and the answer each gives to input it should reject are data rules, so they are asserted against the REST API, where a failure names its own cause and needs no browser: 14 cases. Only behaviour a person can actually see is driven through a browser — the home page and a product's name and price as they render, a search made from the page, a product reaching the cart with the quantity that was asked for, and a sign-in that has to be refused: the 8 browser cases. The 2 end-to-end cases exist because the purchase is the one flow whose whole value is that the separate parts hold together, and because one of them is the only place where the interface and the API are made to answer the same question.
What this suite deliberately does not cover
- The arithmetic. The end-to-end order asserts that it completes and that the delivery address is the one the account was registered with; nothing asserts that the total equals the sum of the lines. Search is checked the same way — that it answers with products that carry a name and a price, not that each result matches the term that was typed.
- Payment beyond the form. The end-to-end order stops at submitting the payment form. There is no processor behind this application, so going further would assert a fixture rather than the product.
- Performance. Nothing here measures response time or load. The timeouts in the suite exist to fail a stuck test, not to hold a budget, and a green functional run says nothing about either.
- Accessibility. It needs its own tooling and its own time budget. Implying that a functional suite covers it would be the dishonest part.
- Cross-browser rendering. The engine is a setting, so the suite can run on Firefox or WebKit, but no assertion anywhere is about how a page renders in one engine versus another.
Four decisions that keep it maintainable
API responses are parsed into typed models, not read as dictionaries
The catalogue and the search results go through ProductsResponse
and Product in src/ecom_taf/models/ before a test
asserts anything about them. A field that is renamed, dropped or returned
with the wrong type fails there, naming the field, instead of surfacing
three assertions later as a KeyError — or as a comparison
against None that happens to pass. A registration travels the
same way in the other direction: UserAccount is the shape of
an account, and the form the API receives is built from it rather than
assembled at the call site.
One fact is checked through two doors
tests/e2e/test_api_ui_product_consistency.py reads a product
from the REST API, opens the same product in the browser and compares the
name and the price. Both layers can be green while the two disagree; that
disagreement is what a customer sees, and neither suite on its own can
catch it.
The suite is a polite guest on a site it does not own
The registered_user fixture in tests/conftest.py
creates a disposable account through the API, hands it to the test and
deletes it afterwards, so a run leaves the application as it found it.
That is also why the hook that captures a failure screenshot is written
never to raise: a crashed session skips teardown, and skipped teardown
means an account left behind on somebody else's site. Against the stand
the same fixture runs unchanged, and the account it creates lives in the
stand's memory for exactly one test.
Per-run data comes from a factory, not from constants
src/ecom_taf/data/user_factory.py builds a fresh account for
every test that needs one, address and all. A suite pinned to a constant
email passes once and then fails on "already registered", and the usual
repair — deleting the record by hand between runs — is a step that cannot
exist inside a pipeline.
The full report
Every case, its steps, its duration and the trend across runs. A failed browser test carries a full-page screenshot and the page's HTML as it was at the moment of the failure, and every run records what it was tested against: the site, the API, the Python that ran it, and — for the leg that drove one — the browser and whether it ran headless.
Failures are sorted by the text of the failure into four groups — a
product defect, a timeout, a selector that no longer matches, and a site
or infrastructure problem — because that is what decides who picks the
failure up. The sorting reads the stack trace, so it is a first guess and
not a diagnosis: a check that fails on a plain assertion is filed as a
product defect whatever the real cause turns out to be. What it does
reliably separate is the pair that matters most for a suite driving a
running application: the application answering wrongly, and the
application not answering at all. Here it has nothing to sort, and by
design never will: this page is only ever published by a run in which
every check passed, so a failing run stops at the pipeline, where the
rule that stops it (.github/workflows/ci.yml), the empty
grouping it leaves behind (report/widgets/categories.json)
and the runs that did fail are all in the repository.