What this run collected
- 1140records collected
- 64pages fetched
- 58requests made
- 18changes detected
| Source | Door | Records | Pages | Requests | Time | Changes |
|---|---|---|---|---|---|---|
| books.toscrape.com | http | 1000 | 50 | 51 | 5.2 s | 0 |
| quotes.toscrape.com | browser | 100 | 10 | 1 | 3.0 s | 0 |
| the demo store | local | 40 | 4 | 6 | 0.01 s | 18 |
That is 8.2 seconds of fetching in all. The requests are counted by the HTTP client itself, which is also what keeps at least half a second between two requests to the same host, so the figure is the traffic this project actually caused — each source's own share of it, not the run's total repeated on every row. A browser page load is not one of them: the source that goes through the browser door counts its rendered pages and times them itself, because the HTTP client never sees them.
The data itself
Not a screenshot of a table: the files this run wrote, exported from the database straight after the scrape.
- books.csv — every book collected in this run, one row per book (186 KB).
- books.json — the same books as JSON (309 KB).
- quotes.json — every quote collected in this run (32 KB).
- demo.json — the demo store's catalogue as it stood during this run (8 KB).
- scrapewatch.xlsx — the same records as one Excel workbook: a sheet per source with the values that changed since the previous snapshot highlighted, and a sheet listing every change (142 KB).
- scrapewatch.sqlite3 — the database itself: every snapshot this project has taken, which is what this run's diff was made against and what the next run will compare with (11.5 MB).
The database in that list is published on purpose. It is the same SQLite file
the next run pulls back before it starts, which is the only reason a nightly
diff has a yesterday to compare against; and it means anyone can open the
history with sqlite3 instead of taking this page's word for it.
It keeps the last 30 nights per source and drops what is older —
a published file that grows for ever is a download nobody makes.
What moved since the previous run
18 changes in all. Every one of them is a field that differs between the last two snapshots of the same record, matched by the identifier the site itself gives it, not by position in a list:
- books.toscrape.com: nothing moved since the previous snapshot.
- quotes.toscrape.com: nothing moved since the previous snapshot.
- the demo store — the catalogue in this repository, whose prices and stock rotate daily by a seeded rule — accounts for 18 of them: 0 added, 0 removed, 18 changed.
A source seen for the first time reports every record as added, because there is no previous snapshot to compare it against — the diff says so instead of pretending nothing changed.
A browser walking an infinite scroll
The demo store's /scroll page loads twelve products at a time as
you reach the bottom of it, and announces its own end by setting
data-done="true" on the container. The scraper scrolls until that
appears, or until the item count has stayed flat for three rounds — never a
fixed number of scrolls, because a page that stalls without saying so must not
spin forever. This is that, recorded during the run above.
The same scroll, one action at a time
The run also recorded a Playwright trace of it, and a trace is more than a video: it can be stepped through action by action, with the page's DOM as it was at each step, the network the page made, the timings, and the source beside the line that ran. Watching it answers "did it work"; stepping through it answers "what exactly did it do".
Step through the scroll Download the trace
The first link opens the file in Playwright's own trace viewer, which is a
static page that reads the trace straight from this site. The second is the
same file to keep: playwright show-trace scroll-trace.zip opens it
locally, with no upload anywhere.
How it is put together
Which door each source goes through, and why
A browser is the expensive answer, so it is used where it is the only answer and nowhere else. The choice is per source and it is defended by a test, not by a preference.
books.toscrape.com — plain HTTP
The catalogue is server-rendered, so src/scrapewatch/sources/books.py
walks it with the HTTP client alone: no browser starts, and a thousand books
cost fifty requests instead of fifty page loads. Choosing not to start a
browser is the engineering decision here, and the parsers are proven against
pages saved from the real site under tests/fixtures/books/, so the
gate can check them with no network and no flakiness.
quotes.toscrape.com — a browser, because nothing else can see the page
The site's /js/ page ships its quotes as a JSON array inside a
<script> block and renders them with client-side JavaScript,
so anything that only speaks HTTP sees an empty page.
tests/parsers/test_quotes_parser.py is the proof rather than the
claim: the same parser finds nothing in the raw fetched page and every quote in
what the browser rendered.
The demo store — a target that changes on purpose
Change detection cannot be demonstrated against a catalogue that never moves,
and it must not be demonstrated by editing the database by hand. The store in
src/scrapewatch/demo_store/ derives every price and stock level
from the date with a seeded rule, so two runs on two days genuinely differ and
the same day always reproduces. It also carries the infinite-scroll page the
browser engine is exercised against, and a login wall, so a session-only view
has somewhere to exist.
What it deliberately does not do
Out of scope, by decision
- Anti-bot evasion. No fingerprint spoofing, no stealth plugins, no
captcha solving. The scraper identifies itself by name in its
User-Agentand links back to the repository. - Proxy rotation. One origin, one honest address. Rotating addresses to get around a rate limit is the opposite of the politeness the client is built around.
- Speed. Nothing here is tuned for throughput. There is a minimum interval between two requests to the same host and a bounded retry budget, and both are floors on politeness rather than ceilings on performance.
- Sites beyond the three named. Two public practice sites that exist to be scraped, and a store that ships in this repository. Pointing this at anything else is a decision for whoever does it.
- A scheduler inside the application. The night's run is a cron entry in the pipeline, not a daemon in the code. A tool that scrapes when it is asked is much easier to reason about than one that scrapes on its own.
Four decisions that keep it honest
Politeness lives in the client, not in each source
src/scrapewatch/http.py reads robots.txt before it
fetches anything from an origin, keeps a minimum interval between requests to
that host, retries on its own budget, and counts every request, retry and byte.
A source that forgot to be polite is not possible, because a source does not
make requests — the client does. A site whose robots.txt cannot be
read at all is treated as disallowed: a scraper that cannot read the rules does
not get to assume there are none.
One bad record costs itself, not the night
src/scrapewatch/pipeline/run.py normalises record by record and
isolates the failures, so one malformed listing out of a thousand is counted as
a parse error and named in the run statistics while the other 999 — some of
which may carry the very price change this tool exists to find — are still
snapshotted. A source is only "skipped" when it produced nothing usable at all,
and that is a different sentence on this page.
Snapshots are kept, not overwritten; the diff compares the last two
src/scrapewatch/storage.py never writes over yesterday. Records are
matched across snapshots by the identifier the site gives them — a slug, a
product id — so a catalogue that reorders itself is not reported as a thousand
changes, and a price that moved is reported as the one field that moved, with
its before and after. The history is bounded on purpose: the last
30 snapshots of every source are kept and the rest are pruned after
each run, because this database is published and an unbounded one eventually
cannot be.
The gate never touches the network
Parsing is proven against pages saved from the real sites and dated, in
tests/fixtures/. That is what makes a red live check meaningful:
the fixtures still parse and the live page no longer does, which is markup
drift and not an outage. The night's run reaches the sites; a push never does.
The tests, and the report from this run
323 checks ran in the same pipeline that published this page: 269 of pure logic, 7 of parsing saved pages, 32 of the pipeline on a real SQLite file, 12 through a browser against the demo store, 3 against the real practice sites. They are counted here rather than at the top because this project's subject is the data above; the tests are how it is kept true. 323 of them passed.
All 3 checks against the real practice sites passed in this run, which is the statement that the markup those parsers were written for is still the markup the sites serve.
Failures in the report are sorted by the text of the failure into four groups — a product or site defect, a timeout, markup drift, and an infrastructure problem — because that is what decides who picks the failure up. The sorting reads the message, so it is a first guess and not a diagnosis; what it does separate reliably is the pair that matters most for a scraper pointed at somebody else's site: the page answering differently, and the page not answering at all.