This CL configures custom host resolver rules for the Chromium binary
that puppeteer is using to run e2e test.
With host resolver rules, all resources can now be loaded via
arbitrary '.test' domains while still being served from localhost.
E.g. an e2e test might navigate to 'devtools.test/foo.html' while
also including an iframe with Url 'bar.test/iframe.html', with both
files being served from localhost.
This unlocks testing for a whole class of features that was
previoulsy impossible to test.
This CL also adds a new helper 'goToResourceWithCustomHost', that
supplements the existing 'goToResource' with a modifiable host part.
R=aerotwist@chromium.org, sigurds@chromium.org
Bug: chromium:1177676
Change-Id: I1f51aa8419619e11352989457b24cc56cfc78e52
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2692148
Commit-Queue: Simon Zünd <szuend@chromium.org>
Reviewed-by: Sigurd Schneider <sigurds@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
When tests fail, debugging the failure includes isolating the test
before opening it in debug mode or adding diagnostics to it. With the
current test output, opening the file is surprisingly tedious, since
there is no direct way to map test log output to the tests on the
filesystem. One has to grep the test directory for test names first.
This CL wraps Mocha's describe function to automatically include the
test filename into the test title in a format that can be copy-pasted
into the test runner's --test-file argument.
Bug: none
Change-Id: I172bd14f5026ec0041b933c64bcdcccf58536fb9
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2681741
Commit-Queue: Philip Pfaffe <pfaffe@chromium.org>
Reviewed-by: Peter Marshall <petermarshall@chromium.org>
Auto-Submit: Philip Pfaffe <pfaffe@chromium.org>
This CL adds the logic to trigger contrast checks on DevTools
startup or on page load events. The check is only triggered if
the corresponding experiment (contrastIssues) is enabled.
Deduplication of the reported issues happens in the IssueManager
on the frontend since the same issue might be reported twice
if there are multiple documents on the target page.
Bug: chromium:1155460
Change-Id: Ib0d5115c527ceb9da9d2ed2dfa48cfdca6f82f60
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2573360
Commit-Queue: Alex Rudenko <alexrudenko@chromium.org>
Reviewed-by: Simon Zünd <szuend@chromium.org>
Reviewed-by: Sigurd Schneider <sigurds@chromium.org>
This CL introduces the infrastructure and first test for what we're calling
"Interaction tests". These are tests that can be used to test component
behaviour that cannot be tested via Karma unit tests but don't require an e2e
test either.
The best example here is complex user interactions: clicking and dragging a
mouse, or doing multiple tabs to navigate through the UI.
Note that the one interaction test added here for theme_colors isn't the best
example of a complex test, but it's a test that's good enough for now whilst we
get everything set up on infrastructure and make sure it runs without flaking.
Bug: 1153281
Change-Id: Ifb37639a557c459974f703b8954ae68a93d5d368
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2566808
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
When debug evaluation hits exceptions in wasm debugging, the
LanguagePluginManager incorrectly propagated those errors, causing weird
warnings in the wrong places or warnings that are missing altogether.
This CL hooks up the plugin manager with the existing error reporting
mechanisms and includes a test to make sure the errors show up.
Drive-by: In the watch window, when evaluation failed, we previously
swallowed the actual exception. That's probably because most of the time
that exception is a reference error because variables aren't valid in
the current scope. This CL adds a tooltip that shows the exception.
Bug: chromium:1137514, chromium:1153315
Change-Id: Ia1ec81ce95598adf7abab8c4e31c3310e3669fb7
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2565376
Commit-Queue: Philip Pfaffe <pfaffe@chromium.org>
Commit-Queue: Benedikt Meurer <bmeurer@chromium.org>
Auto-Submit: Philip Pfaffe <pfaffe@chromium.org>
Reviewed-by: Benedikt Meurer <bmeurer@chromium.org>
console.log() output is not buffered during parallel testing so it's
hard to associate the step() logging output with the test that
actually caused it. If we want to log stuff like this, we should find
a way to redirect it to the test reporter properly so that it gets
buffered per-test-file and printed all at once, like the describe/
it output is.
The step output probably isn't needed anymore as we have the AsyncScope
output which gives the source-mapped file name + line number in the
test output if there is an await failure, which is even more useful
than the step output.
If people find cases where we do really need this we can add it back,
the step function still exists for now.
Bug: 1101784
Change-Id: Ia90b6532ce348b189ee7c5f2e3ed482f074a6561
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2534207
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Almothana Athamneh <almuthanna@chromium.org>
Commit-Queue: Peter Marshall <petermarshall@chromium.org>
`waitForFunction` previously always waited 100 ms before checking if
the given function is truthy. We can improve the total e2e test suite
running time by 40-50 s by eliminating this timeout. This,
unfortunately, makes some of the tests flake since they relied on this
timeout. This CL attempts to eliminate the timeout and patch the tests
that relied on it. In the cases where a test could not be patched the
timeout is inserted in the test explicitly.
Local experiments showed that this CL reduces the total runtime from
on avg. ~309 seconds to an avg. of ~264 seconds.
This amounts to a ~45 second reduction or ~15%.
This is a reland of a26e3406c7
which was a reland of 3be2087f9d
Bug: 1112692
Change-Id: I3a5156d35553415bfbac95c0251b849ed77f4767
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2354096
Commit-Queue: Johan Bay <jobay@google.com>
Reviewed-by: Mathias Bynens <mathias@chromium.org>
it.repeat() is helpful for fixing flaky tests.
Along with repeat, we also have the async tracing which is very helpful
for debugging tests that timeout due to hanging async operations. We
want to make all of these helpers available by default.
This CL introduces a new file mocha-extensions and adds both of these
customizations to the default mocha operations and then re-exports them.
In the future, all tests will import from mocha-extensions instead to
get these extensions by default. We will add lint rules to ensure that
callers don't import from mocha directly and miss out on these.
Bug: 1101782
Change-Id: I15dbb71493e60064288bfc050ac3f818a5c6821f
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2339319
Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
Commit-Queue: Peter Marshall <petermarshall@chromium.org>
`waitForFunction` previously always waited 100 ms before checking if
the given function is truthy. We can improve the total e2e test suite
running time by 40-50 s by eliminating this timeout. This,
unfortunately, makes some of the tests flake since they relied on this
timeout. This CL attempts to eliminate the timeout and patch the tests
that relied on it. In the cases where a test could not be patched the
timeout is inserted in the test explicitly.
Local experiments showed that this CL reduces the total runtime from
on avg. 5:05 minutes to an avg. of 4:19 minutes.
This amounts to a ~46 s reduction or ~15%.
This is a reland of 3be2087f9d
Bug: 1112692
Change-Id: Id92542de85920f95bb922d1b62ddfb63b83a055a
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2346366
Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Changhao Han <changhaohan@chromium.org>
Reviewed-by: Mathias Bynens <mathias@chromium.org>
Commit-Queue: Johan Bay <jobay@google.com>
This CL also changes tests to favor `waitFor` over `$` per default:
In all tests where an element is expected `$` is changed to `waitFor`.
This should be uproblematic in all the cases where an element is
expected to exist immediately, but gives a bit of leeway in case the
element is slow to emerge. Typing wise, `waitFor` never returns null
since it throws if no element is found before it times out.
Change-Id: I00b722eff34df44402d9f3bf6786432c9faf1505
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2327710
Commit-Queue: Mathias Bynens <mathias@chromium.org>
Reviewed-by: Changhao Han <changhaohan@chromium.org>
Reviewed-by: Mathias Bynens <mathias@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
AsyncScope objects allow tracking of the current stack trace during
chains of calls to test helpers. Recording stack traces allows us to
report pending helpers when a mocha timeout occurs.
The CL also introduces an AsyncTrace help that tests use to opt into
stack tracing. Wrapping the mocha it() call is necessary to hook into the
processing of the timeout. I export the AsyncTrace helper for clarity,
but we can also wrap mocha's it() directly into our own it() wrapper to
provide a drop-in replacement. AsyncTrace's it() allows configuring the
mocha timeout locally per test. We can chose to not do that and configure
it globally, or alternatively define standard timeouts for 'fast',
'regular', or 'slow' tests.
Change-Id: I9542cdfdd78a1868a9f0fa56b8d112bd549e6f34
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2328845
Reviewed-by: Peter Marshall <petermarshall@chromium.org>
Commit-Queue: Philip Pfaffe <pfaffe@chromium.org>
If we use a local timeout rather than letting this timeout the whole
mocha process, we get a way better error message which includes
info about the selector/function/whatever we care about, and the
step name. Without this we just get the test file name and nothing
else.
Change-Id: Ia2882fb123a6f2b582c0dd19f3f01135b496161a
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2292276
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Commit-Queue: Peter Marshall <petermarshall@chromium.org>
(...in more than one place).
Lots of tests relied on the hosted mode server using port 8090 which
is not going to be the case once we have the parallel mocha runner
working.
Add the port number to puppeteer-state and have tests use it by
going through test/shared/helper.ts.
The port number is set at runtime rather than being a module-time
constant, so some users of the port needed to be changed to get it
after the root hooks ran instead. Future users should follow this
pattern and get the port in a before() block or during the test
itself. Or just use helpers that treat the port properly already
e.g. goToResource.
We don't send this port number to the hosted mode server yet,
so we still have this hardcoded in two places. A fix is coming in
a future CL.
Bug: 1101784
Change-Id: Ib0a7aa7a25db58d06624a1cdee6fa630cdfbe362
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2284828
Commit-Queue: Peter Marshall <petermarshall@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Changes include:
- Test server handles URLs with escaping better.
- Logging from DevTools frontend now logged by the test.
- Helper to turn on CDP logging - for debugging only.
- click helper improved to allow specifying maximum distance from
left, for when an element extends far beyond its containing
element.
- New pressKey helper that makes keyboard shortcuts easy.
- Helpers to modify and save a source pane and to get the text of
the line of code where we are stopped.
Bug: 1094436
Change-Id: I389eaa680bb0771a45104f470647f67b4fa5d1d9
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2282811
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Commit-Queue: Eric Leese <leese@chromium.org>
This introduces a new goToResource shared helper for all e2e puppeteer
tests to use.
It helps simplify a bunch of tests and other helpers in that they don't
need to replicate the `${resourcesPath}/some/file.html` part.
Because this helper also knows how to get hold of the target on its own
it also further simplifies other tests that don't need to import it
anymore.
The next step would be to create a shared helper that knows how to
navigate to panels inside the DevTools front-end as this is also
duplicated across tests.
Bug: 1091226
Change-Id: I717455a51dec30e153d796ef101eab3d42bc2da1
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2230320
Commit-Queue: Patrick Brosset <patrick.brosset@microsoft.com>
Reviewed-by: Jan Scheffler <janscheffler@chromium.org>
Reviewed-by: Jose Leal <joselea@microsoft.com>
Sigurd Schneider