This CL introduces the infrastructure and first test for what we're calling
"Interaction tests". These are tests that can be used to test component
behaviour that cannot be tested via Karma unit tests but don't require an e2e
test either.
The best example here is complex user interactions: clicking and dragging a
mouse, or doing multiple tabs to navigate through the UI.
Note that the one interaction test added here for theme_colors isn't the best
example of a complex test, but it's a test that's good enough for now whilst we
get everything set up on infrastructure and make sure it runs without flaking.
Bug: 1153281
Change-Id: Ifb37639a557c459974f703b8954ae68a93d5d368
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2566808
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
As part of the work to enable Puppeteer component tests, we need to
configure the test suite to run either the hosted mode server or the
component docs server. This CL updates it to take a flag, and makes some
updates to the component docs server, which now has to run either
directly or in the out/Default/gen directory depending on how it is run.
I suspect I'll make a follow up CL to always run the component server in
out/Default/gen, but for now enabling it to detect its context is the
quickest way to unblock running it in tests. The next CL will add a
component test suite that can run a basic test against the component doc
server, but I have manually verified locally that I can run tests
against that server.
Bug: 1153281
Change-Id: I55bda4edd0a983d03bfecff0446b0f3e3a008b52
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2562707
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
With the prompting in the global setup fixture, we started the hosted
mode server and then paused with the user prompt. This is not how it
worked before, where we launched chrome before we paused.
Set it back to the previous behavior by moving the prompt code inside
the beforeAll hook. Debug mode is mututally exclusive with parallel
mode so the moved code will only run in serial mode, not once per
parallel process.
Bug: 1101784
Change-Id: I74c9a70652056ecf368e439ee747409742ff4ce9
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2537969
Auto-Submit: Peter Marshall <petermarshall@chromium.org>
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Only set the unhandled rejection handler once per process.
We run the setup and teardown multiple times per-process now.
Add clearPuppeteerState() to unset the state between runs on
a parallel test runner. This is because the port changes
between successive runs in the same process.
Remove the hasShutdown logic from mocha_hooks. Also remove the
beforeExit hook which was causing this to get called at least
twice for every process. It already gets called in the ordinary
shutdown case via the afterAll() hook. We only need to call it
in the extraordinary case which is the SIGINT case. Now that we
do the shutdown multiple times per process, we don't care if we
already shutdown on this process or not. The only issue could be
if ctrl+c is sent during the shutdown, but then we are crashing
anyway.
Right now the default # of jobs is 1, so out bots will still run
in serial mode. We want to add parallel mode as an options for
local development while we iron out any last problems with this
approach.
You can test this locally with npm run e2etests -- --jobs=8.
We make use of global setup fixtures to only run one hosted
mode server and share it between all parallel runners. Each
runner still starts its own chrome and restarts it between
files, which is a bit inefficient but still a big improvement
compared to serial mode.
Split the hosted mode server out into its own file inside
conductor as it can be dealt with entirely separately, and
its state is not per-process like the chrome state, but only
per-main-process which launches the test runner sub-processes.
Bug: 1101784
Change-Id: Ic0c9be3559708fd5403a88882b9fb93257633de9
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2288694
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Commit-Queue: Peter Marshall <petermarshall@chromium.org>
Previously we hardcoded the port to 9222. In parallel mode we need each
chrome to have a different debug port.
We get the port from the browser websocket endpoint URL which
puppeteer helpfully exposes.
Factor out launching chrome and getting the port into functional
helpers.
Bug: 1101784
Change-Id: I18707923cee3e8f68ac81e0a2d56e53d3d946e69
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2288690
Commit-Queue: Peter Marshall <petermarshall@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Rather than hardcode the port to use, use port 0 which
requests any free port from the OS. We report the port
we end up using to the caller over IPC, if an IPC
channel was opened by the caller.
We wait for the port to get reported before starting
the chrome instance.
Previously we started the hosted mode server and then
started chrome without waiting for it to start. This is
probably fine in most situations because chrome takes a
while to start and we need to do extra setup there, so
we have a long time before the server needs to respond.
For parallel test mode, we need to make sure we start
the server on a free port. This also makes the mocha
runner more robust to failures e.g. if we accidentally
leave the server running and the port is taken.
This doesn't slow down the startup time of the e2e
test runner at all, according to timing of local runs.
Bug: 1101784
Change-Id: Ibb14d5a177ba5f9e257743824770bff3d5ccb033
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2285124
Commit-Queue: Peter Marshall <petermarshall@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Right now we hardcode the ports and assume that the hosted mode server
hardcodes the same ones. In the future we will have different port
numbers for different parallel test runners, so we need to pass the
correct numbers here.
env.PORT is the hosted mode server port, so it's essential we get that
right. env.REMOTE_DEBUGGING_PORT is only used in a link on the
landing page of the hosted mode server, so it being correct is not
essential for the tests to run properly but there's no reason for
it to be wrong.
Also rename envPort to chromeDebugPort to make things clearer with all
the different port numbers flying around.
Bug: 1101784
Change-Id: I58880a3543021bbcd484cf3578ff8225d3e3d81c
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2284839
Commit-Queue: Peter Marshall <petermarshall@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
(...in more than one place).
Lots of tests relied on the hosted mode server using port 8090 which
is not going to be the case once we have the parallel mocha runner
working.
Add the port number to puppeteer-state and have tests use it by
going through test/shared/helper.ts.
The port number is set at runtime rather than being a module-time
constant, so some users of the port needed to be changed to get it
after the root hooks ran instead. Future users should follow this
pattern and get the port in a before() block or during the test
itself. Or just use helpers that treat the port properly already
e.g. goToResource.
We don't send this port number to the hosted mode server yet,
so we still have this hardcoded in two places. A fix is coming in
a future CL.
Bug: 1101784
Change-Id: Ib0a7aa7a25db58d06624a1cdee6fa630cdfbe362
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2284828
Commit-Queue: Peter Marshall <petermarshall@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Changes include:
- Test server handles URLs with escaping better.
- Logging from DevTools frontend now logged by the test.
- Helper to turn on CDP logging - for debugging only.
- click helper improved to allow specifying maximum distance from
left, for when an element extends far beyond its containing
element.
- New pressKey helper that makes keyboard shortcuts easy.
- Helpers to modify and save a source pane and to get the text of
the line of code where we are stopped.
Bug: 1094436
Change-Id: I389eaa680bb0771a45104f470647f67b4fa5d1d9
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2282811
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Commit-Queue: Eric Leese <leese@chromium.org>
Mocha 8 introduces 'root hook plugins' which is an alternative way of
specifying root hooks, which is required for parallel mode.
This CL uses the new root hook plugins following the migration guide:
https://mochajs.org/#migrating-tests-to-use-root-hook-plugins
This CL does not make us ready for parallel mode yet, because we still
have a few assumptions that we have one instance running. These will be
dealt with in future CLs. This CL just ports the root hooks over to the
plugin system while maintaining the same behavior.
Bug: 1101784
Change-Id: I74dfc7f24decd031ebac656c7cea8ef87f9ac791
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2282346
Commit-Queue: Peter Marshall <petermarshall@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
The e2e-tests now use the build output in resources/inspector in
the out directory. This allows us to introduce TypeScript-authored
files in the source directory `front_end`, which will get compiled
into resourc/inspector.
After to making a change to a e2e-test or the front_end, you need
to rebuild Devtools, after which you can run `npm run e2etest` as
normal.
Since we now use the build output, this also means that you can
run the e2e-tests on the release build. In other words, if you
build DevTools with build optimizations (such as Rollup), the
e2e-tests will now use the output (and thus provide test coverage).
DISABLE_THIRD_PARTY_CHECK=Node fixes
R=aerotwist@chromium.org,jacktfranklin@chromium.org
Fixed: 1088463
Change-Id: I02ec3c2476bc3647158fede9e1d347963b3a720a
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2224809
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Jose Leal <joselea@microsoft.com>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
Currently the query param "panel" only selects a tab if it is already open. With this change a closed tab can be opened and selected.
Gif: https://imgur.com/a/Sc8H04J
To try the Layers panel example:
Using master:
1) Start debugging a website using `npm start` and make sure Layers is closed.
2) append `&panel=layers' to the url and reload the page,
3) you should see that the panel is not opened
Apply the changes from this CL:
3) Repeat steps 1 & 2
4) Layers panel should appear.
Bug: 1094648
Change-Id: Id8f69ebc91058c89dd07488a2480fea12ceb8f79
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2244849
Commit-Queue: Jose Leal <joselea@microsoft.com>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Brandon Goddard <brgoddar@microsoft.com>
Add e2e test for Device Toolbar's media query inspector, as a follow-up
to an earlier issue that media query inspector did not show media query
rules from style tag attributes.
When adding this e2e test for Device Toolbar, it's required that the
e2e test driver can reload DevTools frontend page with "can_dock=true"
parameter in its URL. This CL also adds "canDock" as an option when
reloading DevTools frontend in e2e testing.
Bug: chromium:1069467, chromium:1086159
Change-Id: Ic8685fc231fa7b01554e05a1121b3c2394fb7131
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2215857
Commit-Queue: Changhao Han <changhaohan@chromium.org>
Auto-Submit: Changhao Han <changhaohan@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Mathias Bynens <mathias@chromium.org>
If developers would kill the Mocha process before completion, it would
not run the `after` hook. This meant that its subprocesses (most notably
the Puppeteer process) would not properly cleanup.
To mitigate that, listen for the process exit and `SIGINT` signal (e.g.
a developer doing `ctrl + c`) and cleanup. However, if the `after` hook
ran succesfully, we shouldn't be cleaning up twice.
R=aerotwist@chromium.org
Bug: 1071369
Change-Id: I072b2b662309af1c508a7877c4b897cf915bed71
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2157039
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
The shared test runner logic was reimplementing parts of Mocha,
in particular the test logging and filtering. Moreover, it was
booting the hosted mode server and puppeteer outside of Mocha.
Mocha supports root level hooks [1]. These hooks allow us to
perform work before a test starts. Moreover, by using the
before and after hook, we can run logic before and after
all tests.
By using these hooks, we can extract the "boot the hosted mode
server and hookup puppeteer" part to these root hooks.
Additionally, we can reset the pages in the `beforeEach`, which
means that tests themselves don't have to reset the pages.
We also put the implementation code into third_party/conductor,
as we would like to reuse this logic for the Puppeteer tests.
The Puppeteer test suite now also uses Mocha and has very
similar requirements as to our DevTools tests. By extracting
from DevTools, we can look into expanding the test runner to
other usecases, but that is out of scope for now.
[1]: https://mochajs.org/#root-level-hooks
Bug: 1071369
Change-Id: Ie9f954359d9de84da564b74b6f5517dd535db008
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2150458
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Philip Pfaffe