The CQ bots all run run_test_suite.js, and all our local `npm run X`
scripts have run run_test_suite.js for many months now. I think we are
now safe to remove the Python run_test_suite.py script, as all use cases
are covered by the new JS runner.
Bug: 1186163
Change-Id: Ie8b5fd52e0934176a7da61d61fb9624b87ed764b
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/3075181
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
This CL fixes a bug where the test runner took the `--chrome-features`
flag but Conductor never read that value when booting up a browser
instance.
This CL updates it so it will read the value and pass it into the
relevant argument when executing `puppeteer.launch`.
I also changed the Python test suite runner to not prepend
`--enable-features`; that feels like something that Conductor should
do, not the test runner.
Bug: chromium:1186163
Change-Id: I05916d39d60bb98a606570cd82e688581dd78a40
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2790874
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
This reverts commit 7742a201a1.
Reason for revert: We do have a use case for this flag that we missed, so we need it. Will follow-up with a CL to fix the fact that the test frontend infra doesn't use it currently.
Original change's description:
> [TestRunner] remove --chrome-features flag
>
> The scripts (both the old Python one and the new JS one) took
> `--chrome-features` as a flag, which was then set as
> `process.env.CHROME_FEATURES`, however searching the codebase for
> `CHROME_FEATURES` revealed that whilst we set this value, we never
> read it. Therefore this CL removes it entirely, my logic being that if
> someone needed this flag to work, we would have found this bug a long
> time ago.
>
> We can reintroduce should we find a usecase for it in the future.
>
> Bug: chromium:1186163
> Change-Id: I11d900df4aaad232fd64b9edca2aac12c94fc988
> Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2782555
> Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
> Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
> Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
Bug: chromium:1186163
Change-Id: I336a4a14a3e720f3b0626d78148038ceab9a8774
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2786942
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
The scripts (both the old Python one and the new JS one) took
`--chrome-features` as a flag, which was then set as
`process.env.CHROME_FEATURES`, however searching the codebase for
`CHROME_FEATURES` revealed that whilst we set this value, we never
read it. Therefore this CL removes it entirely, my logic being that if
someone needed this flag to work, we would have found this bug a long
time ago.
We can reintroduce should we find a usecase for it in the future.
Bug: chromium:1186163
Change-Id: I11d900df4aaad232fd64b9edca2aac12c94fc988
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2782555
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
The current test suite runner assumes the paths; you pass
--test-suite=interactions and it assumes that they live in
out/TARGET/gen/test/interactions. This CL removes that assumption by
asking people to pass --test-suite-path, which would be
out/gen/interactions.
We will remove the --test-suite flag, but for now we're keeping it
around so the bots can still run their tests. I will file a bug to ask
that the bots have their recipes updated.
Bug:1182255
Change-Id: I6d9e5d419231845047394c5f7cbd0c74ebad3a58
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2722448
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
depot_tools ships a "vpython" binary that is added to the $PATH
for all Chromium engineers. This binary is versioned by depot_tools
which we roll in ourselves as part of `gclient sync`.
By using `vpython` instead of `python`, we are no longer depended
on the Python version installed locally and instead use the version
we pull in from DEPS.
R=liviurau@chromium.org
Change-Id: If49a54c24b6cf129ff843cd9ca1140a723a78afb
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2571462
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
This CL introduces the infrastructure and first test for what we're calling
"Interaction tests". These are tests that can be used to test component
behaviour that cannot be tested via Karma unit tests but don't require an e2e
test either.
The best example here is complex user interactions: clicking and dragging a
mouse, or doing multiple tabs to navigate through the UI.
Note that the one interaction test added here for theme_colors isn't the best
example of a complex test, but it's a test that's good enough for now whilst we
get everything set up on infrastructure and make sure it runs without flaking.
Bug: 1153281
Change-Id: Ifb37639a557c459974f703b8954ae68a93d5d368
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2566808
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
As part of the work to enable Puppeteer component tests, we need to
configure the test suite to run either the hosted mode server or the
component docs server. This CL updates it to take a flag, and makes some
updates to the component docs server, which now has to run either
directly or in the out/Default/gen directory depending on how it is run.
I suspect I'll make a follow up CL to always run the component server in
out/Default/gen, but for now enabling it to detect its context is the
quickest way to unblock running it in tests. The next CL will add a
component test suite that can run a basic test against the component doc
server, but I have manually verified locally that I can run tests
against that server.
Bug: 1153281
Change-Id: I55bda4edd0a983d03bfecff0446b0f3e3a008b52
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2562707
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Only set the unhandled rejection handler once per process.
We run the setup and teardown multiple times per-process now.
Add clearPuppeteerState() to unset the state between runs on
a parallel test runner. This is because the port changes
between successive runs in the same process.
Remove the hasShutdown logic from mocha_hooks. Also remove the
beforeExit hook which was causing this to get called at least
twice for every process. It already gets called in the ordinary
shutdown case via the afterAll() hook. We only need to call it
in the extraordinary case which is the SIGINT case. Now that we
do the shutdown multiple times per process, we don't care if we
already shutdown on this process or not. The only issue could be
if ctrl+c is sent during the shutdown, but then we are crashing
anyway.
Right now the default # of jobs is 1, so out bots will still run
in serial mode. We want to add parallel mode as an options for
local development while we iron out any last problems with this
approach.
You can test this locally with npm run e2etests -- --jobs=8.
We make use of global setup fixtures to only run one hosted
mode server and share it between all parallel runners. Each
runner still starts its own chrome and restarts it between
files, which is a bit inefficient but still a big improvement
compared to serial mode.
Split the hosted mode server out into its own file inside
conductor as it can be dealt with entirely separately, and
its state is not per-process like the chrome state, but only
per-main-process which launches the test runner sub-processes.
Bug: 1101784
Change-Id: Ic0c9be3559708fd5403a88882b9fb93257633de9
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2288694
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Commit-Queue: Peter Marshall <petermarshall@chromium.org>
The e2e-tests now use the build output in resources/inspector in
the out directory. This allows us to introduce TypeScript-authored
files in the source directory `front_end`, which will get compiled
into resourc/inspector.
After to making a change to a e2e-test or the front_end, you need
to rebuild Devtools, after which you can run `npm run e2etest` as
normal.
Since we now use the build output, this also means that you can
run the e2e-tests on the release build. In other words, if you
build DevTools with build optimizations (such as Rollup), the
e2e-tests will now use the output (and thus provide test coverage).
DISABLE_THIRD_PARTY_CHECK=Node fixes
R=aerotwist@chromium.org,jacktfranklin@chromium.org
Fixed: 1088463
Change-Id: I02ec3c2476bc3647158fede9e1d347963b3a720a
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2224809
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Jose Leal <joselea@microsoft.com>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
This resurrects the perf test suite, as the older test-runner setup
was removed, but the perf test suite was not ported.
Example output:
Spawning hosted mode server
Boot performance
✓ run 1/37 (1185ms)
✓ run 2/37 (1234ms)
✓ run 3/37 (1213ms)
✓ run 4/37 (1376ms)
✓ run 5/37 (1327ms)
✓ run 6/37 (1235ms)
✓ run 7/37 (1352ms)
✓ run 8/37 (1439ms)
✓ run 9/37 (1332ms)
✓ run 10/37 (1290ms)
✓ run 11/37 (1229ms)
✓ run 12/37 (1241ms)
✓ run 13/37 (1233ms)
✓ run 14/37 (1203ms)
✓ run 15/37 (1177ms)
✓ run 16/37 (1336ms)
✓ run 17/37 (1526ms)
✓ run 18/37 (1465ms)
✓ run 19/37 (1537ms)
✓ run 20/37 (1709ms)
✓ run 21/37 (1760ms)
✓ run 22/37 (1704ms)
✓ run 23/37 (2549ms)
✓ run 24/37 (1387ms)
✓ run 25/37 (1934ms)
✓ run 26/37 (1860ms)
✓ run 27/37 (1309ms)
✓ run 28/37 (1295ms)
✓ run 29/37 (1391ms)
✓ run 30/37 (1569ms)
✓ run 31/37 (1562ms)
✓ run 32/37 (1442ms)
✓ run 33/37 (1431ms)
✓ run 34/37 (1646ms)
✓ run 35/37 (1927ms)
✓ run 36/37 (1600ms)
✓ run 37/37 (1587ms)
Mean boot time: 1475.28ms
50th percentile boot time: 1390.83ms
90th percentile boot time: 1860.02ms
99th percentile boot time: 2549.29ms
R=aerotwist@chromium.org
Change-Id: I1b6a79fdfc1b7c98f6156d3309b5b98ff8530476
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2209108
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
This has two benefits:
1. We can now "pre-process" the arguments. It therefore allows us
to specifically set the tests we want to run. Before this change,
we would incorrectly include JavaScript files that were outputs
from (since-removed) TypeScript tests. Since adding these as
arguments to the Mocha invocation causes issues on Windows bots
with "arguments too long" errors, we should be using the config
file.
2. We can add additional arguments here without the need
of passing all the arguments from `run_test_suite.py` in.
R=petermarshall@chromium.orgTBR=aerotwist@chromium.org
Bug: 1071369
Change-Id: Icab02b1117f4095081987b65c8151ddf04239e11
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2157044
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
Auto-Submit: Tim van der Lippe <tvanderlippe@chromium.org>
The shared test runner logic was reimplementing parts of Mocha,
in particular the test logging and filtering. Moreover, it was
booting the hosted mode server and puppeteer outside of Mocha.
Mocha supports root level hooks [1]. These hooks allow us to
perform work before a test starts. Moreover, by using the
before and after hook, we can run logic before and after
all tests.
By using these hooks, we can extract the "boot the hosted mode
server and hookup puppeteer" part to these root hooks.
Additionally, we can reset the pages in the `beforeEach`, which
means that tests themselves don't have to reset the pages.
We also put the implementation code into third_party/conductor,
as we would like to reuse this logic for the Puppeteer tests.
The Puppeteer test suite now also uses Mocha and has very
similar requirements as to our DevTools tests. By extracting
from DevTools, we can look into expanding the test runner to
other usecases, but that is out of scope for now.
[1]: https://mochajs.org/#root-level-hooks
Bug: 1071369
Change-Id: Ie9f954359d9de84da564b74b6f5517dd535db008
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2150458
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
This CL migrates shared code from e2e, screenshots, and unit to a helper
file. It also introduces a unified runner for e2e and screenshots (and
perf once that's in place). Once this has landed I will remove
run_e2e.py and use run_test_suite.py instead.
Change-Id: I590e5eb6f3e8a4692767a2aa653b55685891614b
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2054547
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Commit-Queue: Paul Lewis <aerotwist@chromium.org>
Liviu Rau