This change replaces the 'spec' report with a 'mocha' report in karma
when the '--expanded-reporting' flag is set. If it is not set and the
tests fail, a hint is added to the console to enable the flag to get a
more verbose output of why the tests are failing.
Bug:None
Change-Id: Ia41615362332459a4d89aef0440cea0ec266157b
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2912098
Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
Commit-Queue: Jan Scheffler <janscheffler@chromium.org>
This script can be invoked to obtain the coverage from the unit
and interaction tests and writes them to a coverage-summary.json
file that lives in `test/`. This file can then be used in favor of
karma-coverage/coverage-summary.json to upload to the Chromium
infrastructure.
As a follow-up, we should turn on coverage collection on CQ and
hook up this script after both unit and interaction tests have
completed.
R=jacktfranklin@chromium.orgCC=liviurau@chromium.org
Bug: 1206705
Change-Id: I52199940d6747ec13f208233e04dcf81a618a8fb
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2884240
Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
This CL fixes a bug where the test runner took the `--chrome-features`
flag but Conductor never read that value when booting up a browser
instance.
This CL updates it so it will read the value and pass it into the
relevant argument when executing `puppeteer.launch`.
I also changed the Python test suite runner to not prepend
`--enable-features`; that feels like something that Conductor should
do, not the test runner.
Bug: chromium:1186163
Change-Id: I05916d39d60bb98a606570cd82e688581dd78a40
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2790874
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
This reverts commit 7742a201a1.
Reason for revert: We do have a use case for this flag that we missed, so we need it. Will follow-up with a CL to fix the fact that the test frontend infra doesn't use it currently.
Original change's description:
> [TestRunner] remove --chrome-features flag
>
> The scripts (both the old Python one and the new JS one) took
> `--chrome-features` as a flag, which was then set as
> `process.env.CHROME_FEATURES`, however searching the codebase for
> `CHROME_FEATURES` revealed that whilst we set this value, we never
> read it. Therefore this CL removes it entirely, my logic being that if
> someone needed this flag to work, we would have found this bug a long
> time ago.
>
> We can reintroduce should we find a usecase for it in the future.
>
> Bug: chromium:1186163
> Change-Id: I11d900df4aaad232fd64b9edca2aac12c94fc988
> Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2782555
> Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
> Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
> Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
Bug: chromium:1186163
Change-Id: I336a4a14a3e720f3b0626d78148038ceab9a8774
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2786942
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
The scripts (both the old Python one and the new JS one) took
`--chrome-features` as a flag, which was then set as
`process.env.CHROME_FEATURES`, however searching the codebase for
`CHROME_FEATURES` revealed that whilst we set this value, we never
read it. Therefore this CL removes it entirely, my logic being that if
someone needed this flag to work, we would have found this bug a long
time ago.
We can reintroduce should we find a usecase for it in the future.
Bug: chromium:1186163
Change-Id: I11d900df4aaad232fd64b9edca2aac12c94fc988
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2782555
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
The current tests try to figure out the root directory and then glob
for _test.ts files within there, but this breaks easily if the folder
structure isn't exactly as expected. Instead we can set this via a
flag in the test runner.
This CL only updates the interaction tests, which use the new runner.
e2e tests will be supported in time.
Bug: chromium:1186163
Change-Id: I198b67bf0081ce7334c40d7d661fb8642c2d56d0
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2772045
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Auto-Submit: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
This CL adds a flag to the new test runner to provide the test-server-type.
We intelligently set it if the test suite ends with `e2e` or
`interactions`, but otherwise it's required. Unfortunately this can't
be checked as part of Yargs' API, so we check for it ourselves at run
time.
Bug: chromium:1186163
Change-Id: I6487f1a95c6c86759e1489309d4c18aec1292073
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2767266
Auto-Submit: Jack Franklin <jacktfranklin@chromium.org>
Commit-Queue: Paul Lewis <aerotwist@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
That. Note that this currently isn't used or set because everyone +
all our scripts still run the Python version of the test runner
script, not the new script. But this needs to be in place before we
move people over to the new script.
We provide a fallback value such that this still works and is able to
run the tests despite the test runner not providing the configuration
via an environment variable.
Bug: chromium:1186163
Change-Id: I8255e5c7f9902be845e3c33add7b1703f6e4db9d
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2766904
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
This CL adds two flags to the new test runner for configuring the
components server and then updates the server to detect those values.
Note that whilst this CL adds yet-another-way to configure the
components server, this is only being done for backwards compat whilst
we roll out this change to all the various places. We will remove old
options towards the end of this work.
Bug: chromium:1186163
Change-Id: I4c9b85a5745b9fe36894e3c492952938a5b8f648
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2763875
Auto-Submit: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
This CL lands the initial new run_test_suite.js that can take its
configuration from a config JSON file, or from flags.
This new script is only used to run `npm run auto-interactionstest`
and nothing more; I want to roll it out slowly and ensure that
everyone is aware of it before removing the old Python script. I will
create a doc with all the various steps, and required documentation,
as I've taken the chance to rename some options to make them clearer.
Bug: chromium:1186163
Change-Id: I5a199f82ac7ab0f323988acacaa7aae2d64e6349
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2763866
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
The Localization presubmit steps include checking Loc V1 and V2:
V1: Done with migration, safe to remove
V2:
- check unused string in UIStrings (Moved to ESLint)
- check migrated directory (Done with migration, safe to remove)
- check the shape of API calls (Done by typescript or ESLint)
Since all checks are obsolete or moved, this CL removes the presubmit step to unhook the scripts inside scripts/localization. I will follow up a CL to remove the actual scripts.
The CL that introduced this step https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/1931279
Bug: 1136655
Change-Id: If203607588896693a6331de606c873458b74cd6e
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2746919
Reviewed-by: Simon Zünd <szuend@chromium.org>
Commit-Queue: Christy Chen <chrche@microsoft.com>
The current test suite runner assumes the paths; you pass
--test-suite=interactions and it assumes that they live in
out/TARGET/gen/test/interactions. This CL removes that assumption by
asking people to pass --test-suite-path, which would be
out/gen/interactions.
We will remove the --test-suite flag, but for now we're keeping it
around so the bots can still run their tests. I will file a bug to ask
that the bots have their recipes updated.
Bug:1182255
Change-Id: I6d9e5d419231845047394c5f7cbd0c74ebad3a58
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2722448
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
This reverts commit 3124f1cbbc.
Reason for revert: It breaks `npm run auto-unittest` with the following stacktrace:
```
Traceback (most recent call last):
File "scripts/test/run_auto_unittests.py", line 46, in <module>
main()
File "scripts/test/run_auto_unittests.py", line 42, in main
args.coverage)
File "devtools-frontend/scripts/test/run_unittests.py", line 86, in run_unit_tests_on_ninja_build_target
cwd)
File "devtools-frontend/scripts/test/run_unittests.py", line 28, in run_tests
'unittests', 'karma.conf.js')
File ".vpython-root/7aef5f/lib/python2.7/posixpath.py", line 70, in join
elif path == '' or path.endswith('/'):
AttributeError: 'NoneType' object has no attribute 'endswith'
```
Original change's description:
> [Test] Tweaks to test runners
>
> A couple of minor additions to unit test running.
>
> R=jacktfranklin@chromium.org
>
> Bug: None
> Change-Id: I06705d369bcc93802eb07aa5d15668c73c6bfa2a
> Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2681740
> Commit-Queue: Paul Lewis <aerotwist@chromium.org>
> Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
> Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
> Auto-Submit: Paul Lewis <aerotwist@chromium.org>
TBR=aerotwist@chromium.org,jacktfranklin@chromium.org
Change-Id: I1bbd636e1316659721db634261506428b06d9f99
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: None
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2678688
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
As part of the work to move more scripts to Node, not Python, over time,
picked this one as the starting point. I changed its API slightly to
allow more flags to be taken in, as we'll need that to do a stylelint
pass against TypeScript files, but I will do that in a subsequent CL.
I had to make quite a few changes to devtools_paths.js, but I think it's
now calculating paths correctly. It took a bit of messing to get the
equivalent of Python's path.abspath(__file__), as you'll see from the
large comment that tries to explain what's going on!
Bug: chromium:1166108, chromium:1166572
Change-Id: Ia0b19ff8956b2ede2447530be57876a88046887e
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2631113
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
depot_tools ships a "vpython" binary that is added to the $PATH
for all Chromium engineers. This binary is versioned by depot_tools
which we roll in ourselves as part of `gclient sync`.
By using `vpython` instead of `python`, we are no longer depended
on the Python version installed locally and instead use the version
we pull in from DEPS.
R=liviurau@chromium.org
Change-Id: If49a54c24b6cf129ff843cd9ca1140a723a78afb
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2571462
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
This CL introduces the infrastructure and first test for what we're calling
"Interaction tests". These are tests that can be used to test component
behaviour that cannot be tested via Karma unit tests but don't require an e2e
test either.
The best example here is complex user interactions: clicking and dragging a
mouse, or doing multiple tabs to navigate through the UI.
Note that the one interaction test added here for theme_colors isn't the best
example of a complex test, but it's a test that's good enough for now whilst we
get everything set up on infrastructure and make sure it runs without flaking.
Bug: 1153281
Change-Id: Ifb37639a557c459974f703b8954ae68a93d5d368
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2566808
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
As part of the work to enable Puppeteer component tests, we need to
configure the test suite to run either the hosted mode server or the
component docs server. This CL updates it to take a flag, and makes some
updates to the component docs server, which now has to run either
directly or in the out/Default/gen directory depending on how it is run.
I suspect I'll make a follow up CL to always run the component server in
out/Default/gen, but for now enabling it to detect its context is the
quickest way to unblock running it in tests. The next CL will add a
component test suite that can run a basic test against the component doc
server, but I have manually verified locally that I can run tests
against that server.
Bug: 1153281
Change-Id: I55bda4edd0a983d03bfecff0446b0f3e3a008b52
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2562707
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Only set the unhandled rejection handler once per process.
We run the setup and teardown multiple times per-process now.
Add clearPuppeteerState() to unset the state between runs on
a parallel test runner. This is because the port changes
between successive runs in the same process.
Remove the hasShutdown logic from mocha_hooks. Also remove the
beforeExit hook which was causing this to get called at least
twice for every process. It already gets called in the ordinary
shutdown case via the afterAll() hook. We only need to call it
in the extraordinary case which is the SIGINT case. Now that we
do the shutdown multiple times per process, we don't care if we
already shutdown on this process or not. The only issue could be
if ctrl+c is sent during the shutdown, but then we are crashing
anyway.
Right now the default # of jobs is 1, so out bots will still run
in serial mode. We want to add parallel mode as an options for
local development while we iron out any last problems with this
approach.
You can test this locally with npm run e2etests -- --jobs=8.
We make use of global setup fixtures to only run one hosted
mode server and share it between all parallel runners. Each
runner still starts its own chrome and restarts it between
files, which is a bit inefficient but still a big improvement
compared to serial mode.
Split the hosted mode server out into its own file inside
conductor as it can be dealt with entirely separately, and
its state is not per-process like the chrome state, but only
per-main-process which launches the test runner sub-processes.
Bug: 1101784
Change-Id: Ic0c9be3559708fd5403a88882b9fb93257633de9
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2288694
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Commit-Queue: Peter Marshall <petermarshall@chromium.org>
There previously were two special files that were handled
by `build_release_applications`: root.js and RuntimeInstantiator.js.
Both files are explicitly part of the startup process of DevTools
and its various entrypoints.
To remove the copying from `build_release_applications`, we have
to move these to the relevant `devtools_entrypoint`. Since a
`devtools_entrypoints` bundles all subdirectories, we can't keep
these files in `front_end/` directly. Instead, we move these files
to `startup/` to denote their special-casing in the startup process.
Next to that, we have to fix all usages of these files in the entrypoints.
For all JavaScript entrypoint files, all side-effect legacy files
that are loaded by `startup.js` (previously known as `root.js` and renamed
to prevent confusion with the `root/` module) are removed. All
"additional" legacy files that a particular entrypoint requires are
still loaded as-is.
The RuntimeInstantiator is moved to become an implementation detail
of `startup/`. Therefore, all of the usages that were previously
importing from `RuntimeInstantiator` now import via `startup.js`.
In the end, the special-casing of these files are removed and renamed
for clarity. In the future, we want to remove the complicated
entrypoints startup process, but we are not ready for that yet.
That will require additional cleanups with `resources` in `module.json`
before that change can happen.
R=aerotwist@chromium.org
Bug: 1131500
Change-Id: I198d5a62d2aab70f842c68d9b4c0871fde1587a4
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2485072
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Alex Rudenko