The CQ bots all run run_test_suite.js, and all our local `npm run X`
scripts have run run_test_suite.js for many months now. I think we are
now safe to remove the Python run_test_suite.py script, as all use cases
are covered by the new JS runner.
Bug: 1186163
Change-Id: Ie8b5fd52e0934176a7da61d61fb9624b87ed764b
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/3075181
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Once we've got the `yargsObject`, we pull out all the keys we want,
which are all the configuration flags bar the special yargs keys `$0`
and `_`. The bug here was that once we pulled out that configuration, we
didn't update it, and instead kept referring to `yargsObject` in the
rest of the code. This meant that when we dynamically set the
`test-server-type` flag, we set it on `yargsObject`, but we needed to
set it on the configuration flags object so that the settings propagate
down into the test runner.
Bug: chromium:1186163
Change-Id: Ia88bb25588651fac4c11783f551651d27ccd0334
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/3075379
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
Auto-Submit: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
This change replaces the 'spec' report with a 'mocha' report in karma
when the '--expanded-reporting' flag is set. If it is not set and the
tests fail, a hint is added to the console to enable the flag to get a
more verbose output of why the tests are failing.
Bug:None
Change-Id: Ia41615362332459a4d89aef0440cea0ec266157b
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2912098
Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
Commit-Queue: Jan Scheffler <janscheffler@chromium.org>
This script can be invoked to obtain the coverage from the unit
and interaction tests and writes them to a coverage-summary.json
file that lives in `test/`. This file can then be used in favor of
karma-coverage/coverage-summary.json to upload to the Chromium
infrastructure.
As a follow-up, we should turn on coverage collection on CQ and
hook up this script after both unit and interaction tests have
completed.
R=jacktfranklin@chromium.orgCC=liviurau@chromium.org
Bug: 1206705
Change-Id: I52199940d6747ec13f208233e04dcf81a618a8fb
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2884240
Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
This CL fixes a bug where the test runner took the `--chrome-features`
flag but Conductor never read that value when booting up a browser
instance.
This CL updates it so it will read the value and pass it into the
relevant argument when executing `puppeteer.launch`.
I also changed the Python test suite runner to not prepend
`--enable-features`; that feels like something that Conductor should
do, not the test runner.
Bug: chromium:1186163
Change-Id: I05916d39d60bb98a606570cd82e688581dd78a40
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2790874
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
This reverts commit 7742a201a1.
Reason for revert: We do have a use case for this flag that we missed, so we need it. Will follow-up with a CL to fix the fact that the test frontend infra doesn't use it currently.
Original change's description:
> [TestRunner] remove --chrome-features flag
>
> The scripts (both the old Python one and the new JS one) took
> `--chrome-features` as a flag, which was then set as
> `process.env.CHROME_FEATURES`, however searching the codebase for
> `CHROME_FEATURES` revealed that whilst we set this value, we never
> read it. Therefore this CL removes it entirely, my logic being that if
> someone needed this flag to work, we would have found this bug a long
> time ago.
>
> We can reintroduce should we find a usecase for it in the future.
>
> Bug: chromium:1186163
> Change-Id: I11d900df4aaad232fd64b9edca2aac12c94fc988
> Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2782555
> Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
> Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
> Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
Bug: chromium:1186163
Change-Id: I336a4a14a3e720f3b0626d78148038ceab9a8774
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2786942
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
The scripts (both the old Python one and the new JS one) took
`--chrome-features` as a flag, which was then set as
`process.env.CHROME_FEATURES`, however searching the codebase for
`CHROME_FEATURES` revealed that whilst we set this value, we never
read it. Therefore this CL removes it entirely, my logic being that if
someone needed this flag to work, we would have found this bug a long
time ago.
We can reintroduce should we find a usecase for it in the future.
Bug: chromium:1186163
Change-Id: I11d900df4aaad232fd64b9edca2aac12c94fc988
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2782555
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
The current tests try to figure out the root directory and then glob
for _test.ts files within there, but this breaks easily if the folder
structure isn't exactly as expected. Instead we can set this via a
flag in the test runner.
This CL only updates the interaction tests, which use the new runner.
e2e tests will be supported in time.
Bug: chromium:1186163
Change-Id: I198b67bf0081ce7334c40d7d661fb8642c2d56d0
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2772045
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Auto-Submit: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
This CL adds a flag to the new test runner to provide the test-server-type.
We intelligently set it if the test suite ends with `e2e` or
`interactions`, but otherwise it's required. Unfortunately this can't
be checked as part of Yargs' API, so we check for it ourselves at run
time.
Bug: chromium:1186163
Change-Id: I6487f1a95c6c86759e1489309d4c18aec1292073
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2767266
Auto-Submit: Jack Franklin <jacktfranklin@chromium.org>
Commit-Queue: Paul Lewis <aerotwist@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
That. Note that this currently isn't used or set because everyone +
all our scripts still run the Python version of the test runner
script, not the new script. But this needs to be in place before we
move people over to the new script.
We provide a fallback value such that this still works and is able to
run the tests despite the test runner not providing the configuration
via an environment variable.
Bug: chromium:1186163
Change-Id: I8255e5c7f9902be845e3c33add7b1703f6e4db9d
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2766904
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
This CL adds two flags to the new test runner for configuring the
components server and then updates the server to detect those values.
Note that whilst this CL adds yet-another-way to configure the
components server, this is only being done for backwards compat whilst
we roll out this change to all the various places. We will remove old
options towards the end of this work.
Bug: chromium:1186163
Change-Id: I4c9b85a5745b9fe36894e3c492952938a5b8f648
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2763875
Auto-Submit: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
This CL lands the initial new run_test_suite.js that can take its
configuration from a config JSON file, or from flags.
This new script is only used to run `npm run auto-interactionstest`
and nothing more; I want to roll it out slowly and ensure that
everyone is aware of it before removing the old Python script. I will
create a doc with all the various steps, and required documentation,
as I've taken the chance to rename some options to make them clearer.
Bug: chromium:1186163
Change-Id: I5a199f82ac7ab0f323988acacaa7aae2d64e6349
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2763866
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
The Localization presubmit steps include checking Loc V1 and V2:
V1: Done with migration, safe to remove
V2:
- check unused string in UIStrings (Moved to ESLint)
- check migrated directory (Done with migration, safe to remove)
- check the shape of API calls (Done by typescript or ESLint)
Since all checks are obsolete or moved, this CL removes the presubmit step to unhook the scripts inside scripts/localization. I will follow up a CL to remove the actual scripts.
The CL that introduced this step https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/1931279
Bug: 1136655
Change-Id: If203607588896693a6331de606c873458b74cd6e
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2746919
Reviewed-by: Simon Zünd <szuend@chromium.org>
Commit-Queue: Christy Chen <chrche@microsoft.com>
The current test suite runner assumes the paths; you pass
--test-suite=interactions and it assumes that they live in
out/TARGET/gen/test/interactions. This CL removes that assumption by
asking people to pass --test-suite-path, which would be
out/gen/interactions.
We will remove the --test-suite flag, but for now we're keeping it
around so the bots can still run their tests. I will file a bug to ask
that the bots have their recipes updated.
Bug:1182255
Change-Id: I6d9e5d419231845047394c5f7cbd0c74ebad3a58
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2722448
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
This reverts commit 3124f1cbbc.
Reason for revert: It breaks `npm run auto-unittest` with the following stacktrace:
```
Traceback (most recent call last):
File "scripts/test/run_auto_unittests.py", line 46, in <module>
main()
File "scripts/test/run_auto_unittests.py", line 42, in main
args.coverage)
File "devtools-frontend/scripts/test/run_unittests.py", line 86, in run_unit_tests_on_ninja_build_target
cwd)
File "devtools-frontend/scripts/test/run_unittests.py", line 28, in run_tests
'unittests', 'karma.conf.js')
File ".vpython-root/7aef5f/lib/python2.7/posixpath.py", line 70, in join
elif path == '' or path.endswith('/'):
AttributeError: 'NoneType' object has no attribute 'endswith'
```
Original change's description:
> [Test] Tweaks to test runners
>
> A couple of minor additions to unit test running.
>
> R=jacktfranklin@chromium.org
>
> Bug: None
> Change-Id: I06705d369bcc93802eb07aa5d15668c73c6bfa2a
> Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2681740
> Commit-Queue: Paul Lewis <aerotwist@chromium.org>
> Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
> Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
> Auto-Submit: Paul Lewis <aerotwist@chromium.org>
TBR=aerotwist@chromium.org,jacktfranklin@chromium.org
Change-Id: I1bbd636e1316659721db634261506428b06d9f99
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Bug: None
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2678688
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
As part of the work to move more scripts to Node, not Python, over time,
picked this one as the starting point. I changed its API slightly to
allow more flags to be taken in, as we'll need that to do a stylelint
pass against TypeScript files, but I will do that in a subsequent CL.
I had to make quite a few changes to devtools_paths.js, but I think it's
now calculating paths correctly. It took a bit of messing to get the
equivalent of Python's path.abspath(__file__), as you'll see from the
large comment that tries to explain what's going on!
Bug: chromium:1166108, chromium:1166572
Change-Id: Ia0b19ff8956b2ede2447530be57876a88046887e
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2631113
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
depot_tools ships a "vpython" binary that is added to the $PATH
for all Chromium engineers. This binary is versioned by depot_tools
which we roll in ourselves as part of `gclient sync`.
By using `vpython` instead of `python`, we are no longer depended
on the Python version installed locally and instead use the version
we pull in from DEPS.
R=liviurau@chromium.org
Change-Id: If49a54c24b6cf129ff843cd9ca1140a723a78afb
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2571462
Reviewed-by: Liviu Rau <liviurau@chromium.org>
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
Jack Franklin