This CL updates the run_test_suite.js script to support the --autoninja
flag alongside --target, in which case it will do a build before running
the tests.
Unfortunately I had to duplicate the package.json entry, else we get
into a world of confusion with passing command flags through npm run
commands, and I couldn't get it to work.
Bug: none
Change-Id: I0a339c72cf41a4ffcb0241e8b3efb1b35b2e5a9f
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/4684353
Reviewed-by: Andres Olivares <andoli@chromium.org>
Commit-Queue: Andres Olivares <andoli@chromium.org>
Auto-Submit: Jack Franklin <jacktfranklin@chromium.org>
This adds two scripts: one that divides the E2E tests command into
multiple ones that can be run separately, and another script that runs
these commands in parallel Python processes. This will allow us to run
the E2E tests locally but it is also necessary for running tests in
parallel on the builders. We also remove skipOnParallel since it is
not needed anymore.
Bug: 1419802
Change-Id: I7e4c2a6f1847c3a63cab1cc2365ddaa69b77f2fb
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/4543009
Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
Reviewed-by: Danil Somsikov <dsv@chromium.org>
Commit-Queue: Almothana Athamneh <almuthanna@chromium.org>
For convenience, since the bots will still use the JSON file.
Also, an option to define the folders on which coverage will be
calculated is added to the interaction tests. The flag works
identically to the COVERAGE_FOLDERS of unit tests, i.e. a pattern is
provided to define the included folders. For example: the flag can be
set like COVERAGE_FOLDERS=
front_end/{models/trace,models/timeline_model,panels/timeline}
These two changes should help to calculate the total OPP coverage in
go/opp-test-coverage
Bug: 1424729
Change-Id: Ie982f62dfac0014c05075e53173a8e7710d3b9e0
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/4345493
Commit-Queue: Andres Olivares <andoli@chromium.org>
Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
Porting the internal script to devtools-frontend. Folders are created for each platform to prevent file not found errors.
The implementation is almost identical to the internal version, except:
- Support for windows presubmit bots, on which this didn't run before.
- Interactions root folder is passed as an argument to the script.
Bug: 1407638
Change-Id: Ia373ad258071cdcc0f7e5b417fcc2301d43084ce
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/4207837
Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
Commit-Queue: Andres Olivares <andoli@chromium.org>
This CL adds a `--log-level` flag to the unit test runner that will
allow the log level to be configured. This flag is passed right through
to Karma, but is also respected by the logs output by our test runner
before initialising Karma.
The default value for `--log-level` is `info`, so this CL doesn't change
the built in functionality; but you can now pass `debug`, `info`, `warn`
or `error` to configure it.
Bug: none
Change-Id: I82006a1a8283c3d93389f124317a37bac69b5038
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/3963112
Commit-Queue: Thiago Perrotta <tperrotta@chromium.org>
Reviewed-by: Thiago Perrotta <tperrotta@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
This CL adds the ability to run:
```
npm run auto-unittest -- --mocha-fgrep=breadcrumb
```
To the Karma unit test suite to mirror the similar flag available in the
interactions and e2e test runner script. This is also why it's named
`fgrep`, as that's the same flag as used in the other script, and we
should be consistent.
Bug: none
Change-Id: Ic119f7186e0e97c83e00bf92a66d69181c931a8b
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/3452723
Auto-Submit: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Tim Van der Lippe <tvanderlippe@chromium.org>
Commit-Queue: Tim Van der Lippe <tvanderlippe@chromium.org>
ESLint 8 removed CLIEngine. Therefore, we need to migrate away from
CLIEngine to the main ESLint class (which already exists in ESLint 7).
However, that class is based on promises, which means we can't use them
in a CommonJS context.
Therefore, introduce a new `.mjs` version that uses ES modules (which
can do as we are on a recent version of Node) and use the promise
versions. There are no other functional changes compared to the old
`.js` script.
The reason that we first have to add this file here, is that our CQ
directly refers to the `.js` version. That means we first land this CL,
we update our CQ configuration and only then can we remove the `.js`
file from our repository.
R=jacktfranklin@chromium.org
Bug: none
Change-Id: Iad3656931ae09eb7f9f23b3e6ad7d39c8a761a31
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/3268299
Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Auto-Submit: Tim van der Lippe <tvanderlippe@chromium.org>
The CQ bots all run run_test_suite.js, and all our local `npm run X`
scripts have run run_test_suite.js for many months now. I think we are
now safe to remove the Python run_test_suite.py script, as all use cases
are covered by the new JS runner.
Bug: 1186163
Change-Id: Ie8b5fd52e0934176a7da61d61fb9624b87ed764b
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/3075181
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Once we've got the `yargsObject`, we pull out all the keys we want,
which are all the configuration flags bar the special yargs keys `$0`
and `_`. The bug here was that once we pulled out that configuration, we
didn't update it, and instead kept referring to `yargsObject` in the
rest of the code. This meant that when we dynamically set the
`test-server-type` flag, we set it on `yargsObject`, but we needed to
set it on the configuration flags object so that the settings propagate
down into the test runner.
Bug: chromium:1186163
Change-Id: Ia88bb25588651fac4c11783f551651d27ccd0334
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/3075379
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
Auto-Submit: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
This change replaces the 'spec' report with a 'mocha' report in karma
when the '--expanded-reporting' flag is set. If it is not set and the
tests fail, a hint is added to the console to enable the flag to get a
more verbose output of why the tests are failing.
Bug:None
Change-Id: Ia41615362332459a4d89aef0440cea0ec266157b
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2912098
Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
Commit-Queue: Jan Scheffler <janscheffler@chromium.org>
This script can be invoked to obtain the coverage from the unit
and interaction tests and writes them to a coverage-summary.json
file that lives in `test/`. This file can then be used in favor of
karma-coverage/coverage-summary.json to upload to the Chromium
infrastructure.
As a follow-up, we should turn on coverage collection on CQ and
hook up this script after both unit and interaction tests have
completed.
R=jacktfranklin@chromium.orgCC=liviurau@chromium.org
Bug: 1206705
Change-Id: I52199940d6747ec13f208233e04dcf81a618a8fb
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2884240
Reviewed-by: Jack Franklin <jacktfranklin@chromium.org>
Commit-Queue: Tim van der Lippe <tvanderlippe@chromium.org>
This CL fixes a bug where the test runner took the `--chrome-features`
flag but Conductor never read that value when booting up a browser
instance.
This CL updates it so it will read the value and pass it into the
relevant argument when executing `puppeteer.launch`.
I also changed the Python test suite runner to not prepend
`--enable-features`; that feels like something that Conductor should
do, not the test runner.
Bug: chromium:1186163
Change-Id: I05916d39d60bb98a606570cd82e688581dd78a40
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2790874
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
This reverts commit 7742a201a1.
Reason for revert: We do have a use case for this flag that we missed, so we need it. Will follow-up with a CL to fix the fact that the test frontend infra doesn't use it currently.
Original change's description:
> [TestRunner] remove --chrome-features flag
>
> The scripts (both the old Python one and the new JS one) took
> `--chrome-features` as a flag, which was then set as
> `process.env.CHROME_FEATURES`, however searching the codebase for
> `CHROME_FEATURES` revealed that whilst we set this value, we never
> read it. Therefore this CL removes it entirely, my logic being that if
> someone needed this flag to work, we would have found this bug a long
> time ago.
>
> We can reintroduce should we find a usecase for it in the future.
>
> Bug: chromium:1186163
> Change-Id: I11d900df4aaad232fd64b9edca2aac12c94fc988
> Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2782555
> Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
> Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
> Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
Bug: chromium:1186163
Change-Id: I336a4a14a3e720f3b0626d78148038ceab9a8774
No-Presubmit: true
No-Tree-Checks: true
No-Try: true
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2786942
Bot-Commit: Rubber Stamper <rubber-stamper@appspot.gserviceaccount.com>
Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
The scripts (both the old Python one and the new JS one) took
`--chrome-features` as a flag, which was then set as
`process.env.CHROME_FEATURES`, however searching the codebase for
`CHROME_FEATURES` revealed that whilst we set this value, we never
read it. Therefore this CL removes it entirely, my logic being that if
someone needed this flag to work, we would have found this bug a long
time ago.
We can reintroduce should we find a usecase for it in the future.
Bug: chromium:1186163
Change-Id: I11d900df4aaad232fd64b9edca2aac12c94fc988
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2782555
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Tim van der Lippe <tvanderlippe@chromium.org>
Reviewed-by: Philip Pfaffe <pfaffe@chromium.org>
The current tests try to figure out the root directory and then glob
for _test.ts files within there, but this breaks easily if the folder
structure isn't exactly as expected. Instead we can set this via a
flag in the test runner.
This CL only updates the interaction tests, which use the new runner.
e2e tests will be supported in time.
Bug: chromium:1186163
Change-Id: I198b67bf0081ce7334c40d7d661fb8642c2d56d0
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2772045
Commit-Queue: Jack Franklin <jacktfranklin@chromium.org>
Auto-Submit: Jack Franklin <jacktfranklin@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
This CL adds a flag to the new test runner to provide the test-server-type.
We intelligently set it if the test suite ends with `e2e` or
`interactions`, but otherwise it's required. Unfortunately this can't
be checked as part of Yargs' API, so we check for it ourselves at run
time.
Bug: chromium:1186163
Change-Id: I6487f1a95c6c86759e1489309d4c18aec1292073
Reviewed-on: https://chromium-review.googlesource.com/c/devtools/devtools-frontend/+/2767266
Auto-Submit: Jack Franklin <jacktfranklin@chromium.org>
Commit-Queue: Paul Lewis <aerotwist@chromium.org>
Reviewed-by: Paul Lewis <aerotwist@chromium.org>
Jack Franklin