This patch makes the framework verify not only the result but also
the execution order of the tests. It also allows each test to
specify expected return code and strings printed by TH and DUT.
The final test results depends on the return code and the expectation.
Therefore, the output now includes 'RESULT' column showing PASS or FAIL:
test name TH_RETURN_CODE DUT_RETURN_CODE TH_STR DUT_STR RESULT
test_task_switch SUCCESS SUCCESS 1 1 PASS
test_task_priority SUCCESS FAILURE 1 1 FAIL
test_stack_overflow DID_NOT_END DID_NOT_END 1 1 PASS
Additionally, this patch:
* Adds CTS_RC_DID_NOT_START and CTS_RC_DID_NOT_END to indicate whether
the test did start or end, respectively.
* Makes stack overflow test check whether stack overflow was detected
and reboot occurred
* Removes post_corruption_test and conflict test since now
the test results are stricly compared against expected results.
* Fixes gpylint errors.
BUG=none
BRANCH=none
TEST=Run gpio, meta, timer, interrupt, and cts/cts.py -m task
Change-Id: I3b7005236e705dcac0c8f4711b44c85ff9a4f676
Reviewed-on: https://chromium-review.googlesource.com/538878
Commit-Ready: Daisuke Nojiri <dnojiri@chromium.org>
Tested-by: Daisuke Nojiri <dnojiri@chromium.org>
Reviewed-by: Daisuke Nojiri <dnojiri@chromium.org>
Currently, read_tty reads characters from tty as long as there
is something to read. This causes read_tty to loop forever if the board
is in reboot loop.
This patch makes cts.py stop reading tty if boot count exceeds
max_boot_count. Reboot is detected by detecting REBOOT_MARKER.
This patch also does:
- Remove debug option: This adds complexity for no real value. Developers
should debug tests using regular tools (make, uart console, etc.).
- Remove html output. Nobody use it. Should be redone when it's needed
using proper libraries.
BUG=chromium:664309
BRANCH=none
TEST=cts.py -m task/gpio/interrupt
Change-Id: I51d1dd51c4097e8115ef04ad46853720295141b4
Signed-off-by: Daisuke Nojiri <dnojiri@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/410281
Reviewed-by: Randall Spangler <rspangler@chromium.org>
This patch hides expected messages from output to the terminal
and reorganizes log directory as follows:
- All test output goes under /tmp/ects/$dut/$module
- Openocd output is recorded in openocd.log
- uart outputs are recorded in uart_th.log and uart_dut.log
- build output is recorded in build.log
- Check exit code from all subprocess calls
- Dump build log if build fails
- Dump openocd log if openocd fails
BUG=chromium:664309
BRANCH=none
TEST=cts.py --setup and build images in chroot then run cts.py -m meta
outside chroot.
Change-Id: I13294c3bf777ad7ae590459d3cf5aea405d59f96
Signed-off-by: Daisuke Nojiri <dnojiri@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/409536
Reviewed-by: Randall Spangler <rspangler@chromium.org>
Add a nested interrupt test to eCTS. Higher priority IRQ is fired,
followed by lower priority IRQ. Handlers should be executed
sequentially.
P1 *-----*
/ \
P2 / *-----*
/ \
task_cts ----* *----
B C A D
BUG=chromium:653195
BRANCH=none
TEST=cts.py -m interrupt; make buildall
Change-Id: Ia9f1bf4205cefe8bdc11cc0aa3ad2057359b73ef
Signed-off-by: Daisuke Nojiri <dnojiri@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/409611
Reviewed-by: Randall Spangler <rspangler@chromium.org>
After a board is reset for setting up a tty port, the host should
wait for a few seconds before flushing the port as the board may
still be booting. This should prevent output from the previous boot
from creeping into a test run.
BUG=none
BRANCH=none
TEST=cts.py -m gpio, interrupt, timer
Change-Id: I1fb567a3a8ddcfff61865b6db3866c56be386c4a
Signed-off-by: Daisuke Nojiri <dnojiri@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/408759
Reviewed-by: Randall Spangler <rspangler@chromium.org>
Currently eCTS suites share the same directory (e.g. build/stm32l476g
-eval) to put build artifacts even though some files (e.g. board.c)
compile differently suite to suite. So, if cts-i2c-stm32l476g-eval is
built, followed by cts-gpio-stm32l476g-eval, build fails or produces
incorrect binary.
This patch makes eCTS create different directories for each suite.
As a bonus, we can now builds eCTS suites in parallel.
BUG=chromium:654549
BRANCH=none
TEST=make buildall -j (with uncommitted change)
Change-Id: I4abedc917787be5f79b97e0e50d0d08e01bd5f9d
Signed-off-by: Daisuke Nojiri <dnojiri@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/398281
This patch fixes the error messages displayed when a UART port connected
to DUT or TH is being occupied.
BUG=none
BRANCH=none
TEST=run cts.py -m i2c
Change-Id: I3fbb4068e8ee3af7a1b04f70ae70b3d870a19d2e
Signed-off-by: Daisuke Nojiri <dnojiri@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/393327
This change makes cts.py fail when building or flashing a module for
DUT or TH fails.
BUG=none
BRANCH=none
TEST=Made cts.py fail by injecting build and flash error
Change-Id: Iec1e11f4a8c261eb4c989b118df218e86cb6f5f1
Signed-off-by: Daisuke Nojiri <dnojiri@chromium.org>
Reviewed-on: https://chromium-review.googlesource.com/393326
Reviewed-by: Randall Spangler <rspangler@chromium.org>
Noteworthy changes:
- Move Board and its child classes in common/board.py
- Separate flashing and resetting. Flashing used to imply running tests.
- Move up constants up for better visibility
- Change default suite to 'meta'
- Removed redundant code
- Lots of renames (all lower case names, shorter names, etc.)
BUG=none
BRANCH=none
TEST=Ran meta test and verify the results match the expectations
Change-Id: I158d96e2ee104767d25b2e721d5206e528600381
Reviewed-on: https://chromium-review.googlesource.com/383911
Commit-Ready: Daisuke Nojiri <dnojiri@chromium.org>
Tested-by: Daisuke Nojiri <dnojiri@chromium.org>
Reviewed-by: Randall Spangler <rspangler@chromium.org>
You can now add debug messages into a cts test and
they will be displayed with the test report html page.
The macro to use is CTS_DEBUG_PRINTF
Adding debug messages can potentially change test
results by slowing down the test, so you can choose
when compiling a test suite if you want the debug
messages present or not by adding --debug as an
argument when you call ./cts --build.
BRANCH=None
BUG=None
TEST=Manual
- Add a debug statement to a test
- Build the test suite with --debug specified
- Flash the boards
- Run './cts/cts.py -r'
- Open /tmp/cts_results/<board_name>/<test_suite>.html
to view see your debug message for the test
Change-Id: Icad8e0ac5cc905010caa4e7616f81572ce6ac771
Reviewed-on: https://chromium-review.googlesource.com/362475
Commit-Ready: Chris Chen <twothreecc@google.com>
Tested-by: Chris Chen <twothreecc@google.com>
Reviewed-by: Randall Spangler <rspangler@chromium.org>
If test suite doesn't finish or results are received
out of order, this is likely an indication of reset
or hang by one of the boards and all tests after the
point of corruption are marked as corrupted.
BRANCH=None
BUG=None
TEST=Manual
- Edit the gpio th code to change ordering of tests
or hang or reset, etc.
- Build and flash tests
- Run './cts/cts.py -r'
- You should see the results for all of the tests,
with all corrupted tests marked as corrupted
Change-Id: I7925e37db285a4e90e6e09bf3b187400ddfe9edf
Reviewed-on: https://chromium-review.googlesource.com/362614
Commit-Ready: Chris Chen <twothreecc@google.com>
Tested-by: Chris Chen <twothreecc@google.com>
Reviewed-by: Randall Spangler <rspangler@chromium.org>
When you run a test suite, cts will now save your
results for the suite/dut combo as a basic html page
BRANCH=None
BUG=None
TEST=Manual
- Connect dut, th
- Build/flash desired test suite
- Run './cts -r' to run tests
- Open /tmp/cts_results/<board_name>/<test_suite>.html
- You should see a table with test names/results
Change-Id: Id3de3bd7833be1bc5dde437c516db411aac47579
Reviewed-on: https://chromium-review.googlesource.com/362091
Commit-Ready: Chris Chen <twothreecc@google.com>
Tested-by: Chris Chen <twothreecc@google.com>
Reviewed-by: Randall Spangler <rspangler@chromium.org>
Added in classes for Board (parent), DeviceUnderTest,
and TestHarness. Reading, etc. should be easier now
BRANCH=None
BUG=None
TEST=Manual
- Build default
- Flash default
- Run
- Open /tmp/cts_results/nucleo-f072rb/gpio.html
- Should see a clean results page
Change-Id: Ide3f75281f0b5b8b40dabd36f8c239737dc527d6
Reviewed-on: https://chromium-review.googlesource.com/364236
Commit-Ready: Chris Chen <twothreecc@google.com>
Tested-by: Chris Chen <twothreecc@google.com>
Reviewed-by: Daisuke Nojiri <dnojiri@chromium.org>
Added test recording when calling reset from command
line. These results are printed on the screen and
saved in /tmp/results/<board>/<module>.txt
BRANCH=None
BUG=None
TEST=Manual
- Connect, build and flash boards
- Navigate to ec/cts
- ./cts.py --run
- Find test results /tmp/results/<board>/<module>.txt
- Tests names should be left aligned in one column
and their results right aligned in a 2nd column
Change-Id: I3429d6092f2bd5d5f6825245f5439ace3f47f1fa
Reviewed-on: https://chromium-review.googlesource.com/360653
Commit-Ready: Chris Chen <twothreecc@google.com>
Tested-by: Chris Chen <twothreecc@google.com>
Reviewed-by: Randall Spangler <rspangler@chromium.org>
Contains code for all the gpio tests so far. Code in
cts_task for th and dut is for testing purposes and
test result reporting will be updated in the next
patch.
BRANCH=None
BUG=None
TEST=Manual
- Connect handshake and gpio test lines between th and dut
- Build tests
- run 'cat /dev/ttyACM0' in one terminal
- run 'cat /def/ttyACM1' in another
- Flash boards
- All test results should print either passed or unknown
Change-Id: I7142fb87a6ce0a20c571cde608fbbe60e35898ea
Reviewed-on: https://chromium-review.googlesource.com/359935
Commit-Ready: Daisuke Nojiri <dnojiri@chromium.org>
Tested-by: Daisuke Nojiri <dnojiri@chromium.org>
Reviewed-by: Daisuke Nojiri <dnojiri@chromium.org>
The first time you use this with a particular th,
connect only th and run ./cts.py --th
Then connect both boards and you can run
./cts.py to build/flash both boards.
BRANCH=None
BUG=None
TEST=manual
- Enter chroot
- Navigate to ec/cts
- Connect only th
- 'sudo ./cts.py --th'
- './cts.py -b'
- Exit chroot
- Connect both boards
- './cts.py -f'
Each board should flash successfully
Change-Id: Ib14fccabcd9fdad04f9b92817da597bc0dcb3d89
Reviewed-on: https://chromium-review.googlesource.com/358100
Commit-Ready: Chris Chen <twothreecc@google.com>
Tested-by: Chris Chen <twothreecc@google.com>
Reviewed-by: Daisuke Nojiri <dnojiri@chromium.org>
