## TIP Open WiFi Test Automation

### Overview

This repository contains the test automation framework and scripts for TIP Open WiFi.
Test Framework is implemented in pytest

![Automation Overview](.img/automation_overview.png)

The vision is to create an open test framework to ensure production grade Open Wifi Solution.

This test automation tied into our overall tooling at TIP as shown below:

![Tools and Test Automation](.img/Tools_and_Testing.png)

You can get the full list of tests on [GitHub Pages](https://telecominfraproject.github.io/wlan-testing/) that's updated automatically.

### General guidelines

This testing code adheres to generic [pep8](https://www.python.org/dev/peps/pep-0008/#introduction) style guidelines, most notably:

1. [Documentation strings](https://www.python.org/dev/peps/pep-0008/#documentation-strings)
2. [Naming conventions](https://www.python.org/dev/peps/pep-0008/#prescriptive-naming-conventions)
3. [Sphynx docstring format](https://sphinx-rtd-tutorial.readthedocs.io/en/latest/docstrings.html)

Using [pytest](https://docs.pytest.org/en/6.2.x/) as the test execution framework.

Using [pylint](http://pylint.pycqa.org) for code quality monitoring.

We are using the `pylint` package to do the linting. Documentation for it can be found [here](http://pylint.pycqa.org/en/latest/).
In general, the customizations are possible via the `.pylintrc` file:

1. Line length below 120 characters is fine (search for max-line-length)
2. No new line at the end of file is fine (search for missing-final-newline)
3. Multiple new lines at the end of file are fine (search for trailing-newlines)
4. Indent using 4 spaces (search for indent-string)

In future we should enforce a policy, where we cannot merge a code where the pylint scoe goes below 7:

```shell
pylint --fail-under=7 *py
```

the command above would produce a non-zero exit code if the score drops below 7.

### TIP Controller Integration

Using Swagger Autogenerated CloudSDK Library pypi package (implemented with [swagger codegen](https://github.com/swagger-api/swagger-codegen)).

### Reporting

Currently we use the [pytest allure integration](https://docs.qameta.io/allure/#_pytest) to create visual reports for the test outcomes.

Additionally we generate overview pages, e.g. like [this one](http://openwifi-allure-reports.s3-website-us-east-1.amazonaws.com/sanity/overview/) to consolidate multiple test reports of the same test type.

These overview pages are being deployed by [this workflow](.github/workflows/update-overviews.yml) and their configuration is set in [this file](./.allure-overview/overviews.json).

### Miscellaneous

1. Do not use old style string formatting: `"Hello %s" % var`; use `f"Hello {var}` instead
2. use `"""` in Docstrings

### Useful links

https://docs.pytest.org/en/latest/example/markers.html  
https://docs.pytest.org/en/latest/usage.html  
http://pythontesting.net/framework/pytest/pytest-introduction/

### Test status

![total test count](https://raw.githubusercontent.com/Telecominfraproject/wlan-testing/badges/total-count.svg)

[![R1.x sanity testing](https://github.com/Telecominfraproject/wlan-testing/actions/workflows/quali.yml/badge.svg)](https://github.com/Telecominfraproject/wlan-testing/actions/workflows/quali.yml)
![sanity test count](https://raw.githubusercontent.com/Telecominfraproject/wlan-testing/badges/sanity-count.svg)

[![interop testing](https://github.com/Telecominfraproject/wlan-testing/actions/workflows/interop.yml/badge.svg)](https://github.com/Telecominfraproject/wlan-testing/actions/workflows/interop.yml)
![interop test count](https://raw.githubusercontent.com/Telecominfraproject/wlan-testing/badges/interop-count.svg)

[![performance testing](https://github.com/Telecominfraproject/wlan-testing/actions/workflows/performance.yml/badge.svg)](https://github.com/Telecominfraproject/wlan-testing/actions/workflows/performance.yml)
![performance test count](https://raw.githubusercontent.com/Telecominfraproject/wlan-testing/badges/performance-count.svg)

[![RF Testing](https://github.com/Telecominfraproject/wlan-testing/actions/workflows/quali-advanced.yml/badge.svg)](https://github.com/Telecominfraproject/wlan-testing/actions/workflows/quali-advanced.yml)
![Advance test count](https://raw.githubusercontent.com/Telecominfraproject/wlan-testing/badges/advance-count.svg)

[![regression testing](https://github.com/Telecominfraproject/wlan-testing/actions/workflows/regression.yml/badge.svg)](https://github.com/Telecominfraproject/wlan-testing/actions/workflows/regression.yml)
![Regression test count](https://raw.githubusercontent.com/Telecominfraproject/wlan-testing/badges/regression-count.svg)

[![interop performance testing](https://github.com/Telecominfraproject/wlan-testing/actions/workflows/interop_performance.yml/badge.svg)](https://github.com/Telecominfraproject/wlan-testing/actions/workflows/interop_performance.yml)
![Interop Performance test count](https://raw.githubusercontent.com/Telecominfraproject/wlan-testing/badges/performance-interop-count.svg)

### Best Practice

1. Ensure to have a python version > 3
2. We strongly recommend to use virtualenv to ensure that your dev environment sandbox is created.

### Code style

All code must be written in python 3 and conform to PEP 8 style guide. The test framework is built using pytest.  

[![made-with-python](https://img.shields.io/badge/Made%20with-Python-1f425f.svg)](https://www.python.org/)
[![PEP8](https://img.shields.io/badge/code%20style-pep8-orange.svg)](https://www.python.org/dev/peps/pep-0008/)  

### Directory Structure

```bash
├── lanforge              - /* to be migrated */
├── libs
│   ├── controller_tests  -/* Library Support for controller_tests part  */
    ├── apnos             -/* Library Support for Access Points (uses AP SSH)  */
│   ├── lanforge          -/* Library Support for LanForge Traffic Generator */
│   ├── perfecto          -/* Library Support for Perfecto Systems */
│   ├── testrails         -/* Result Visualization (will be depreciated ) */
├── tests                 -/* Pytest cases Directory */
│   ├── _basic_test_setup
│   ├── access_point_tests
│   ├── controller_tests
│   ├── e2e
      ├── advanced
      ├── basic
      ├── interOp
      ├── mdu
      |── mesh
      |── scale
    |── README.md         -/* Pytest framework and testcases information */
```

# Test Automation Setup Instructions 
```
# Let's assume '~/automation' directory for setting wlan-testing
cd ~/
mkdir automation
cd automation

git clone https://github.com/Telecominfraproject/wlan-testing 
cd wlan-testing/

cd tests/
vim lab_info.json    # Setup your testbed data info here
docker build -f docker/Dockerfile -t wlantest .
docker run -i -t -v $(YOUR_ALLURE_RESULT_DIR):/allure-results wlantest -c “cd tests;pytest -m ‘marker’ -s -vvv –testbed=basic-testbed –alluredir=allure-results”


# Check List of Available Tests
https://telecominfraproject.github.io/wlan-testing/

# Test Suites Marker
ow_sanity_lf: OpenWifi Sanity Test Plan



# List of Available Markers

client_connectivity_tests: Client Connectivity Test Cases with bridge|nat|vlan modes across 2.4|5|6 GHz bands with Various Encryptions
dfs_tests: Dynamic Frequency Selection Test Cases
multi_psk_tests: Multi PSK Test Cases
rate_limiting_tests: Rate Limiting Test Cases
dvlan_tests: Dynamic VLAN Test Cases
dynamic_qos_tests: Dynamic QOS Test Cases
multi_vlan_tests: Multi VLAN Combination based Test Cases
client_scale_tests: Client Capacity Tests with maximum possible Stations bridge|nat|vlan 2.4|5|6 GHz Bands
peak_throughput_tests: Single Client Peak Performance Test with various Bandwidths across 2.4|5|6 GHz Bands with various Client Types
dataplane_tests: Single Client Throughput Test with various pkt sizes with UL|DL|BI with AC|AX Client Types across 2.4|5|6 GHz Bands
country_code: Country Code Test Cases 

# Supported Modes, Bands, Encryptions
bridge: Use this marker to run bridge mode tests in each of the above test plans/suites
nat: Use this marker to run nat mode tests in each of the above test plans/suites
vlan: Use this marker to run vlan mode tests in each of the above test plans/suites

twog: Use this marker to run 2.4 GHz tests in each of the above test plans/suites
fiveg: Use this marker to run 5 GHz tests in each of the above test plans/suites
sixg: Use this marker to run 6 GHz tests in each of the above test plans/suites

open: Use this marker to run open Encryption tests in each of the above test plans/suites
wpa: Use this marker to run wpa Encryption tests in each of the above test plans/suites
wpa2_personal: Use this marker to run wpa2_personal Encryption tests in each of the above test plans/suites
wpa3_personal: Use this marker to run wpa3_personal Encryption tests in each of the above test plans/suites
wpa3_personal_mixed: Use this marker to run wpa3_personal_mixed Encryption tests in each of the above test plans/suites
wpa_wpa2_personal_mixed: Use this marker to run wpa_wpa2_personal_mixed Encryption tests in each of the above test plans/suites

wpa_enterprise: Use this marker to run wpa_enterprise Encryption tests in each of the above test plans/suites
wpa2_enterprise: Use this marker to run wpa2_enterprise Encryption tests in each of the above test plans/suites
wpa3_enterprise: Use this marker to run wpa3_enterprise Encryption tests in each of the above test plans/suites
wpa_wpa2_enterprise_mixed: Use this marker to run wpa_wpa2_enterprise_mixed Encryption tests in each of the above test plans/suites
wpa3_enterprise_mixed: Use this marker to run wpa3_enterprise_mixed Encryption tests in each of the above test plans/suites

# for visualizing allure reports, Install nodejs
# Set Java home

# install allure command
npm install -g allure-commandline --save-dev
```
### [Test Plans and Markers](TEST_PLANS.md):bookmark_tabs:

# Docker Setup Instructions

This repository support building Docker image from bundled [Dockerfile](./docker/Dockerfile). If you want to use it for testing, please refer to the [related directory for details](./docker).

### Executing and Contributing new test cases

Follow instructions provided [here](./tests/README.md)  to understand the Testing Framework, and How to run and 
write new tests

### Issue Reporting
Please submit issues using our JIRA project [here](https://telecominfraproject.atlassian.net/browse/WIFI)

Use the Test Automation [backlog](https://telecominfraproject.atlassian.net/secure/RapidBoard.jspa?projectKey=WIFI&rapidView=48&view=planning.nodetail)
