Newer
Older
deqp_runner and deqp_submit
===========================
Tools to run tests from the deqp suite and submit them to git.
### Build the OpenGL/Vulkan test suite
Instructions based on the following document:
https://github.com/KhronosGroup/VK-GL-CTS/blob/master/external/openglcts/README.md
git clone https://github.com/KhronosGroup/VK-GL-CTS.git
cd VK-GL-CTS
mkdir build
cd build
cmake .. -DDEQP_TARGET=x11_egl -DGLCTS_GTF_TARGET=gles2
cmake --build .
To run the tools you need the 'sh' python3 module. Get it with:
$ sudo apt install python3-sh
The deqp_runner tool
--------------------
deqp_runner.py [-h] [--hasty] [--cts-build-dir CTS_BUILD_DIR]
[--test-names-file TEST_NAMES_FILE] [--timeout TIMEOUT]
[filter]
If you don't specify a CTS-BUILD-DIR the current directory must be the CTS
build directory. The tests to run can be specified either as a filter or
as a list of tests names (one test name per line) in a file.
Run the whole GLES2 dEQP suite, from the provide CTS build dir:
deqp_runner.py --cts-build-dir ~/VK-GL-CTS/build dEQP-GLES2
Run only a subset of the GLES2 dEQP suite:
deqp_runner.py dEQP-GLES2.stress.draw.unaligned_data
By default, each test matching the provided TEST-PREFIX will be run
individually. You can run tests in larger batches using the ``--hasty`` flag:
deqp_runner.py --hasty dEQP-GLES2.stress.draw.unaligned_data
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
Run specific tests listed in tests.txt with a timeout of 100 seconds:
deqp_runner.py --timeout 100 --test-names-file tests.txt
The deqp_submit tool
--------------------
deqp_submit.py [-h] diff --results-file RESULTS_FILE file [file ...]
deqp_submit.py [-h] upload --results-file RESULTS_FILE
--message MESSAGE [--commit-only] file [file ...]
The deqp_submit.py tool process the output of the deqp_runner.py tool and
commits and uploads the results to git, merging with existing results.
The deqp_submit.py tools provides two subcommands, diff and upload. The diff
command shows a diff of the changes to the central repository which would
result from processing the specified input files. The upload command
commits and uploads the updated results to the central repository.
Both subcommand require the user to specify the name of the results file, which
will be stored in a user-specific directory in the repository. The directory
name is determined using the git email, so the ```user.email``` field should be
set in the git settings. If the results file already exists, then the result in
that file will be updated with the new results. It is recommended to use
meaningful names like, e.g., dEQP-GLES2 for the results of dEQP-GLES2 test
runs.
NOTE: For the submit command to work you need to have access to the central
results repository in way that doesn't require user input. If the initial clone
seems to take forever, ensure that you have all your ssh keys set up correctly.
Do a manual test of 'git clone git@gitlab.collabora.com:virgl-es/deqp_results'
to ensure everything is working correctly.
### Examples
Show the diff that would result when processing a results file:
deqp_submit.py diff --results-file dEQP-GLES2 deqp-gles2-results.txt | less -r
Upload results:
deqp_submit.py upload --results-file dEQP-GLES2 --message "Initial run" deqp-gles2-results.txt
Upload results and change commiter email permanently:
deqp_submit.py upload --email this.is@.really.me --results-file dEQP-GLES2 --message "Initial run" deqp-gles2-results.txt
Commit changes to locally cached repo, but don't upload:
deqp_submit.py upload --results-file dEQP-GLES2 --commity-only --message "Initial run" deqp-gles2-results.txt
Process multiple files at once:
deqp_submit.py diff --results-file dEQP-GLES2 results1.txt results2.txt
The deqp_filter tool
--------------------
deqp_filter.py [-h] --test-names-file TEST_NAMES_FILE [results_file]
The deqp_filter.py tool processes the specifed results file (or data from stdin
if no results file is specfied) and output to stdout a new results file
containing only the test listed in the specified TEST_NAMES_FILE.
If the input results file is missing tests contained in TEST_NAMES_FILE then
the names of the missing tests are added as comments at the top of the output
results file.
### Examples
Output a new results file containing only the results from tests in
deqp-gles2-on-gl-host which are listed in gles2-master.txt
deqp_filter.py --test-names-file gles2-master.txt deqp-gles2-on-gl-host