Skip to content
Snippets Groups Projects
README.md 4.52 KiB
Newer Older
deqp_runner and deqp_submit
===========================
Tools to run tests from the deqp suite and submit them to git.
Requirements 
------------
### Build the OpenGL/Vulkan test suite

Instructions based on the following document:
https://github.com/KhronosGroup/VK-GL-CTS/blob/master/external/openglcts/README.md

Alexandros Frantzis's avatar
Alexandros Frantzis committed
    git clone https://github.com/KhronosGroup/VK-GL-CTS.git
    cd VK-GL-CTS
    mkdir build
    cd build
    cmake .. -DDEQP_TARGET=x11_egl -DGLCTS_GTF_TARGET=gles2
    cmake --build .
### Required python modules
To run the tools you need the 'sh' python3 module. Get it with:

    $ sudo apt install python3-sh

The deqp_runner tool
--------------------

    deqp_runner.py [-h] [--hasty] [--cts-build-dir CTS_BUILD_DIR]
                   [--test-names-file TEST_NAMES_FILE] [--timeout TIMEOUT]
                   [filter]

If you don't specify a CTS-BUILD-DIR the current directory must be the CTS
build directory. The tests to run can be specified either as a filter or
as a list of tests names (one test name per line) in a file.

Run the whole GLES2 dEQP suite, from the provide CTS build dir:

Alexandros Frantzis's avatar
Alexandros Frantzis committed
    deqp_runner.py --cts-build-dir ~/VK-GL-CTS/build dEQP-GLES2

Run only a subset of the GLES2 dEQP suite:

Alexandros Frantzis's avatar
Alexandros Frantzis committed
    deqp_runner.py dEQP-GLES2.stress.draw.unaligned_data

By default, each test matching the provided TEST-PREFIX will be run
individually. You can run tests in larger batches using the ``--hasty`` flag:

Alexandros Frantzis's avatar
Alexandros Frantzis committed
    deqp_runner.py --hasty dEQP-GLES2.stress.draw.unaligned_data

Run specific tests listed in tests.txt with a timeout of 100 seconds:

    deqp_runner.py --timeout 100 --test-names-file tests.txt

The deqp_submit tool
--------------------

    deqp_submit.py [-h] diff --results-file RESULTS_FILE file [file ...]
    deqp_submit.py [-h] upload --results-file RESULTS_FILE
                   --message MESSAGE [--commit-only] file [file ...]

The deqp_submit.py tool process the output of the deqp_runner.py tool and
commits and uploads the results to git, merging with existing results.

The deqp_submit.py tools provides two subcommands, diff and upload. The diff
command shows a diff of the changes to the central repository which would
result from processing the specified input files. The upload command
commits and uploads the updated results to the central repository.

Both subcommand require the user to specify the name of the results file, which
will be stored in a user-specific directory in the repository. The directory
name is determined using the git email, so the ```user.email``` field should be
set in the git settings. If the results file already exists, then the result in
that file will be updated with the new results.  It is recommended to use
meaningful names like, e.g., dEQP-GLES2 for the results of dEQP-GLES2 test
runs.

NOTE: For the submit command to work you need to have access to the central
results repository in way that doesn't require user input. If the initial clone
seems to take forever, ensure that you have all your ssh keys set up correctly.
Do a manual test of 'git clone git@gitlab.collabora.com:virgl-es/deqp_results'
to ensure everything is working correctly.

### Examples

Show the diff that would result when processing a results file:

    deqp_submit.py diff --results-file dEQP-GLES2 deqp-gles2-results.txt | less -r

Upload results:

    deqp_submit.py upload --results-file dEQP-GLES2 --message "Initial run" deqp-gles2-results.txt


Upload results and change commiter email permanently:

    deqp_submit.py upload --email this.is@.really.me --results-file dEQP-GLES2 --message "Initial run" deqp-gles2-results.txt

Commit changes to locally cached repo, but don't upload:

    deqp_submit.py upload --results-file dEQP-GLES2 --commity-only --message "Initial run" deqp-gles2-results.txt

Process multiple files at once:

    deqp_submit.py diff --results-file dEQP-GLES2 results1.txt results2.txt

The deqp_filter tool
--------------------

    deqp_filter.py [-h] --test-names-file TEST_NAMES_FILE [results_file]

The deqp_filter.py tool processes the specifed results file (or data from stdin
if no results file is specfied) and output to stdout a new results file
containing only the test listed in the specified TEST_NAMES_FILE.

If the input results file is missing tests contained in TEST_NAMES_FILE then
the names of the missing tests are added as comments at the top of the output
results file.

### Examples

Output a new results file containing only the results from tests in
deqp-gles2-on-gl-host which are listed in gles2-master.txt

    deqp_filter.py --test-names-file gles2-master.txt deqp-gles2-on-gl-host