|
| 1 | +# Devfile Parser Library Tests |
| 2 | + |
| 3 | +## About |
| 4 | + |
| 5 | +The tests use the go language and are intended to test every apsect of the parser for every schema attribute. Some basic aspects of the tests: |
| 6 | + |
| 7 | +* A first test (parser_v200_schema_test.go) feeds pre-created devfiles to the parser to ensure the parser can parse all attribues and return an appropriate error when the devfile contains an error. This test is not currently available. |
| 8 | +* A second set of tests (parser_v200_verify_test.go) create devfile content at runtime: |
| 9 | + * Devfile content is randomly generated and as a result the tests are designed to run multiple times. |
| 10 | + * Parser functions covered: |
| 11 | + * Read an existing devfile. |
| 12 | + * Write a new devfile. |
| 13 | + * Modify content of a devfile. |
| 14 | + * Multi-threaded access to the parser. |
| 15 | + * The tests use the devfile schema to create a structure containing expected content for a devfile. These structures are compared with those returned by the parser. |
| 16 | + * sigs.k8s.io/yaml is used to write out devfiles. |
| 17 | + * github.com/google/go-cmp/cmp is used to compare structures. |
| 18 | + |
| 19 | +## Current tests: |
| 20 | + |
| 21 | +The tests using pre-created devfiles are not currently available (update in progress due to schema changes) |
| 22 | + |
| 23 | +The tests which generate devfiles with random content at run time currently cover the following properties and items. |
| 24 | + |
| 25 | +* Commands: |
| 26 | + * Exec |
| 27 | + * Composite |
| 28 | +* Components |
| 29 | + * Container |
| 30 | + * Volume |
| 31 | + |
| 32 | +## Running the tests |
| 33 | + |
| 34 | +1. Go to directory tests/src/tests |
| 35 | +1. Run ```go test``` or ```go test -v``` |
| 36 | +1. The test creates the following files: |
| 37 | + 1. ```./tmp/test.log``` contains log output from the tests. |
| 38 | + 1. ```./tmp/parser_v200_verify_test/Test_*.yaml``` are the devfiles which are randomly generated at runtime. The file name matches the name of the test function which resulted in them being created. |
| 39 | + 1. ```./tmp/parser_v200_schema_test/*.yaml``` are the pre-created devfiles. |
| 40 | + 1. If a test detects an error when compariing properties returned by the parser with expected properties |
| 41 | + * ```./tmp/parser_v200_schema_test/Test_*_<property id>_Parser.yaml``` - property as returned by the parser |
| 42 | + * ```./tmp/parser_v200_schema_test/Test_*_<property id>_Test.yaml``` - property as expected by the test |
| 43 | + |
| 44 | +Note: each run of the test removes the existing conents of the ```./tmp``` directory |
| 45 | + |
| 46 | +## Anatomy of parser_v200_verify_test.go test |
| 47 | + |
| 48 | +Each test in ```parser_v200_verify_test.go``` sets values in a test structure which defines the test to run (additions will new made for new properties as support is added): |
| 49 | + |
| 50 | + type TestContent struct { |
| 51 | + CommandTypes []schema.CommandType |
| 52 | + ComponentTypes []schema.ComponentType |
| 53 | + FileName string |
| 54 | + CreateWithParser bool |
| 55 | + EditContent bool |
| 56 | + } |
| 57 | + |
| 58 | + |
| 59 | +The test then uses one (or both) of two functions to run a test |
| 60 | + |
| 61 | +* For a single thread test: |
| 62 | + * ```func runTest(testContent TestContent, t *testing.T)``` |
| 63 | +* For a multi-thread test: |
| 64 | + * ```func runMultiThreadTest(testContent TestContent, t *testing.T)``` |
| 65 | + |
| 66 | +An example test: |
| 67 | + |
| 68 | + func Test_MultiCommand(t *testing.T) { |
| 69 | + testContent := TestContent{} |
| 70 | + testContent.CommandTypes = []schema.CommandType{schema.ExecCommandType, schema.CompositeCommandType} |
| 71 | + testContent.CreateWithParser = true |
| 72 | + testContent.EditContent = true |
| 73 | + testContent.FileName = GetDevFileName() |
| 74 | + runTest(testContent, t) |
| 75 | + runMultiThreadTest(testContent, t) |
| 76 | + } |
| 77 | + |
| 78 | +Note: ```GetDevFileName()``` is a function which returns the name of a temporary file to create which uses the name of the test file as a subdirectory and the name of test function function as file name. In this example it returns ```./tmp/parser_v200_verify_test/Test_MultiCommand``` |
| 79 | + |
| 80 | +There are also some constants which control execution of the tests: |
| 81 | + |
| 82 | + const numThreads = 5 // Number of threads used by multi-thread tests |
| 83 | + const maxCommands = 10 // The maximum number of commands to include in a generated devfile |
| 84 | + const maxComponents = 10 // The maximum number of components to include in a generated devfile |
| 85 | + |
| 86 | +## Basic principles of the tests which randomly generates devfiles |
| 87 | + |
| 88 | +* Each devfile is created in a schema structure. |
| 89 | +* Which attributes are set and the values used are randomized. |
| 90 | + * For example, the number of commands included in a devfile is randomly generated. |
| 91 | + * For example, attribute values are set to randomized strings, numbers or binary. |
| 92 | + * For example, a particular optional attribute has a 50% chance of being uncluded in a devfiles. |
| 93 | + * Repeated tests give more variety and wider coverage. |
| 94 | +* Once the schema structure is complete it is written in one of two ways. |
| 95 | + * using the sigs.k8s.io/yaml. |
| 96 | + * using the parser. |
| 97 | +* Once the devfile is created on disk the parser is used to read and validate it. |
| 98 | +* If editing the devfile |
| 99 | + * each object is retrieved, modified and written back to the parser |
| 100 | + * the parser is used to write the devfile to disk |
| 101 | + * the parser is then used to read and validate the modified devfile. |
| 102 | +* Each array of objects in then devfile are then retrieved from the parser and compared. If this fails: |
| 103 | + * Each object returned by the parser is compared to the equivalent object tracked in the test. |
| 104 | + * if the obejcts do not match the test fails |
| 105 | + * Files are output with the content of each object. |
| 106 | + * If the parser returns more or fewer objects than expected, the test fails. |
| 107 | + |
| 108 | +## Updating tests |
| 109 | + |
| 110 | +### Files |
| 111 | +* ```parser_v200_verify_test.go``` contains the tests |
| 112 | +* ```test-utils.go``` provides property agnostic functions for use by the tests and other utils |
| 113 | +* ```<property>-test-utils.go``` for example ```command-test-utils.go```, provides property related functions |
| 114 | + |
| 115 | +### Adding, modifying attributes of existing properties. |
| 116 | + |
| 117 | +In the ```<property>-test-utils.go``` files there are: |
| 118 | +* ```set<Item>Values``` functions. |
| 119 | + * for example in ```command-test-utils.go``` : |
| 120 | + * ```func setExecCommandValues(execCommand *schema.ExecCommand)``` |
| 121 | + * ```func setCompositeCommandValues(compositeCommand *schema.CompositeCommand)``` |
| 122 | +* These may use utility functions to set more complex attributes. |
| 123 | +* Modify these functions to add/modify test for new/changed attributes. |
| 124 | + |
| 125 | +### Add new item to an existing property. |
| 126 | + |
| 127 | +For example add support for apply command to existing command support: |
| 128 | + |
| 129 | +1. In ```command-test-utils.go``` |
| 130 | + * add functions: |
| 131 | + * ```func setApplyCommandValues(applyCommand *schema.ApplyCommand)``` |
| 132 | + * randomly set attribute values in the provided apply command object |
| 133 | + * ```func createApplyCommand() *schema.ApplyCommand``` |
| 134 | + * creates the apply command object and calls setApplyCommandValues to add attribute values |
| 135 | + * follow the implementation of other similar functions. |
| 136 | + * modify: |
| 137 | + * ```func generateCommand(command *schema.Command, genericCommand *GenericCommand)``` |
| 138 | + * add logic to call createApplyCommand if commandType indicates such. |
| 139 | + * ```func (devfile *TestDevfile) UpdateCommand(command *schema.Command) error``` |
| 140 | + * add logic to call setApplyCommandValues if commandType indicates such. |
| 141 | +1. In ```parser_v200_verify_test.go``` |
| 142 | + * add new tests. for example: |
| 143 | + * Test_ApplyCommand - CreateWithParser set to false, EditContent set to false |
| 144 | + * Test_CreateApplyCommand - CreateWithParser set to true, EditContent set to false |
| 145 | + * Test_EditApplyCommand - CreateWithParser set to false, EditContent set to true |
| 146 | + * Test_CreateEditApplyCommand - CreateWithParser set to true, EditContent set to true |
| 147 | + * modify existing test to include Apply commands |
| 148 | + * Test_MultiCommand |
| 149 | + * Test_Everything |
| 150 | + |
| 151 | +### Add new property |
| 152 | + |
| 153 | +Using existing support for commands as an illustration, any new property support added should follow the same structure: |
| 154 | + |
| 155 | +1. ```command-test-utils.go```: |
| 156 | + * Specific to commands |
| 157 | + * Commands require support for 5 different command types: |
| 158 | + * Exec |
| 159 | + * Appy (to be implemented) |
| 160 | + * Composite |
| 161 | + * VSCodeLaunch (to be implemented) |
| 162 | + * VSCodeTask (to be implemented) |
| 163 | + * Each of these command-types have equivalent functions: |
| 164 | + * ```func create<command-type>Command() *schema.<command-type>``` |
| 165 | + * creates the command object and calls ```set<command-type>CommandValues``` to add attribute values |
| 166 | + * for example see: ```func createExecCommand(execCommand *schema.ExecCommand)``` |
| 167 | + * ```func set<command-type>CommandValues(project-sourceProject *schema.<project-source>)``` |
| 168 | + * sets random attributes into the provided object |
| 169 | + * for example see: ```func setExecCommandValues(execCommand *schema.ExecCommand)``` |
| 170 | + * Functions general to all commands |
| 171 | + * ```func generateCommand(command *schema.Command, genericCommand *GenericCommand)``` |
| 172 | + * includes logic to call the ```create<Command-Type>Command``` function for the command-Type of the supplied command object. |
| 173 | + * ```func (devfile *TestDevfile) addCommand(commandType schema.CommandType) string``` |
| 174 | + * main entry point for a test to add a command |
| 175 | + * maintains the array of commands in the schema structure |
| 176 | + * calls generateCommand() |
| 177 | + * ```func (devfile *TestDevfile) UpdateCommand(command *schema.Command) error``` |
| 178 | + * includes logic to call set<commad-type>CommandValues for each commandType. |
| 179 | + * ```func (devfile TestDevfile) VerifyCommands(parserCommands []schema.Command) error``` |
| 180 | + * includes logic to compare the array of commands obtained from the parser with those created by the test. if the compare fails: |
| 181 | + * each individual command is compared. |
| 182 | + * if a command compare fails, the parser version and test version of the command are oputput as yaml files to the tmp directory |
| 183 | + * a check is made to determine if the parser returned a command not known to the test or the pasrer omitted a command expected by the test. |
| 184 | +1. ```test-utils.go``` |
| 185 | + * ```func (devfile TestDevfile) Verify()``` |
| 186 | + * includes code to get object from the paser and verify their content. |
| 187 | + * for commands code is required to: |
| 188 | + 1. Retrieve each command from the parser |
| 189 | + 1. Use command Id to obtain the GenericCommand object which matches |
| 190 | + 1. Compare the command structure returned by the parser with the command structure saved in the GenericCommand object. |
| 191 | + * ```func (devfile TestDevfile) EditCommands() error``` |
| 192 | + * specific to command objects. |
| 193 | + 1. Ensure devfile is written to disk |
| 194 | + 1. Use parser to read devfile and get all command object |
| 195 | + 1. For each command call: |
| 196 | + * ```func (devfile *TestDevfile) UpdateCommand(command *schema.Command) error``` |
| 197 | + 1. When all commands have been updated, use parser to write the updated devfile to disk |
| 198 | +1. ```parser-v200-test.go``` |
| 199 | + * ```type TestContent struct``` |
| 200 | + * includes an array of command types: ```CommandTypes []schema.CommandType``` |
| 201 | + * ```func Test_ExecCommand(t *testing.T)``` |
| 202 | + 1. Creates a TestContent object |
| 203 | + 1. Adds a single entry array containg schema.ExecCommandType to the array of command types |
| 204 | + 1. Calls runTest for a single thread test |
| 205 | + 1. Calls runMultiThreadTest for a multi-thread test. |
| 206 | + * See also |
| 207 | + * ```func Test_<string>ExecCommand(t *testing.T)``` |
| 208 | + * ```func Test_MultiCommand(t *testing.T)``` |
| 209 | + * ```func Test_Everything(t *testing.T)``` |
| 210 | + * Add logic to ```func runTest(testContent TestContent, t *testing.T)``` |
| 211 | + 1. Add commands to the test. |
| 212 | + 2. Start edits of commands if required. |
| 213 | + |
| 214 | + |
| 215 | +#### Code flow |
| 216 | + |
| 217 | +Create, modify and verify an exec command: |
| 218 | +1. parser_v200_verify_test.Test_ExecCommand |
| 219 | + 1. parser-v200-test.runTest |
| 220 | + 1. command-test-utils.AddCommand |
| 221 | + 1. command-test-utils.GenerateCommand |
| 222 | + 1. command-test-utils.createExecCommand |
| 223 | + 1. command-test-utils.setExecCommandValues |
| 224 | + 1. test-utils.CreateDevfile |
| 225 | + 1. test-utils.EditCommands |
| 226 | + 1. command-test-utils.UpdateCommand |
| 227 | + 1. command-test-utils.setExecCommandValues |
| 228 | + 1. test-utils.Verify |
| 229 | + 1. command-test-utils.VerifyCommands |
| 230 | + |
| 231 | + |
| 232 | + |
| 233 | + |
| 234 | + |
| 235 | + |
0 commit comments