Lecture 7A - Syntax Based Testing

Question 1

What is the general understanding of modelling and testing?

Answer 1

In brief, we come up with tests by looking at requirements and specifications, and thinking about the system – modelling it – in different ways.

Question 2

What are the 4 main models for testing?

Answer 2

As covered in the lectures:
- Input Space Partitioning (Equivalence classes of inputs)
- Logic-based Testing (Look inside the parts of boolean expressions making up a decision point)
- Graph-based Testing (on Program Control Flows)
Syntax-based Testing

DONT REALLY NEED TO KNOW, THIS IS MORE A CHECKLIST OF WHAT WE HAVE COVERED

Question 3

What is Syntax?

Answer 3

Rules and regulations for writing an artefact correctly

Question 4

What is Semantics?

Answer 4

Its meaning and behaviour, dictating what a piece of code does when executed, rather than just how it’s written

Question 5

What sort of things can be tested with syntax-based test methods

Answer 5

Anything thing that can be modelled by a syntactic description such as a grammar:
- Input commands eg command line applications (aws, docker, etc)
- Syntax of types eg email addresses, datetime format, URLs, passwords etc
- Data and file formats eg XML
Computer Program Syntax eg Java or Python programs etc

Question 6

Understanding Grammar

Answer 6

REFER TO SLIDES FOR EXAMPLE

Question 7

Where can we get syntactic descriptions from?

Answer 7

Syntactic descriptions can be obtained from many sources:
- program source code
- design documents
- input descriptions (e.g. file formats, network message formats, etc)

Question 8

What are the two ways we can use grammars?

Answer 8

Grammars can be used to build recognizers (programs which decide whether a string is in the grammar – i.e., parsers) and also generators, which produce strings of symbols.
In other words as recognisers or generators

Question 9

When to Use Grammars to Generate Tests

Answer 9

Need to determine when the best time is to apply the tests

Question 10

What is the theory of grammar?

Answer 10

Grammars just give us a way of formally specifying what things are and are not syntactically correct
- Every grammar defines what is called a language

Question 11

What is a language?

Answer 11

A language is a set of acceptable strings
REFER TO SLIDES FOR EXAMPLE

Question 12

What is BNF (Backus-Naur Form)?

Answer 12

REFER TO SLIDES FOR EXAMPLE

Question 13

What is the notation of grammar - terminal symbols?

Answer 13

The things in quotes are called terminal symbols – they are the equivalent of “words” in our language.
- They are like atoms, in that they are the smallest, indivisible parts of our language.
REFER TO SLIDES

Question 14

What is the notation of grammar - non-terminal symbols?

Answer 14

The things between angle brackets are called non-terminal symbols
- The above grammar contains five rules (also called productions)
- In the sorts of grammar we will consider, every rule is of the form: non-terminal “::=” sequence of terminals and non-terminals
REFER TO SLIDES

Question 15

What is the general structure of grammar?

Answer 15

When we specify a grammar, there will normally be a start symbol representing the “top level” of whatever construct we’re specifying.
REFER TO SLIDES

Question 16

What are other symbols we can add?

Answer 16

We can also insert on the RHS the following symbols, between or after terminals and non-terminals:
- bars to indicate “or” (alternatives)
- an asterisk (called the “Kleene star”) to indicate “zero or more of the preceding thing”
- a plus sign to indicate “one or more of the preceding thing”
- a range of numbers (e.g. “3–4”) to indicate a number of possible instances of the preceding thing.
REFER TO SLIDES

Question 17

What is BNF Coverage Criteria?

Answer 17

Once we have written a grammar to model our system
- If we’re developing tests based on syntax . . .
- The most straightforward coverage criterion is: use every terminal and every production rule at least once
- This includes:
○ Terminal Symbol coverage
○ Production coverage
○ Derivation coverage
But except in special cases, this will be impractical
REFER TO SLIDES FOR EXAMPLES

Question 18

What is Terminal Symbol coverage (TSC)?

Answer 18

Test requirements contain each terminal symbol t in the grammar G.
REFER TO SLIDES FOR EXAMPLES

Question 19

What is Production coverage?

Answer 19

Test requirements contain each production p in the grammar G.
- Production coverage subsumes terminal symbol coverage; if we’ve used every production, we’ve also used every terminal.
- (Since every terminal must be part of some production.)
REFER TO SLIDES FOR EXAMPLES

Question 20

What is Derivation coverage?

Answer 20

Test requirements contain every possible string that can be derived from grammar G.
REFER TO SLIDES FOR EXAMPLES

Question 21

Why do we need syntax based testing?

Answer 21

Much of the software we rely on makes use of grammars (though not always explicitly).
- For very simple programs, we might analyse the arguments “by hand”.
- For complex programs – we typically use a command-line argument parser to work out whether a user has supplied a valid set of arguments (and what we should do with them).
Whenever we validate entries into web forms or databases, we are often are defining a syntax to do so.

Question 22

Application areas for syntax based testing - What are command lines?

Answer 22

Command-line programs often take arguments – sometimes adhering to very complex rules
- The syntax of command line apps can be defined by a grammar

Question 23

Application areas for syntax based testing - What is Input Syntax?

Answer 23

Grammars are used to define whether something is a valid
- phone number
- postcode
- URL
- HTML page
- email address
- and many other formats.

Question 24

Application areas for syntax based testing - What are domain specific languages?

Answer 24

Often, grammars will be useful to define what are called “domain-specific languages” (DSLs) which describe entities in a domain and things to do with them – e.g. Makefiles are an example of this.
- Syntaxes are typically used to define such languages.

Question 25

Application areas for syntax based testing - What is the Syntactically well-formed Java class?

Answer 25

Syntax formed for the use in Java REFER TO SLIDES FOR EXAMPLES

Question 26

Application areas for syntax based testing - What are generators - network traffic?

Answer 26

Can be used in network testing

Question 27

What is an example of network traffic generator?

Answer 27

For example: - Being able to generate things that follow a syntax-like structure is extremely useful for testing. ○ We can use BNF to create traffic generators, for instance – we could generate random valid TCP traffic with which to test a router. ○ TCP packets follow a syntax-like structure, so it’s fairly straightforward to generate them randomly. ○ A TCP packet consists of: 2 bytes representing a source port (0 through 65535), 2 bytes representing a destination port, then 4 bytes representing a “sequence number”, then . . . (see the TCP specification for detailed rules). - BUT: ○ Not all the validity rules for a TCP packet can be expressed in a syntactic way – for instance, it contains a checksum towards the end, which is calculated based on previous information – but quite a bit can. ○ This is very handy for “stress” or “load” or “performance” testing – generating large amounts of data, and seeing how our system performs under the load.

Question 28

Application areas for syntax based testing - What are generators - HTTP traffic?

Answer 28

Can be used when creating web apps For example: - HTTP requests for web pages also follow a syntax, so we could easily generate random HTTP traffic (for instance, to stress-test a web-server, and see how it performs under high load). REFER TO SLIDES

Question 29

Application areas for syntax based testing - What are File Generators?

Answer 29

Likewise, HTML and XML documents, JSON, and many other formats all follow syntactical rules, so we can randomly generate them. Likewise for custom formats we may come up with. - e.g. If we were writing a word processor, we might want to be able generate very large random documents in our word-processor format, to see how our program holds up. REFER TO SLIDES

Question 30

What are some common tools for file generators?

Answer 30

For common formats, there are often already data generators with many capabilities: - Tools for constructing and generating network traffic: Ostinato, Scapy Traffic Generator, flowgrind, jtg . . . see this list for many more. - HTTP request generators: see for example httperf - Random bitmap generators: see for example random.org If not, it is perfectly possible to write our own.

Question 31

What is context free languages?

Answer 31

A context-free grammar (CFG) is one in which each production rule can be applied regardless of the surrounding symbols. It only looks at one nonterminal at a time and replaces it with some sequence of terminals/nonterminals. Allowed: “A student number consists of NNNN-NNN-NN” (no context needed). Not allowed: “If the first digit is 1, then the second must be ≤2” (needs context of the previous digit’s value).

Question 32

What is the motivation for mutating inputs?

Answer 32

It is quite common to require a program to reject malformed inputs, and this property should definitely be tested as a form of stress testing. - Malformed inputs may slip past the attention of programmers who are focused on happy path - From a practical perspective, invalid inputs sometimes matter a great deal because they hold the key to unintended functionality. For example, unhandled invalid inputs often represent security vulnerabilities

Question 33

What are mutating operators?

Answer 33

When mutating grammars, the mutants are the tests and we create valid and invalid strings REFER TO SLIDES

Question 34

What are the mutate grammar rules?

Answer 34

How to mutate grammar rules: (see A and O for details) - Replace term/nonterm - Delete term/nonterm - Duplicate term/nonterm REFER TO SLIDES