summaryrefslogtreecommitdiffstats
path: root/tap/tap.3
diff options
context:
space:
mode:
Diffstat (limited to 'tap/tap.3')
-rw-r--r--tap/tap.3368
1 files changed, 368 insertions, 0 deletions
diff --git a/tap/tap.3 b/tap/tap.3
new file mode 100644
index 0000000..4b23c24
--- /dev/null
+++ b/tap/tap.3
@@ -0,0 +1,368 @@
1.Dd December 20, 2004
2.Os
3.Dt TAP 3
4.Sh NAME
5.Nm tap
6.Nd write tests that implement the Test Anything Protocol
7.Sh SYNOPSIS
8.In tap.h
9.Sh DESCRIPTION
10The
11.Nm
12library provides functions for writing test scripts that produce output
13consistent with the Test Anything Protocol. A test harness that parses
14this protocol can run these tests and produce useful reports indicating
15their success or failure.
16.Ss PRINTF STRINGS
17In the descriptions that follow, for any function that takes as the
18last two parameters
19.Dq Fa char * , Fa ...
20it can be assumed that the
21.Fa char *
22is a
23.Fn printf
24-like format string, and the optional arguments are values to be placed
25in that string.
26.Ss TEST PLANS
27.Bl -tag -width indent
28.It Xo
29.Ft int
30.Fn plan_tests "unsigned int"
31.Xc
32.It Xo
33.Ft int
34.Fn plan_no_plan "void"
35.Xc
36.It Xo
37.Ft int
38.Fn plan_skip_all "char *" "..."
39.Xc
40.El
41.Pp
42You must first specify a test plan. This indicates how many tests you
43intend to run, and allows the test harness to notice if any tests were
44missed, or if the test program exited prematurely.
45.Pp
46To do this, use
47.Fn plan_tests ,
48which always returns 0. The function will cause your program to exit
49prematurely if you specify 0 tests.
50.Pp
51In some situations you may not know how many tests you will be running, or
52you are developing your test program, and do not want to update the
53.Fn plan_tests
54parameter every time you make a change. For those situations use
55.Fn plan_no_plan .
56It returns 0, and indicates to the test harness that an indeterminate number
57of tests will be run.
58.Pp
59Both
60.Fn plan_tests
61and
62.Fn plan_no_plan
63will cause your test program to exit prematurely with a diagnostic
64message if they are called more than once.
65.Pp
66If your test program detects at run time that some required functionality
67is missing (for example, it relies on a database connection which is not
68present, or a particular configuration option that has not been included
69in the running kernel) use
70.Fn plan_skip_all ,
71passing as parameters a string to display indicating the reason for skipping
72the tests.
73.Ss SIMPLE TESTS
74.Bl -tag -width indent
75.It Xo
76.Ft unsigned int
77.Fn ok "expression" "char *" "..."
78.Xc
79.It Xo
80.Ft unsigned int
81.Fn ok1 "expression"
82.Xc
83.It Xo
84.Ft unsigned int
85.Fn pass "char *" "..."
86.Xc
87.It Xo
88.Ft unsigned int
89.Fn fail "char *" "..."
90.Xc
91.El
92.Pp
93Tests are implemented as expressions checked by calls to the
94.Fn ok
95and
96.Fn ok1
97macros. In both cases
98.Fa expression
99should evaluate to true if the test succeeded.
100.Pp
101.Fn ok
102allows you to specify a name, or comment, describing the test which will
103be included in the output.
104.Fn ok1
105is for those times when the expression to be tested is self
106explanatory and does not need an associated comment. In those cases
107the test expression becomes the comment.
108.Pp
109These four calls are equivalent:
110.Bd -literal -offset indent
111int i = 5;
112
113ok(i == 5, "i equals 5"); /* Overly verbose */
114ok(i == 5, "i equals %d", i); /* Just to demonstrate printf-like
115 behaviour of the test name */
116ok(i == 5, "i == 5"); /* Needless repetition */
117ok1(i == 5); /* Just right */
118.Ed
119.Pp
120It is good practice to ensure that the test name describes the meaning
121behind the test rather than what you are testing. Viz
122.Bd -literal -offset indent
123ok(db != NULL, "db is not NULL"); /* Not bad, but */
124ok(db != NULL, "Database conn. succeeded"); /* this is better */
125.Ed
126.Pp
127.Fn ok
128and
129.Fn ok1
130return 1 if the expression evaluated to true, and 0 if it evaluated to
131false. This lets you chain calls from
132.Fn ok
133to
134.Fn diag
135to only produce diagnostic output if the test failed. For example, this
136code will include diagnostic information about why the database connection
137failed, but only if the test failed.
138.Bd -literal -offset indent
139ok(db != NULL, "Database conn. succeeded") ||
140 diag("Database error code: %d", dberrno);
141.Ed
142.Pp
143You also have
144.Fn pass
145and
146.Fn fail .
147From the Test::More documentation:
148.Bd -literal -offset indent
149Sometimes you just want to say that the tests have passed.
150Usually the case is you've got some complicated condition
151that is difficult to wedge into an ok(). In this case,
152you can simply use pass() (to declare the test ok) or fail
153(for not ok).
154
155Use these very, very, very sparingly.
156.Ed
157.Pp
158These are synonyms for ok(1, ...) and ok(0, ...).
159.Ss SKIPPING TESTS
160.Bl -tag -width indent
161.It Xo
162.Ft int
163.Fn skip "unsigned int" "char *" "..."
164.Xc
165.It Xo
166.Fn skip_start "expression" "unsigned int" "char *" "..."
167.Xc
168.It Xo
169.Sy skip_end
170.Xc
171.El
172.Pp
173Sets of tests can be skipped. Ordinarily you would do this because
174the test can't be run in this particular testing environment.
175.Pp
176For example, suppose some tests should be run as root. If the test is
177not being run as root then the tests should be skipped. In this
178implementation, skipped tests are flagged as being ok, with a special
179message indicating that they were skipped. It is your responsibility
180to ensure that the number of tests skipped (the first parameter to
181.Fn skip )
182is correct for the number of tests to skip.
183.Pp
184One way of implementing this is with a
185.Dq do { } while(0);
186loop, or an
187.Dq if( ) { } else { }
188construct, to ensure that there are no additional side effects from the
189skipped tests.
190.Bd -literal -offset indent
191if(getuid() != 0) {
192 skip(1, "because test only works as root");
193} else {
194 ok(do_something_as_root() == 0, "Did something as root");
195}
196.Ed
197.Pp
198Two macros are provided to assist with this. The previous example could
199be re-written as follows.
200.Bd -literal -offset indent
201skip_start(getuid() != 0, 1, "because test only works as root");
202
203ok(do_something_as_root() == 0, "Did something as root");
204
205skip_end; /* It's a macro, no parentheses */
206.Ed
207.Ss MARKING TESTS AS Dq TODO
208.Bl -tag -width indent
209.It Xo
210.Ft void
211.Fn todo_start "char *" "..."
212.Xc
213.It Xo
214.Ft void
215.Fn todo_end "void"
216.Xc
217.El
218.Pp
219Sets of tests can be flagged as being
220.Dq TODO .
221These are tests that you expect to fail, probably because you haven't
222fixed a bug, or finished a new feature yet. These tests will still be
223run, but with additional output that indicates that they are expected
224to fail. Should a test start to succeed unexpectedly, tools like
225.Xr prove 1
226will indicate this, and you can move the test out of the todo
227block. This is much more useful than simply commenting out (or
228.Dq #ifdef 0 ... #endif )
229the tests.
230.Bd -literal -offset indent
231todo_start("dwim() not returning true yet");
232
233ok(dwim(), "Did what the user wanted");
234
235todo_end();
236.Ed
237.Pp
238Should
239.Fn dwim
240ever start succeeding you will know about it as soon as you run the
241tests. Note that
242.Em unlike
243the
244.Fn skip_*
245family, additional code between
246.Fn todo_start
247and
248.Fn todo_end
249.Em is
250executed.
251.Ss SKIP vs. TODO
252From the Test::More documentation;
253.Bd -literal -offset indent
254If it's something the user might not be able to do, use SKIP.
255This includes optional modules that aren't installed, running
256under an OS that doesn't have some feature (like fork() or
257symlinks), or maybe you need an Internet connection and one
258isn't available.
259
260If it's something the programmer hasn't done yet, use TODO.
261This is for any code you haven't written yet, or bugs you have
262yet to fix, but want to put tests in your testing script
263(always a good idea).
264.Ed
265.Ss DIAGNOSTIC OUTPUT
266.Bl -tag -width indent
267.It Xo
268.Fr unsigned int
269.Fn diag "char *" "..."
270.Xc
271.El
272.Pp
273If your tests need to produce diagnostic output, use
274.Fn diag .
275It ensures that the output will not be considered by the TAP test harness.
276.Fn diag
277adds the necessary trailing
278.Dq \en
279for you.
280.Bd -literal -offset indent
281diag("Expected return code 0, got return code %d", rcode);
282.Ed
283.Pp
284.Fn diag
285always returns 0.
286.Ss EXIT STATUS
287.Bl -tag -width indent
288.It Xo
289.Fr int
290.Fn exit_status void
291.Xc
292.El
293.Pp
294For maximum compatability your test program should return a particular
295exit code. This is calculated by
296.Fn exit_status
297so it is sufficient to always return from
298.Fn main
299with either
300.Dq return exit_status();
301or
302.Dq exit(exit_status());
303as appropriate.
304.Sh EXAMPLES
305The
306.Pa tests
307directory in the source distribution contains numerous tests of
308.Nm
309functionality, written using
310.Nm .
311Examine them for examples of how to construct test suites.
312.Sh COMPATABILITY
313.Nm
314strives to be compatible with the Perl Test::More and Test::Harness
315modules. The test suite verifies that
316.Nm
317is bug-for-bug compatible with their behaviour. This is why some
318functions which would more naturally return nothing return constant
319values.
320.Pp
321If the
322.Lb libpthread
323is found at compile time,
324.Nm
325.Em should
326be thread safe. Indications to the contrary (and test cases that expose
327incorrect behaviour) are very welcome.
328.Sh SEE ALSO
329.Xr Test::More 1 ,
330.Xr Test::Harness 1 ,
331.Xr prove 1
332.Sh STANDARDS
333.Nm
334requires a
335.St -isoC-99
336compiler. Some of the
337.Nm
338functionality is implemented as variadic macros, and that functionality
339was not formally codified until C99. Patches to use
340.Nm
341with earlier compilers that have their own implementation of variadic
342macros will be gratefully received.
343.Sh HISTORY
344.Nm
345was written to help improve the quality and coverage of the FreeBSD
346regression test suite, and released in the hope that others find it
347a useful tool to help improve the quality of their code.
348.Sh AUTHORS
349.An "Nik Clayton" Aq nik@ngo.org.uk ,
350.Aq nik@FreeBSD.org
351.Pp
352.Nm
353would not exist without the efforts of
354.An "Michael G Schwern" Aq schqern@pobox.com ,
355.An "Andy Lester" Aq andy@petdance.com ,
356and the countless others who have worked on the Perl QA programme.
357.Sh BUGS
358Ideally, running the tests would have no side effects on the behaviour
359of the application you are testing. However, it is not always possible
360to avoid them. The following side effects of using
361.Nm
362are known.
363.Bl -bullet -offset indent
364.It
365stdout is set to unbuffered mode after calling any of the
366.Fn plan_*
367functions.
368.El