xref: /curl/tests/CI.md (revision e53523fe)
1<!--
2Copyright (C) Daniel Stenberg, <daniel@haxx.se>, et al.
3
4SPDX-License-Identifier: curl
5-->
6
7# Continuous Integration for curl
8
9Curl runs in many different environments, so every change is run against a
10large number of test suites.
11
12Every pull request is verified for each of the following:
13
14 - ... it still builds, warning-free, on Linux and macOS, with both
15   clang and gcc
16 - ... it still builds fine on Windows with several MSVC versions
17 - ... it still builds with cmake on Linux, with gcc and clang
18 - ... it follows rudimentary code style rules
19 - ... the test suite still runs 100% fine
20 - ... the release tarball (the "dist") still works
21 - ... it builds fine in-tree as well as out-of-tree
22 - ... code coverage does not shrink drastically
23 - ... different TLS backends still compile and pass tests
24
25If the pull-request fails one of these tests, it shows up as a red X and you
26are expected to fix the problem. If you do not understand when the issue is or
27have other problems to fix the complaint, just ask and other project members
28can likely help out.
29
30Consider the following table while looking at pull request failures:
31
32 | CI platform as shown in PR          | State  | What to look at next       |
33 | ----------------------------------- | ------ | -------------------------- |
34 | CI / codeql                         | stable | quality check results      |
35 | CI / fuzzing                        | stable | fuzzing results            |
36 | CI / macos ...                      | stable | all errors and failures    |
37 | Code scanning results / CodeQL      | stable | quality check results      |
38 | FreeBSD FreeBSD: ...                | stable | all errors and failures    |
39 | LGTM analysis: Python               | stable | new findings               |
40 | LGTM analysis:  C/C++               | stable | new findings               |
41 | buildbot/curl_winssl_ ...           | stable | all errors and failures    |
42 | AppVeyor                            | flaky  | all errors and failures    |
43 | curl.curl (linux ...)               | stable | all errors and failures    |
44 | curl.curl (windows ...)             | flaky  | repetitive errors/failures |
45 | CodeQL                              | stable | new findings               |
46
47Sometimes the tests fail due to a dependency service temporarily being offline
48or otherwise unavailable, for example package downloads. In this case you can
49just try to update your pull requests to rerun the tests later as described
50below.
51
52## CI servers
53
54Here are the different CI environments that are currently in use, and how they
55are configured:
56
57### GitHub Actions
58
59GitHub Actions runs the following tests:
60
61- macOS tests with a variety of different compilation options
62- Fuzz tests ([see the curl-fuzzer repo for more
63  info](https://github.com/curl/curl-fuzzer)).
64- Curl compiled using the Rust TLS backend with Hyper
65- CodeQL static analysis
66
67These are each configured in different files in `.github/workflows`.
68
69### Azure
70
71Not used anymore.
72
73### AppVeyor
74
75AppVeyor runs a variety of different Windows builds, with different compilation
76options.
77
78As of November 2021 `@bagder`, `@mback2k`, `@jay`, `@vszakats`, `@dfandrich`
79and `@danielgustafsson` have administrator access to the AppVeyor CI
80environment.  Additional admins/group members can be added on request.
81
82The tests are configured in `appveyor.yml`.
83
84### Zuul
85
86Not used anymore.
87
88### Circle CI
89
90Circle CI runs a basic Linux test suite on Ubuntu for both x86 and ARM
91processors. This is configured in `.circleci/config.yml`.
92
93You can [view the full list of CI jobs on Circle CI's
94website](https://app.circleci.com/pipelines/github/curl/curl).
95
96`@bagder` has access to edit the "Project Settings" on that page. Additional
97admins/group members can be added on request.
98
99### Cirrus CI
100
101Not used anymore.
102