add proxy argument and improve dns cache usage (#2209)

* add proxy argument and improve dns cache usage

add proxy argument that useing the -x and --proxy argument. add it to
the static curl config struct, command usage and help outputs of the
cli.

parse these argument together with the environment variables like
http_proxy before setting the CURLOPT_PROXY in the curl configuration
option. this is required, as there is no easy way to ascertain/get what
the CURLOPT_PROXY that libcurl will use. by the point it is set by
libcurl, we have no control over it anymore, and need it for the other
steps in the configuration.

if the CURLOPT_PROXY is set, skip the DNS cache population which would
set the CURLOPT_RESOLVE. this is currently not perfect however. if a
proxy is set with socks4 or socks5 scheme, the host should be resolving
the hostname.

* codespell, clang-format and hints fixes

* add curl version and ssl enabelement macro checks

might fix rocky linux 8 compilation issues.

* add proxy_resolves_hostname, determined by proxy scheme

leave the functions that print out an curl_easyoption, but dont use it. organize the code slightly, print out the final CURLOPT_PROXY and proxy_resolves_hostname flag on verbose mode, add comments

* remove unused handle_curl_easyoption and format_curl_easyoption functions

* fix typo in the proxy argument

* fix typo with proxy scheme socks5a->socks5h

* improve proxy environment parsing

add another argument: --no-proxy , which is used when setting
CURL_NOPROXY

additionally parse all_proxy, ALL_PROXY, no_proxy and NO_PROXY
environment variables in the correct order.

set the curlopt_proxy and curlopt_noproxy of libcurl, and additionally
save them in check_curl_working_state.

add function determine_hostname_resolver, uses the working state and
static config. it can tokenize the no_proxy variable and check for exact
matches, but cannot determine subnet matches for ip addresses yet.

* document proxy cli arguments

clarify and add more examples of proxy environment variables and their
behavior when multiple are specified, overriden etc.

add single wildcard '*' checking for no_proxy to
determine_hostname_resolver, special case per curlopt_noproxy
documentation

* check curlopt_noproxy before accessing it

* switch argument from --no-proxy to --noproxy like curl cli

* check if host name is a subdomain of an noproxy item

* use strdup where destination working_state.curlopt_proxy may be NULL

* add disclaimer about uppercase HTTP_PROXY

* add subdomain checks for each item in the no_proxy, if the target host is a subdomain proxy wont resolve it

add function ip_addr_inside_cidr, use it for checking possible cidr ranges  given in the no_proxy

* wip tests that work on local perl http/https server

* wip tests that work on the live debian image

* fix subnet definition

* make apache2 listen on [::1] for ipv6 tests

* remove squid certificate

* rewrite ip_addr_inside_cidr, split ipv4 and ipv6 parsing path and copy them to a shared buffer later on for prefix check

* Adapt tests for the squid sever, disable checking return code for socks 4/5 proxies. Squid does not support it, and we do not install a capable proxy for these schemes.

* specify localhost acl and allow it through the proxy. used in check_curl tests

* typo in comment

* move function comments to header

* fix failing tests

* handle case where proxy is set as empty string

* removed duplicate tests, corrected wrong comments

* corrected some annotations

* move docker apache subdomain setup files to /tools/subdomain1

* add a newline before dying in handle_curl_option_return_code

* fix the -ssl better, now does not segfault on empty --ssl argument as well.


---------

Co-authored-by: Ahmet Oeztuerk <Ahmet.Oeztuerk@consol.de>

1 files changed, 132 insertions, 3 deletions

diff --git a/plugins/tests/check_curl.t b/plugins/tests/check_curl.t
index 248eb4c5..94058d5b 100755
--- a/plugins/tests/check_curl.t
+++ b/plugins/tests/check_curl.t
@@ -27,8 +27,8 @@ use HTTP::Daemon::SSL;
 
 $ENV{'LC_TIME'} = "C";
 
-my $common_tests = 95;
-my $ssl_only_tests = 8;
+my $common_tests = 111;
+my $ssl_only_tests = 12;
 # Check that all dependent modules are available
 eval "use HTTP::Daemon 6.01;";
 plan skip_all => 'HTTP::Daemon >= 6.01 required' if $@;
@@ -41,7 +41,7 @@ my $plugin = 'check_http';
 $plugin    = 'check_curl' if $0 =~ m/check_curl/mx;
 
 # look for libcurl version to see if some advanced checks are possible (>= 7.49.0)
-my $advanced_checks = 12;
+my $advanced_checks = 16;
 my $use_advanced_checks = 0;
 my $required_version = '7.49.0';
 my $virtual_host = 'www.somefunnyhost.com';
@@ -410,6 +410,41 @@ SKIP: {
         $result = NPTest->testCmd( $cmd );
         is( $result->return_code, 0, $cmd);
         like( $result->output, '/.*HTTP/1.1 200 OK - \d+ bytes in [\d\.]+ second.*/', "Output correct: ".$result->output );
+
+        # curlopt proxy/noproxy parsing tests, ssl disabled
+        {
+                # Make a scope and change environment variables here, to not mess them up for other tests using environment variables
+
+                local $ENV{"http_proxy"} = 'http://proxy.example.com:8080';
+                $cmd = "$command -u /statuscode/200 -v";
+                $result = NPTest->testCmd( $cmd );
+                like( $result->output, '/.*CURLOPT_PROXY: http://proxy.example.com:8080 */', "Correctly took 'http_proxy' environment variable: ".$result->output );
+                delete($ENV{"http_proxy"});
+
+                local $ENV{"http_proxy"} = 'http://taken.proxy.example:8080';
+                local $ENV{"HTTP_PROXY"} = 'http://discarded.proxy.example:8080';
+                $cmd = "$command -u /statuscode/200 -v";
+                $result = NPTest->testCmd( $cmd );
+                like( $result->output, '/.*CURLOPT_PROXY: http://taken.proxy.example:8080 */', "Correctly took 'http_proxy' environment variable over 'HTTP_PROXY': ".$result->output );
+                delete(local $ENV{"http_proxy"});
+                delete(local $ENV{"HTTP_PROXY"});
+
+                local $ENV{"http_proxy"} = 'http://discarded1.proxy.example:8080';
+                local $ENV{"HTTP_PROXY"} = 'http://discarded2.proxy.example:8080';
+                $cmd = "$command -u /statuscode/200 -x 'http://taken.proxy.example:8080' -v";
+                $result = NPTest->testCmd( $cmd );
+                like( $result->output, '/.*CURLOPT_PROXY: http://taken.proxy.example:8080 */', "Argument -x overwrote 'http_proxy' and 'HTTP_PROXY' environment variables: ".$result->output );
+                delete(local $ENV{"http_proxy"});
+                delete(local $ENV{"HTTP_PROXY"});
+
+                local $ENV{"http_proxy"} = 'http://discarded1.proxy.example:8080';
+                local $ENV{"HTTP_PROXY"} = 'http://discarded2.proxy.example:8080';
+                $cmd = "$command -u /statuscode/200 --proxy 'http://taken.example.com:8080' -v";
+                $result = NPTest->testCmd( $cmd );
+                like( $result->output, '/.*CURLOPT_PROXY: http://taken.example.com:8080 */', "Argument --proxy overwrote 'http_proxy' and 'HTTP_PROXY' environment variables: ".$result->output );
+                delete(local $ENV{"http_proxy"});
+                delete(local $ENV{"HTTP_PROXY"});
+        }
 }
 
 # and the same for SSL
@@ -432,6 +467,41 @@ SKIP: {
         $result = NPTest->testCmd( $cmd );
         is( $result->return_code, 0, $cmd);
         like( $result->output, '/.*HTTP/1.1 200 OK - \d+ bytes in [\d\.]+ second.*/', "Output correct: ".$result->output );
+
+        # curlopt proxy/noproxy parsing tests, ssl enabled
+        {
+                # Make a scope and change environment variables here, to not mess them up for other tests using environment variables
+
+                local $ENV{"https_proxy"} = 'http://proxy.example.com:8080';
+                $cmd = "$command -u /statuscode/200 --ssl -v";
+                $result = NPTest->testCmd( $cmd );
+                like( $result->output, '/.*CURLOPT_PROXY: http://proxy.example.com:8080 */', "Correctly took 'https_proxy' environment variable: ".$result->output );
+                delete($ENV{"https_proxy"});
+
+                local $ENV{"https_proxy"} = 'http://taken.proxy.example:8080';
+                local $ENV{"HTTPS_PROXY"} = 'http://discarded.proxy.example:8080';
+                $cmd = "$command -u /statuscode/200 --ssl -v";
+                $result = NPTest->testCmd( $cmd );
+                like( $result->output, '/.*CURLOPT_PROXY: http://taken.proxy.example:8080 */', "Correctly took 'https_proxy' environment variable over 'HTTPS_PROXY': ".$result->output );
+                delete(local $ENV{"https_proxy"});
+                delete(local $ENV{"HTTPS_PROXY"});
+
+                local $ENV{"https_proxy"} = 'http://discarded1.proxy.example:8080';
+                local $ENV{"HTTPS_PROXY"} = 'http://discarded2.proxy.example:8080';
+                $cmd = "$command -u /statuscode/200 --ssl -x 'http://taken.example.com:8080' -v";
+                $result = NPTest->testCmd( $cmd );
+                like( $result->output, '/.*CURLOPT_PROXY: http://taken.example.com:8080 */', "Argument -x overwrote environment variables 'https_proxy' and 'HTTPS_PROXY': ".$result->output );
+                delete(local $ENV{"https_proxy"});
+                delete(local $ENV{"HTTPS_PROXY"});
+
+                local $ENV{"https_proxy"} = 'http://discarded1.proxy.example:8080';
+                local $ENV{"HTTPS_PROXY"} = 'http://discarded2.proxy.example:8080';
+                $cmd = "$command -u /statuscode/200 --ssl --proxy 'http://taken.example.com:8080' -v";
+                $result = NPTest->testCmd( $cmd );
+                like( $result->output, '/.*CURLOPT_PROXY: http://taken.example.com:8080 */', "Argument --proxy overwrote environment variables 'https_proxy' and 'HTTPS_PROXY': ".$result->output );
+                delete(local $ENV{"https_proxy"});
+                delete(local $ENV{"HTTPS_PROXY"});
+        }
 }
 
 
@@ -712,4 +782,63 @@ sub run_common_tests {
                 $result = NPTest->testCmd( $cmd, 5 );
         };
         is( $@, "", $cmd );
+
+        # curlopt proxy/noproxy parsing tests
+        {
+                # Make a scope and change environment variables here, to not mess them up for other tests using environment variables
+
+                local $ENV{"no_proxy"} = 'internal.acme.org';
+                $cmd = "$command -u /statuscode/200 -v";
+                $result = NPTest->testCmd( $cmd );
+                like( $result->output, '/.* curl CURLOPT_NOPROXY: internal.acme.org */', "Correctly took 'no_proxy' environment variable: ".$result->output );
+                delete($ENV{"no_proxy"});
+
+                local $ENV{"no_proxy"} = 'taken.acme.org';
+                local $ENV{"NO_PROXY"} = 'discarded.acme.org';
+                $cmd = "$command -u /statuscode/200 -v";
+                $result = NPTest->testCmd( $cmd );
+                is( $result->return_code, 0, $cmd);
+                like( $result->output, '/.*CURLOPT_NOPROXY: taken.acme.org*/', "Correctly took 'no_proxy' environment variable over 'NO_PROXY': ".$result->output );
+                delete(local $ENV{"no_proxy"});
+                delete(local $ENV{"NO_PROXY"});
+
+                local $ENV{"no_proxy"} = 'taken.acme.org';
+                local $ENV{"NO_PROXY"} = 'discarded.acme.org';
+                $cmd = "$command -u /statuscode/200 --noproxy 'taken.acme.org' -v";
+                $result = NPTest->testCmd( $cmd );
+                is( $result->return_code, 0, $cmd);
+                like( $result->output, '/.*CURLOPT_NOPROXY: taken.acme.org*/', "Argument --noproxy overwrote environment variables 'no_proxy' and 'NO_PROXY': ".$result->output );
+                delete(local $ENV{"no_proxy"});
+                delete(local $ENV{"NO_PROXY"});
+
+                $cmd = "$command -u /statuscode/200 --noproxy 'internal1.acme.org,internal2.acme.org,internal3.acme.org' -v";
+                $result = NPTest->testCmd( $cmd );
+                is( $result->return_code, 0, $cmd);
+                like( $result->output, '/.*CURLOPT_NOPROXY: internal1.acme.org,internal2.acme.org,internal3.acme.org*/', "Argument --noproxy read multiple noproxy domains: ".$result->output );
+
+                $cmd = "$command -u /statuscode/200 --noproxy '10.11.12.13,256.256.256.256,0.0.0.0,192.156.0.0/22,10.0.0.0/4' -v";
+                $result = NPTest->testCmd( $cmd );
+                is( $result->return_code, 0, $cmd);
+                like( $result->output, '/.*CURLOPT_NOPROXY: 10.11.12.13,256.256.256.256,0.0.0.0,192.156.0.0/22,10.0.0.0/4*/', "Argument --noproxy took multiple noproxy domains: ".$result->output );
+
+                $cmd = "$command -u /statuscode/200 --noproxy '0123:4567:89AB:CDEF:0123:4567:89AB:CDEF,0123::CDEF,0123:4567/96,[::1],::1,[1234::5678:ABCD/4]' -v";
+                $result = NPTest->testCmd( $cmd );
+                is( $result->return_code, 0, $cmd);
+                like( $result->output, '/.*CURLOPT_NOPROXY: 0123:4567:89AB:CDEF:0123:4567:89AB:CDEF,0123::CDEF,0123:4567\/96,\[::1\],::1,\[1234::5678:ABCD\/4\].*/', "Argument --noproxy took multiple noproxy domains: ".$result->output );
+
+                $cmd = "$command -u /statuscode/200 --noproxy '300.400.500.600,1.2.3,XYZD:0123::,1:2:3:4:5:6:7,1::2::3,1.1.1.1/64,::/256' -v";
+                $result = NPTest->testCmd( $cmd );
+                is( $result->return_code, 0, $cmd);
+
+                $cmd = "$command -u /statuscode/200 --proxy http://proxy.example.com:8080 --noproxy '*' -v";
+                $result = NPTest->testCmd( $cmd );
+                is( $result->return_code, 0, $cmd);
+                like( $result->output, '/.*proxy_resolves_hostname: 0.*/', "Proxy will not be used due to '*' in noproxy: ".$result->output );
+
+                $cmd = "$command -u /statuscode/200 --proxy http://proxy.example.com:8080 --noproxy '127.0.0.1' -v";
+                $result = NPTest->testCmd( $cmd );
+                is( $result->return_code, 0, $cmd);
+                like( $result->output, '/.*proxy_resolves_hostname: 0.*/', "Proxy will not be used due to '127.0.0.1' in noproxy: ".$result->output );
+        }
+
 }