Module: Utils::Curl Private
- Included in:
- Archive, Bintray, CurlDownloadStrategy, GitHubReleases, SPDX, SPDX, SharedAudits, SharedAudits
- Defined in:
- utils/curl.rb
Overview
This module is part of a private API. This module may only be used in the Homebrew/brew repository. Third parties should avoid using this module if possible, as it may be removed or changed without warning.
Helper function for interacting with curl
.
Class Method Summary collapse
-
.curl(*args, print_stdout: true, **options) ⇒ Object
private
-
.curl_args(*extra_args, **options) ⇒ Object
private
-
.curl_check_http_content(url, specs: {}, user_agents: [:default], check_content: false, strict: false) ⇒ Object
private
-
.curl_download(*args, to: nil, partial: true, **options) ⇒ Object
private
-
.curl_executable ⇒ Object
private
-
.curl_http_content_headers_and_checksum(url, specs: {}, hash_needed: false, user_agent: :default) ⇒ Object
private
-
.curl_output(*args, **options) ⇒ Object
private
-
.curl_with_workarounds(*args, secrets: nil, print_stdout: nil, print_stderr: nil, debug: nil, verbose: nil, env: {}, timeout: nil, **options) ⇒ Object
private
-
.http_status_ok?(status) ⇒ Boolean
private
-
.url_protected_by_cloudflare?(details) ⇒ Boolean
private
Check if a URL is protected by CloudFlare (e.g. badlion.net and jaxx.io).
-
.url_protected_by_incapsula?(details) ⇒ Boolean
private
Check if a URL is protected by Incapsula (e.g. corsair.com).
Class Method Details
.curl(*args, print_stdout: true, **options) ⇒ Object
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
115 116 117 118 119 |
# File 'utils/curl.rb', line 115 def curl(*args, print_stdout: true, **) result = curl_with_workarounds(*args, print_stdout: print_stdout, **) result.assert_success! result end |
.curl_args(*extra_args, **options) ⇒ Object
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 |
# File 'utils/curl.rb', line 28 def curl_args(*extra_args, **) args = [] # do not load .curlrc unless requested (must be the first argument) args << "--disable" unless Homebrew::EnvConfig.curlrc? args << "--globoff" args << "--show-error" args << "--user-agent" << case [:user_agent] when :browser, :fake HOMEBREW_USER_AGENT_FAKE_SAFARI when :default, nil HOMEBREW_USER_AGENT_CURL when String [:user_agent] end args << "--header" << "Accept-Language: en" unless [:show_output] == true args << "--fail" args << "--progress-bar" unless Context.current.verbose? args << "--verbose" if Homebrew::EnvConfig.curl_verbose? args << "--silent" unless $stdout.tty? end args << "--connect-timeout" << connect_timeout.round(3) if [:connect_timeout] args << "--max-time" << max_time.round(3) if [:max_time] args << "--retry" << Homebrew::EnvConfig.curl_retries unless [:retry] == false args << "--retry-max-time" << retry_max_time.round if [:retry_max_time] args + extra_args end |
.curl_check_http_content(url, specs: {}, user_agents: [:default], check_content: false, strict: false) ⇒ Object
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 |
# File 'utils/curl.rb', line 171 def curl_check_http_content(url, specs: {}, user_agents: [:default], check_content: false, strict: false) return unless url.start_with? "http" secure_url = url.sub(/\Ahttp:/, "https:") secure_details = nil hash_needed = false if url != secure_url user_agents.each do |user_agent| secure_details = begin curl_http_content_headers_and_checksum(secure_url, specs: specs, hash_needed: true, user_agent: user_agent) rescue Timeout::Error next end next unless http_status_ok?(secure_details[:status]) hash_needed = true user_agents = [user_agent] break end end details = nil user_agents.each do |user_agent| details = curl_http_content_headers_and_checksum(url, specs: specs, hash_needed: hash_needed, user_agent: user_agent) break if http_status_ok?(details[:status]) end unless details[:status] # Hack around https://github.com/Homebrew/brew/issues/3199 return if MacOS.version == :el_capitan return "The URL #{url} is not reachable" end unless http_status_ok?(details[:status]) return if url_protected_by_cloudflare?(details) || url_protected_by_incapsula?(details) return "The URL #{url} is not reachable (HTTP status code #{details[:status]})" end if url.start_with?("https://") && Homebrew::EnvConfig.no_insecure_redirect? && !details[:final_url].start_with?("https://") return "The URL #{url} redirects back to HTTP" end return unless secure_details return if !http_status_ok?(details[:status]) || !http_status_ok?(secure_details[:status]) etag_match = details[:etag] && details[:etag] == secure_details[:etag] content_length_match = details[:content_length] && details[:content_length] == secure_details[:content_length] file_match = details[:file_hash] == secure_details[:file_hash] if (etag_match || content_length_match || file_match) && secure_details[:final_url].start_with?("https://") && url.start_with?("http://") return "The URL #{url} should use HTTPS rather than HTTP" end return unless check_content no_protocol_file_contents = %r{https?:\\?/\\?/} http_content = details[:file]&.gsub(no_protocol_file_contents, "/") https_content = secure_details[:file]&.gsub(no_protocol_file_contents, "/") # Check for the same content after removing all protocols if (http_content && https_content) && (http_content == https_content) && url.start_with?("http://") && secure_details[:final_url].start_with?("https://") return "The URL #{url} should use HTTPS rather than HTTP" end return unless strict # Same size, different content after normalization # (typical causes: Generated ID, Timestamp, Unix time) if http_content.length == https_content.length return "The URL #{url} may be able to use HTTPS rather than HTTP. Please verify it in a browser." end lenratio = (100 * https_content.length / http_content.length).to_i return unless (90..110).cover?(lenratio) "The URL #{url} may be able to use HTTPS rather than HTTP. Please verify it in a browser." end |
.curl_download(*args, to: nil, partial: true, **options) ⇒ Object
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
121 122 123 124 125 126 127 128 129 130 131 132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 |
# File 'utils/curl.rb', line 121 def curl_download(*args, to: nil, partial: true, **) destination = Pathname(to) destination.dirname.mkpath if partial range_stdout = curl_output("--location", "--range", "0-1", "--dump-header", "-", "--write-out", "%\{http_code}", "--output", "/dev/null", *args, **).stdout headers, _, http_status = range_stdout.partition("\r\n\r\n") supports_partial_download = http_status.to_i == 206 # Partial Content if supports_partial_download && destination.exist? && destination.size == %r{^.*Content-Range: bytes \d+-\d+/(\d+)\r\n.*$}m.match(headers)&.[](1)&.to_i return # We've already downloaded all the bytes end else supports_partial_download = false end continue_at = if destination.exist? && supports_partial_download "-" else 0 end curl( "--location", "--remote-time", "--continue-at", continue_at.to_s, "--output", destination, *args, ** ) end |
.curl_executable ⇒ Object
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
17 18 19 20 21 22 23 24 25 26 |
# File 'utils/curl.rb', line 17 def curl_executable @curl ||= [ ENV["HOMEBREW_CURL"], which("curl"), "/usr/bin/curl", ].compact.map { |c| Pathname(c) }.find(&:executable?) raise "No executable `curl` was found" unless @curl @curl end |
.curl_http_content_headers_and_checksum(url, specs: {}, hash_needed: false, user_agent: :default) ⇒ Object
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291 292 293 294 295 296 297 298 299 300 |
# File 'utils/curl.rb', line 262 def curl_http_content_headers_and_checksum(url, specs: {}, hash_needed: false, user_agent: :default) file = Tempfile.new.tap(&:close) specs = specs.flat_map { |option, argument| ["--#{option.to_s.tr("_", "-")}", argument] } max_time = hash_needed ? "600" : "25" output, _, status = curl_output( *specs, "--dump-header", "-", "--output", file.path, "--location", "--connect-timeout", "15", "--max-time", max_time, "--retry-max-time", max_time, url, user_agent: user_agent ) status_code = :unknown while status_code == :unknown || status_code.to_s.start_with?("3") headers, _, output = output.partition("\r\n\r\n") status_code = headers[%r{HTTP/.* (\d+)}, 1] location = headers[/^Location:\s*(.*)$/i, 1] final_url = location.chomp if location end if status.success? file_contents = File.read(file.path) file_hash = Digest::SHA2.hexdigest(file_contents) if hash_needed end final_url ||= url { url: url, final_url: final_url, status: status_code, etag: headers[%r{ETag: ([wW]/)?"(([^"]|\\")*)"}, 2], content_length: headers[/Content-Length: (\d+)/, 1], headers: headers, file_hash: file_hash, file: file_contents, } ensure file.unlink end |
.curl_output(*args, **options) ⇒ Object
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
153 154 155 |
# File 'utils/curl.rb', line 153 def curl_output(*args, **) curl_with_workarounds(*args, print_stderr: false, show_output: true, **) end |
.curl_with_workarounds(*args, secrets: nil, print_stdout: nil, print_stderr: nil, debug: nil, verbose: nil, env: {}, timeout: nil, **options) ⇒ Object
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 |
# File 'utils/curl.rb', line 64 def curl_with_workarounds( *args, secrets: nil, print_stdout: nil, print_stderr: nil, debug: nil, verbose: nil, env: {}, timeout: nil, ** ) end_time = Time.now + timeout if timeout = { secrets: secrets, print_stdout: print_stdout, print_stderr: print_stderr, debug: debug, verbose: verbose, }.compact # SSL_CERT_FILE can be incorrectly set by users or portable-ruby and screw # with SSL downloads so unset it here. result = system_command curl_executable, args: curl_args(*args, **), env: { "SSL_CERT_FILE" => nil }.merge(env), timeout: end_time&.remaining, ** return result if result.success? || !args.exclude?("--http1.1") raise Timeout::Error, result.stderr.lines.last.chomp if timeout && result.status.exitstatus == 28 # Error in the HTTP2 framing layer if result.status.exitstatus == 16 return curl_with_workarounds( *args, "--http1.1", timeout: end_time&.remaining, **, ** ) end # This is a workaround for https://github.com/curl/curl/issues/1618. if result.status.exitstatus == 56 # Unexpected EOF out = curl_output("-V").stdout # If `curl` doesn't support HTTP2, the exception is unrelated to this bug. return result unless out.include?("HTTP2") # The bug is fixed in `curl` >= 7.60.0. curl_version = out[/curl (\d+(\.\d+)+)/, 1] return result if Gem::Version.new(curl_version) >= Gem::Version.new("7.60.0") return curl_with_workarounds(*args, "--http1.1", **, **) end result end |
.http_status_ok?(status) ⇒ Boolean
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
302 303 304 |
# File 'utils/curl.rb', line 302 def http_status_ok?(status) (100..299).cover?(status.to_i) end |
.url_protected_by_cloudflare?(details) ⇒ Boolean
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
Check if a URL is protected by CloudFlare (e.g. badlion.net and jaxx.io).
158 159 160 161 162 |
# File 'utils/curl.rb', line 158 def url_protected_by_cloudflare?(details) [403, 503].include?(details[:status].to_i) && details[:headers].match?(/^Set-Cookie: __cfduid=/i) && details[:headers].match?(/^Server: cloudflare/i) end |
.url_protected_by_incapsula?(details) ⇒ Boolean
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
Check if a URL is protected by Incapsula (e.g. corsair.com).
165 166 167 168 169 |
# File 'utils/curl.rb', line 165 def url_protected_by_incapsula?(details) details[:status].to_i == 403 && details[:headers].match?(/^Set-Cookie: visid_incap_/i) && details[:headers].match?(/^Set-Cookie: incap_ses_/i) end |