Module: Utils::Curl Private
- Included in:
- Bintray, CurlDownloadStrategy, SPDX, SPDX, SharedAudits, SharedAudits
- Defined in:
- brew/Library/Homebrew/utils/curl.rb
Overview
This module is part of a private API. You should avoid using this module if possible, as it may be removed or be changed in the future.
Helper function for interacting with curl
.
Class Method Summary collapse
-
.curl(*args, print_stdout: true, **options) ⇒ Object
private
-
.curl_args(*extra_args, **options) ⇒ Object
private
-
.curl_check_http_content(url, user_agents: [:default], check_content: false, strict: false) ⇒ Object
private
-
.curl_download(*args, to: nil, partial: true, **options) ⇒ Object
private
-
.curl_executable ⇒ Object
private
-
.curl_http_content_headers_and_checksum(url, hash_needed: false, user_agent: :default) ⇒ Object
private
-
.curl_output(*args, **options) ⇒ Object
private
-
.curl_with_workarounds(*args, secrets: nil, print_stdout: nil, print_stderr: nil, debug: nil, verbose: nil, env: {}, **options) ⇒ Object
private
-
.http_status_ok?(status) ⇒ Boolean
private
-
.url_protected_by_cloudflare?(details) ⇒ Boolean
private
Check if a URL is protected by CloudFlare (e.g. badlion.net and jaxx.io).
-
.url_protected_by_incapsula?(details) ⇒ Boolean
private
Check if a URL is protected by Incapsula (e.g. corsair.com).
Class Method Details
.curl(*args, print_stdout: true, **options) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
98 99 100 101 102 |
# File 'brew/Library/Homebrew/utils/curl.rb', line 98 def curl(*args, print_stdout: true, **) result = curl_with_workarounds(*args, print_stdout: print_stdout, **) result.assert_success! result end |
.curl_args(*extra_args, **options) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 |
# File 'brew/Library/Homebrew/utils/curl.rb', line 24 def curl_args(*extra_args, **) args = [] # do not load .curlrc unless requested (must be the first argument) args << "--disable" unless Homebrew::EnvConfig.curlrc? args << "--globoff" args << "--show-error" args << "--user-agent" << case [:user_agent] when :browser, :fake HOMEBREW_USER_AGENT_FAKE_SAFARI when :default, nil HOMEBREW_USER_AGENT_CURL when String [:user_agent] end args << "--header" << "Accept-Language: en" unless [:show_output] == true args << "--fail" args << "--progress-bar" unless Context.current.verbose? args << "--verbose" if Homebrew::EnvConfig.curl_verbose? args << "--silent" unless $stdout.tty? end args << "--retry" << Homebrew::EnvConfig.curl_retries unless [:retry] == false args + extra_args end |
.curl_check_http_content(url, user_agents: [:default], check_content: false, strict: false) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230 231 232 233 234 235 236 237 238 |
# File 'brew/Library/Homebrew/utils/curl.rb', line 154 def curl_check_http_content(url, user_agents: [:default], check_content: false, strict: false) return unless url.start_with? "http" secure_url = url.sub(/\Ahttp:/, "https:") secure_details = nil hash_needed = false if url != secure_url user_agents.each do |user_agent| secure_details = curl_http_content_headers_and_checksum(secure_url, hash_needed: true, user_agent: user_agent) next unless http_status_ok?(secure_details[:status]) hash_needed = true user_agents = [user_agent] break end end details = nil user_agents.each do |user_agent| details = curl_http_content_headers_and_checksum(url, hash_needed: hash_needed, user_agent: user_agent) break if http_status_ok?(details[:status]) end unless details[:status] # Hack around https://github.com/Homebrew/brew/issues/3199 return if MacOS.version == :el_capitan return "The URL #{url} is not reachable" end unless http_status_ok?(details[:status]) return if url_protected_by_cloudflare?(details) || url_protected_by_incapsula?(details) return "The URL #{url} is not reachable (HTTP status code #{details[:status]})" end if url.start_with?("https://") && Homebrew::EnvConfig.no_insecure_redirect? && !details[:final_url].start_with?("https://") return "The URL #{url} redirects back to HTTP" end return unless secure_details return if !http_status_ok?(details[:status]) || !http_status_ok?(secure_details[:status]) etag_match = details[:etag] && details[:etag] == secure_details[:etag] content_length_match = details[:content_length] && details[:content_length] == secure_details[:content_length] file_match = details[:file_hash] == secure_details[:file_hash] if (etag_match || content_length_match || file_match) && secure_details[:final_url].start_with?("https://") && url.start_with?("http://") return "The URL #{url} should use HTTPS rather than HTTP" end return unless check_content no_protocol_file_contents = %r{https?:\\?/\\?/} http_content = details[:file]&.gsub(no_protocol_file_contents, "/") https_content = secure_details[:file]&.gsub(no_protocol_file_contents, "/") # Check for the same content after removing all protocols if (http_content && https_content) && (http_content == https_content) && url.start_with?("http://") && secure_details[:final_url].start_with?("https://") return "The URL #{url} should use HTTPS rather than HTTP" end return unless strict # Same size, different content after normalization # (typical causes: Generated ID, Timestamp, Unix time) if http_content.length == https_content.length return "The URL #{url} may be able to use HTTPS rather than HTTP. Please verify it in a browser." end lenratio = (100 * https_content.length / http_content.length).to_i return unless (90..110).cover?(lenratio) "The URL #{url} may be able to use HTTPS rather than HTTP. Please verify it in a browser." end |
.curl_download(*args, to: nil, partial: true, **options) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132 133 134 |
# File 'brew/Library/Homebrew/utils/curl.rb', line 104 def curl_download(*args, to: nil, partial: true, **) destination = Pathname(to) destination.dirname.mkpath if partial range_stdout = curl_output("--location", "--range", "0-1", "--dump-header", "-", "--write-out", "%\{http_code}", "--output", "/dev/null", *args, **).stdout headers, _, http_status = range_stdout.partition("\r\n\r\n") supports_partial_download = http_status.to_i == 206 # Partial Content if supports_partial_download && destination.exist? && destination.size == %r{^.*Content-Range: bytes \d+-\d+/(\d+)\r\n.*$}m.match(headers)&.[](1)&.to_i return # We've already downloaded all the bytes end else supports_partial_download = false end continue_at = if destination.exist? && supports_partial_download "-" else 0 end curl( "--location", "--remote-time", "--continue-at", continue_at.to_s, "--output", destination, *args, ** ) end |
.curl_executable ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
13 14 15 16 17 18 19 20 21 22 |
# File 'brew/Library/Homebrew/utils/curl.rb', line 13 def curl_executable @curl ||= [ ENV["HOMEBREW_CURL"], which("curl"), "/usr/bin/curl", ].compact.map { |c| Pathname(c) }.find(&:executable?) raise "no executable curl was found" unless @curl @curl end |
.curl_http_content_headers_and_checksum(url, hash_needed: false, user_agent: :default) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268 269 270 271 272 273 274 275 276 277 |
# File 'brew/Library/Homebrew/utils/curl.rb', line 240 def curl_http_content_headers_and_checksum(url, hash_needed: false, user_agent: :default) file = Tempfile.new.tap(&:close) max_time = hash_needed ? "600" : "25" output, _, status = curl_output( "--dump-header", "-", "--output", file.path, "--location", "--connect-timeout", "15", "--max-time", max_time, "--retry-max-time", max_time, url, user_agent: user_agent ) status_code = :unknown while status_code == :unknown || status_code.to_s.start_with?("3") headers, _, output = output.partition("\r\n\r\n") status_code = headers[%r{HTTP/.* (\d+)}, 1] location = headers[/^Location:\s*(.*)$/i, 1] final_url = location.chomp if location end if status.success? file_contents = File.read(file.path) file_hash = Digest::SHA2.hexdigest(file_contents) if hash_needed end final_url ||= url { url: url, final_url: final_url, status: status_code, etag: headers[%r{ETag: ([wW]/)?"(([^"]|\\")*)"}, 2], content_length: headers[/Content-Length: (\d+)/, 1], headers: headers, file_hash: file_hash, file: file_contents, } ensure file.unlink end |
.curl_output(*args, **options) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
136 137 138 |
# File 'brew/Library/Homebrew/utils/curl.rb', line 136 def curl_output(*args, **) curl_with_workarounds(*args, print_stderr: false, show_output: true, **) end |
.curl_with_workarounds(*args, secrets: nil, print_stdout: nil, print_stderr: nil, debug: nil, verbose: nil, env: {}, **options) ⇒ Object
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 |
# File 'brew/Library/Homebrew/utils/curl.rb', line 57 def curl_with_workarounds( *args, secrets: nil, print_stdout: nil, print_stderr: nil, debug: nil, verbose: nil, env: {}, ** ) = { secrets: secrets, print_stdout: print_stdout, print_stderr: print_stderr, debug: debug, verbose: verbose, }.compact # SSL_CERT_FILE can be incorrectly set by users or portable-ruby and screw # with SSL downloads so unset it here. result = system_command curl_executable, args: curl_args(*args, **), env: { "SSL_CERT_FILE" => nil }.merge(env), ** if !result.success? && args.exclude?("--http1.1") # This is a workaround for https://github.com/curl/curl/issues/1618. if result.status.exitstatus == 56 # Unexpected EOF out = curl_output("-V").stdout # If `curl` doesn't support HTTP2, the exception is unrelated to this bug. return result unless out.include?("HTTP2") # The bug is fixed in `curl` >= 7.60.0. curl_version = out[/curl (\d+(\.\d+)+)/, 1] return result if Gem::Version.new(curl_version) >= Gem::Version.new("7.60.0") return curl_with_workarounds(*args, "--http1.1", **, **) end if result.status.exitstatus == 16 # Error in the HTTP2 framing layer return curl_with_workarounds(*args, "--http1.1", **, **) end end result end |
.http_status_ok?(status) ⇒ Boolean
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
279 280 281 |
# File 'brew/Library/Homebrew/utils/curl.rb', line 279 def http_status_ok?(status) (100..299).cover?(status.to_i) end |
.url_protected_by_cloudflare?(details) ⇒ Boolean
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Check if a URL is protected by CloudFlare (e.g. badlion.net and jaxx.io).
141 142 143 144 145 |
# File 'brew/Library/Homebrew/utils/curl.rb', line 141 def url_protected_by_cloudflare?(details) [403, 503].include?(details[:status].to_i) && details[:headers].match?(/^Set-Cookie: __cfduid=/i) && details[:headers].match?(/^Server: cloudflare/i) end |
.url_protected_by_incapsula?(details) ⇒ Boolean
This method is part of a private API. You should avoid using this method if possible, as it may be removed or be changed in the future.
Check if a URL is protected by Incapsula (e.g. corsair.com).
148 149 150 151 152 |
# File 'brew/Library/Homebrew/utils/curl.rb', line 148 def url_protected_by_incapsula?(details) details[:status].to_i == 403 && details[:headers].match?(/^Set-Cookie: visid_incap_/i) && details[:headers].match?(/^Set-Cookie: incap_ses_/i) end |