Class: Archive Private
- Inherits:
-
Object
- Object
- Archive
- Extended by:
- T::Sig
- Includes:
- Context, Utils::Curl
- Defined in:
- archive.rb
Overview
This class is part of a private API. This class may only be used in the Homebrew/brew repository. Third parties should avoid using this class if possible, as it may be removed or changed without warning.
The Internet Archive API client.
Defined Under Namespace
Classes: Error
Constant Summary collapse
- URL_PREFIX =
This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning.
"https://archive.org"
- S3_DOMAIN =
This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning.
"s3.us.archive.org"
Instance Method Summary collapse
-
#file_delete_instructions(directory, filename) ⇒ String
private
-
#initialize(item: "homebrew") ⇒ void
constructor
private
-
#inspect ⇒ String
private
-
#mirror_formula(formula, directory: "mirror", warn_on_error: false) ⇒ String
private
-
#open_api(url, *args, auth: true) ⇒ Object
private
-
#remote_md5(directory:, remote_file:) ⇒ String?
private
Gets the MD5 hash of the specified remote file.
-
#upload(local_file, directory:, remote_file:, warn_on_error: false) ⇒ void
private
-
#upload_bottles(bottles_hash, warn_on_error: false) ⇒ void
private
Methods included from Utils::Curl
curl, curl_args, curl_check_http_content, curl_download, curl_executable, curl_http_content_headers_and_checksum, curl_output, curl_with_workarounds, http_status_ok?, url_protected_by_cloudflare?, url_protected_by_incapsula?
Methods included from Context
current, current=, #debug?, #quiet?, #verbose?, #with_context
Constructor Details
#initialize(item: "homebrew") ⇒ void
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
28 29 30 31 32 |
# File 'archive.rb', line 28 def initialize(item: "homebrew") raise UsageError, "Must set the Archive item!" unless item @archive_item = item end |
Instance Method Details
#file_delete_instructions(directory, filename) ⇒ String
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
119 120 121 122 123 124 125 126 |
# File 'archive.rb', line 119 def file_delete_instructions(directory, filename) <<~EOS Run: curl -X DELETE -H "Authorization: AWS $HOMEBREW_INTERNET_ARCHIVE_KEY" https://#{@archive_item}.#{S3_DOMAIN}/#{directory}/#{filename} Or run: ia delete #{@archive_item} #{directory}/#{filename} EOS end |
#inspect ⇒ String
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
23 24 25 |
# File 'archive.rb', line 23 def inspect "#<Archive: item=#{@archive_item}>" end |
#mirror_formula(formula, directory: "mirror", warn_on_error: false) ⇒ String
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 |
# File 'archive.rb', line 84 def mirror_formula(formula, directory: "mirror", warn_on_error: false) formula.downloader.fetch filename = ERB::Util.url_encode(formula.downloader.basename) destination_url = "#{URL_PREFIX}/download/#{@archive_item}/#{directory}/#{filename}" odebug "Uploading to #{destination_url}" upload( formula.downloader.cached_location, directory: directory, remote_file: filename, warn_on_error: warn_on_error, ) destination_url end |
#open_api(url, *args, auth: true) ⇒ Object
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
34 35 36 37 38 39 40 41 42 43 44 45 46 47 |
# File 'archive.rb', line 34 def open_api(url, *args, auth: true) if auth key = Homebrew::EnvConfig.internet_archive_key raise UsageError, "HOMEBREW_INTERNET_ARCHIVE_KEY is unset." if key.blank? if key.exclude?(":") raise UsageError, "Use HOMEBREW_INTERNET_ARCHIVE_KEY=access:secret. See #{URL_PREFIX}/account/s3.php" end args += ["--header", "Authorization: AWS #{key}"] end curl(*args, url, print_stdout: false, secrets: key) end |
#remote_md5(directory:, remote_file:) ⇒ String?
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
Gets the MD5 hash of the specified remote file.
106 107 108 109 110 111 112 113 114 115 116 |
# File 'archive.rb', line 106 def remote_md5(directory:, remote_file:) url = "https://#{@archive_item}.#{S3_DOMAIN}/#{directory}/#{remote_file}" result = curl_output "--fail", "--silent", "--head", "--location", url if result.success? result.stdout.match(/^ETag: "(\h{32})"/)&.values_at(1)&.first || "" else raise Error if result.status.exitstatus != 22 && result.stderr.exclude?("404 Not Found") nil end end |
#upload(local_file, directory:, remote_file:, warn_on_error: false) ⇒ void
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
This method returns an undefined value.
55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 |
# File 'archive.rb', line 55 def upload(local_file, directory:, remote_file:, warn_on_error: false) local_file = Pathname.new(local_file) unless local_file.exist? msg = "#{local_file} for upload doesn't exist!" raise Error, msg unless warn_on_error # Warn and return early here since we know this upload is going to fail. opoo msg return end md5_base64 = Digest::MD5.base64digest(local_file.read) url = "https://#{@archive_item}.#{S3_DOMAIN}/#{directory}/#{remote_file}" args = ["--upload-file", local_file, "--header", "Content-MD5: #{md5_base64}"] args << "--fail" unless warn_on_error result = T.unsafe(self).open_api(url, *args) return if result.success? && result.stdout.exclude?("Error") msg = "Bottle upload failed: #{result.stdout}" raise msg unless warn_on_error opoo msg end |
#upload_bottles(bottles_hash, warn_on_error: false) ⇒ void
This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.
This method returns an undefined value.
132 133 134 135 136 137 138 139 140 141 142 143 144 145 146 147 148 149 150 151 152 153 154 155 156 157 158 159 160 161 162 163 164 165 166 167 168 169 170 171 172 173 174 175 176 177 178 179 |
# File 'archive.rb', line 132 def upload_bottles(bottles_hash, warn_on_error: false) bottles_hash.each do |_formula_name, bottle_hash| directory = bottle_hash["bintray"]["repository"] bottle_count = bottle_hash["bottle"]["tags"].length bottle_hash["bottle"]["tags"].each_value do |tag_hash| filename = tag_hash["filename"] # URL encoded in Bottle::Filename#archive delete_instructions = file_delete_instructions(directory, filename) local_filename = tag_hash["local_filename"] md5 = Digest::MD5.hexdigest(File.read(local_filename)) odebug "Checking remote file #{@archive_item}/#{directory}/#{filename}" result = remote_md5(directory: directory, remote_file: filename) case result when nil # File doesn't exist. odebug "Uploading #{@archive_item}/#{directory}/#{filename}" upload(local_filename, directory: directory, remote_file: filename, warn_on_error: warn_on_error) when md5 # File exists, hash matches. odebug "#{filename} is already published with matching hash." bottle_count -= 1 when "" # File exists, but can't find hash = "#{filename} is already published!" raise Error, "#{}\n#{delete_instructions}" unless warn_on_error opoo else # File exists, but hash either doesn't exist or is mismatched. = <<~EOS #{filename} is already published with a mismatched hash! Expected: #{md5} Actual: #{result} EOS raise Error, "#{}#{delete_instructions}" unless warn_on_error opoo end end odebug "Uploaded #{bottle_count} bottles" end end |