Class: Archive Private

Inherits:
Object
  • Object
show all
Extended by:
T::Sig
Includes:
Context, Utils::Curl
Defined in:
archive.rb

Overview

This class is part of a private API. This class may only be used in the Homebrew/brew repository. Third parties should avoid using this class if possible, as it may be removed or changed without warning.

The Internet Archive API client.

Defined Under Namespace

Classes: Error

Constant Summary collapse

URL_PREFIX =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning. 

"https://archive.org"
S3_DOMAIN =

This constant is part of a private API. This constant may only be used in the Homebrew/brew repository. Third parties should avoid using this constant if possible, as it may be removed or changed without warning. 

"s3.us.archive.org"

Instance Method Summary collapse

Methods included from Utils::Curl

curl, curl_args, curl_check_http_content, curl_download, curl_executable, curl_http_content_headers_and_checksum, curl_output, curl_with_workarounds, http_status_ok?, parse_headers, url_protected_by_cloudflare?, url_protected_by_incapsula?

Methods included from Context

current, current=, #debug?, #quiet?, #verbose?, #with_context

Constructor Details

#initialize(item: "homebrew") ⇒ void

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

  • item (String, nil) (defaults to: "homebrew")

Raises:



28
29
30
31
32
 
# File 'archive.rb', line 28

def initialize(item: "homebrew")
  raise UsageError, "Must set the Archive item!" unless item

  @archive_item = item
end

Instance Method Details

#file_delete_instructions(directory, filename) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

Returns:



119
120
121
122
123
124
125
126
 
# File 'archive.rb', line 119

def file_delete_instructions(directory, filename)
  <<~EOS
    Run:
      curl -X DELETE -H "Authorization: AWS $HOMEBREW_INTERNET_ARCHIVE_KEY" https://#{@archive_item}.#{S3_DOMAIN}/#{directory}/#{filename}
    Or run:
      ia delete #{@archive_item} #{directory}/#{filename}
  EOS
end

#inspect ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Returns:



23
24
25
 
# File 'archive.rb', line 23

def inspect
  "#<Archive: item=#{@archive_item}>"
end

#mirror_formula(formula, directory: "mirror", warn_on_error: false) ⇒ String

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Parameters:

  • formula (Formula)
  • directory (String) (defaults to: "mirror")
  • warn_on_error (Boolean) (defaults to: false)

Returns:



84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
 
# File 'archive.rb', line 84

def mirror_formula(formula, directory: "mirror", warn_on_error: false)
  formula.downloader.fetch

  filename = ERB::Util.url_encode(formula.downloader.basename)
  destination_url = "#{URL_PREFIX}/download/#{@archive_item}/#{directory}/#{filename}"

  odebug "Uploading to #{destination_url}"

  upload(
    formula.downloader.cached_location,
    directory:     directory,
    remote_file:   filename,
    warn_on_error: warn_on_error,
  )

  destination_url
end

#open_api(url, *args, auth: true) ⇒ Object

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning. 



34
35
36
37
38
39
40
41
42
43
44
45
46
47
 
# File 'archive.rb', line 34

def open_api(url, *args, auth: true)
  if auth
    key = Homebrew::EnvConfig.internet_archive_key
    raise UsageError, "HOMEBREW_INTERNET_ARCHIVE_KEY is unset." if key.blank?

    if key.exclude?(":")
      raise UsageError, "Use HOMEBREW_INTERNET_ARCHIVE_KEY=access:secret. See #{URL_PREFIX}/account/s3.php"
    end

    args += ["--header", "Authorization: AWS #{key}"]
  end

  curl(*args, url, print_stdout: false, secrets: key)
end

#remote_md5(directory:, remote_file:) ⇒ String?

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

Gets the MD5 hash of the specified remote file.

Parameters:

Returns:

  • (String, nil) —

    the hash, the empty string (if the file doesn’t have a hash), nil (if the file doesn’t exist)



106
107
108
109
110
111
112
113
114
115
116
 
# File 'archive.rb', line 106

def remote_md5(directory:, remote_file:)
  url = "https://#{@archive_item}.#{S3_DOMAIN}/#{directory}/#{remote_file}"
  result = curl_output "--fail", "--silent", "--head", "--location", url
  if result.success?
    result.stdout.match(/^ETag: "(\h{32})"/)&.values_at(1)&.first || ""
  else
    raise Error if result.status.exitstatus != 22 && result.stderr.exclude?("404 Not Found")

    nil
  end
end

#upload(local_file, directory:, remote_file:, warn_on_error: false) ⇒ void

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

This method returns an undefined value.

Parameters:

  • local_file (String)
  • directory (String)
  • remote_file (String)
  • warn_on_error (Boolean, nil) (defaults to: false) 


55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
 
# File 'archive.rb', line 55

def upload(local_file, directory:, remote_file:, warn_on_error: false)
  local_file = Pathname.new(local_file)
  unless local_file.exist?
    msg = "#{local_file} for upload doesn't exist!"
    raise Error, msg unless warn_on_error

    # Warn and return early here since we know this upload is going to fail.
    opoo msg
    return
  end

  md5_base64 = Digest::MD5.base64digest(local_file.read)
  url = "https://#{@archive_item}.#{S3_DOMAIN}/#{directory}/#{remote_file}"
  args = ["--upload-file", local_file, "--header", "Content-MD5: #{md5_base64}"]
  args << "--fail" unless warn_on_error
  result = T.unsafe(self).open_api(url, *args)
  return if result.success? && result.stdout.exclude?("Error")

  msg = "Bottle upload failed: #{result.stdout}"
  raise msg unless warn_on_error

  opoo msg
end

#upload_bottles(bottles_hash, warn_on_error: false) ⇒ void

This method is part of a private API. This method may only be used in the Homebrew/brew repository. Third parties should avoid using this method if possible, as it may be removed or changed without warning.

This method returns an undefined value.

Parameters:

  • bottles_hash (Hash{String => T.untyped})
  • warn_on_error (Boolean, nil) (defaults to: false) 


132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
 
# File 'archive.rb', line 132

def upload_bottles(bottles_hash, warn_on_error: false)
  bottles_hash.each do |formula_full_name, bottle_hash|
    _, formula_tap_repo, = formula_full_name.split("/")
    directory = if formula_tap_repo
      "bottles-#{tap.repo}"
    else
      "bottles"
    end

    bottle_count = bottle_hash["bottle"]["tags"].length

    bottle_hash["bottle"]["tags"].each_value do |tag_hash|
      filename = tag_hash["filename"] # URL encoded in Bottle::Filename#archive
      delete_instructions = file_delete_instructions(directory, filename)

      local_filename = tag_hash["local_filename"]
      md5 = Digest::MD5.hexdigest(File.read(local_filename))

      odebug "Checking remote file #{@archive_item}/#{directory}/#{filename}"
      result = remote_md5(directory: directory, remote_file: filename)
      case result
      when nil
        # File doesn't exist.
        odebug "Uploading #{@archive_item}/#{directory}/#{filename}"
        upload(local_filename,
               directory:     directory,
               remote_file:   filename,
               warn_on_error: warn_on_error)
      when md5
        # File exists, hash matches.
        odebug "#{filename} is already published with matching hash."
        bottle_count -= 1
      when ""
        # File exists, but can't find hash
        failed_message = "#{filename} is already published!"
        raise Error, "#{failed_message}\n#{delete_instructions}" unless warn_on_error

        opoo failed_message
      else
        # File exists, but hash either doesn't exist or is mismatched.
        failed_message = <<~EOS
          #{filename} is already published with a mismatched hash!
            Expected: #{md5}
            Actual:   #{result}
        EOS
        raise Error, "#{failed_message}#{delete_instructions}" unless warn_on_error

        opoo failed_message
      end
    end

    odebug "Uploaded #{bottle_count} bottles"
  end
end