Supporting package directory downloads

[guybedford]

I get upstream JSPM issues from time to time about the fact that JSPM modules reference relative paths like ../_/chunk-134234.js to load chunks for multi-entry packages that use chunking optimizations.

The workflows here don't support these by default by not downloading all files of the package into a location that matches the relative pathing.

I believe #235 and the associated work on a download listing API was the right direction to resolve this, but that work seemed to stall out.

Given it's a common issue and there isn't currently a tracking issue I wanted to post this for further follow-ups in future. Happy to discuss further or help where I can too.

[4lllex]

Has the option to download into public directory been considered? Vendored files already have a version in their path and don't require an extra digest from sprockets or propshaft. If we keep the same directory structure as the cdn then relative imports don't require an import-map. A quick example for bin/importmap pin command:

#!/usr/bin/env ruby

# File: bin/importmap

require_relative "../config/application"

class Pkg
  def self.download(packages, from: "jspm", env: "production")
    response = Net::HTTP.post(
      URI("https://api.jspm.io/generate"),
      {install: Array(packages), flattenScope: true, env: ["browser", "module", env], provider: from}.to_json,
      "Content-Type" => "application/json"
    )
    json = JSON.parse(response.body)
    json["error"]&.tap(&method(:puts))
    response.value # 200 or raise
    json["staticDeps"].each do |url|
      Pkg.new(url:).save(to: "public")
    end
    json["map"]["imports"].map do |import, url|
      Pkg.new(import:, url:)
    end
  end

  def initialize(url:, import: nil)
    @url = url.to_s
    @uri = URI(url)
    @import = import
    @version = url.match(/@\d+\.\d+\.\d+/)&.to_a&.first
    @provider = /jspm|jsdelivr|unpkg/.match(@url).to_s
    @path = File.join("vendor", @provider, @uri.path)
  end

  attr_reader :import, :path

  def save(to:)
    local_path = Pathname.new(File.join(to, @path))
    local_path.parent.mkpath
    puts "Downloading #{@uri}  =>  #{local_path}"
    File.open(local_path, "wb") do |file|
      Net::HTTP.get_response(@uri) do |response|
        response.value
        file << response.body.gsub(/^\/\/# sourceMappingURL=.*/, "")
      end
    end
  end

  def public_pin = %(pin "#{@import}", to: "/#{@path}" # #{@version})

  def self.test_download(packages, from: "jspm", env: "production")
    require "vcr"
    VCR.configure do |config|
      config.cassette_library_dir = "tmp/vcr"
      config.hook_into :webmock
    end
    VCR.use_cassette([ __method__, packages, from ]) do
      download(packages, from:, env:)
    end
  end
end

module DownloadPatch
  def pin(*packages)
    require "irb/color"
    # Pkg.test_download(packages, **options.symbolize_keys).each do |pkg|
    Pkg.download(packages, **options.symbolize_keys).each do |pkg|
      puts IRB::Color.colorize_code pkg.public_pin
      update_importmap_with_pin(pkg.import, pkg.public_pin)
    end
  end
end

require "thor"
class Importmap::Commands < Thor
  prepend DownloadPatch
end

require "importmap/commands"

The few libraries I've tried all load without any 404s.

$ bin/importmap pin date-fns

Downloading https://ga.jspm.io/npm:date-fns@4.1.0/_/BDJn6Y19.js  =>  public/vendor/jspm/npm:date-fns@4.1.0/_/BDJn6Y19.js
Downloading https://ga.jspm.io/npm:date-fns@4.1.0/_/BHOttRZ-.js  =>  public/vendor/jspm/npm:date-fns@4.1.0/_/BHOttRZ-.js
Downloading https://ga.jspm.io/npm:date-fns@4.1.0/_/BkvIkgYg.js  =>  public/vendor/jspm/npm:date-fns@4.1.0/_/BkvIkgYg.js
... 257 more downloads

pin "date-fns", to: "/vendor/jspm/npm:date-fns@4.1.0/index.js" # @4.1.0

$ bin/importmap pin bootstrap --from unpkg

... 58 downloads

pin "bootstrap", to: "/vendor/unpkg/bootstrap@5.3.8/dist/js/bootstrap.esm.js" # @5.3.8
pin "@popperjs/core", to: "/vendor/unpkg/@popperjs/core@2.11.8/lib/index.js" # @2.11.8

Doing this in vendor/javascript requires adding 300+ relative pins like pin "/assets/npm:date-fns@4.1.0/_/BDJn6Y19.js", to: "npm:date-fns@4.1.0/_/BDJn6Y19.js" to importmap.rb

[doits]

Thank for this @4lllex! With your importmap I could successfully do a ./bin/importmap pin '@videojs/html@10.0.0-beta.13' --from unpkg and it works without a change! There are assets etc. which now can be used (and are required by videojs) by having everything available.