class Prism::Translation::Parser

This class is the entry-point for converting a prism syntax tree into the whitequark/parser gem’s syntax tree. It inherits from the base parser for the parser gem, and overrides the parse* methods to parse with prism and then translate.

Note that this version of the parser always parses using the latest version of Ruby syntax supported by Prism. If you want specific version support, use one of the version-specific subclasses, such as Prism::Translation::Parser34. If you want to parse using the same version of Ruby syntax as the currently running version of Ruby, use Prism::Translation::ParserCurrent.

Public Class Methods

new(builder = Prism::Translation::Parser::Builder.new, parser: Prism) click to toggle source

The builder argument is used to create the parser using our custom builder class by default.

By using the :parser keyword argument, you can translate in a way that is compatible with the Parser gem using any parser.

For example, in RuboCop for Ruby LSP, the following approach can be used to improve performance by reusing a pre-parsed Prism::ParseLexResult:

class PrismPreparsed def initialize(prism_result) @prism_result = prism_result end

def parse_lex(source, **options)
  @prism_result
end

end

prism_preparsed = PrismPreparsed.new(prism_result)

Prism::Translation::Ruby34.new(builder, parser: prism_preparsed)

In an object passed to the :parser keyword argument, the parse and parse_lex methods should be implemented as needed.

Calls superclass method

# File lib/prism/translation/parser.rb, line 74
      def initialize(builder = Prism::Translation::Parser::Builder.new, parser: Prism)
        if !builder.is_a?(Prism::Translation::Parser::Builder)
          warn(<<~MSG, uplevel: 1, category: :deprecated)
            [deprecation]: The builder passed to `Prism::Translation::Parser.new` is not a \
            `Prism::Translation::Parser::Builder` subclass. This will raise in the next major version.
          MSG
        end
        @parser = parser

        super(builder)
      end

Public Instance Methods

default_encoding() click to toggle source

The default encoding for Ruby files is UTF-8.

# File lib/prism/translation/parser.rb, line 91
def default_encoding
  Encoding::UTF_8
end

parse(source_buffer) click to toggle source

Parses a source buffer and returns the AST.

# File lib/prism/translation/parser.rb, line 99
def parse(source_buffer)
  @source_buffer = source_buffer
  source = source_buffer.source

  offset_cache = build_offset_cache(source)
  result = unwrap(@parser.parse(source, **prism_options), offset_cache)

  build_ast(result.value, offset_cache)
ensure
  @source_buffer = nil
end

parse_with_comments(source_buffer) click to toggle source

Parses a source buffer and returns the AST and the source code comments.

# File lib/prism/translation/parser.rb, line 112
def parse_with_comments(source_buffer)
  @source_buffer = source_buffer
  source = source_buffer.source

  offset_cache = build_offset_cache(source)
  result = unwrap(@parser.parse(source, **prism_options), offset_cache)

  [
    build_ast(result.value, offset_cache),
    build_comments(result.comments, offset_cache)
  ]
ensure
  @source_buffer = nil
end

tokenize(source_buffer, recover = false) click to toggle source

Parses a source buffer and returns the AST, the source code comments, and the tokens emitted by the lexer.

# File lib/prism/translation/parser.rb, line 129
def tokenize(source_buffer, recover = false)
  @source_buffer = source_buffer
  source = source_buffer.source

  offset_cache = build_offset_cache(source)
  result =
    begin
      unwrap(@parser.parse_lex(source, **prism_options), offset_cache)
    rescue ::Parser::SyntaxError
      raise if !recover
    end

  program, tokens = result.value
  ast = build_ast(program, offset_cache) if result.success?

  [
    ast,
    build_comments(result.comments, offset_cache),
    build_tokens(tokens, offset_cache)
  ]
ensure
  @source_buffer = nil
end

try_declare_numparam(node) click to toggle source

Since prism resolves num params for us, we don’t need to support this kind of logic here.

# File lib/prism/translation/parser.rb, line 155
def try_declare_numparam(node)
  node.children[0].match?(/\A_[1-9]\z/)
end

Private Instance Methods

build_ast(program, offset_cache) click to toggle source

Build the parser gem AST from the prism AST.

# File lib/prism/translation/parser.rb, line 313
def build_ast(program, offset_cache)
  program.accept(Compiler.new(self, offset_cache))
end

build_comments(comments, offset_cache) click to toggle source

Build the parser gem comments from the prism comments.

# File lib/prism/translation/parser.rb, line 318
def build_comments(comments, offset_cache)
  comments.map do |comment|
    ::Parser::Source::Comment.new(build_range(comment.location, offset_cache))
  end
end

build_offset_cache(source) click to toggle source

Prism deals with offsets in bytes, while the parser gem deals with offsets in characters. We need to handle this conversion in order to build the parser gem AST.

If the bytesize of the source is the same as the length, then we can just use the offset directly. Otherwise, we build an array where the index is the byte offset and the value is the character offset.

# File lib/prism/translation/parser.rb, line 296
def build_offset_cache(source)
  if source.bytesize == source.length
    -> (offset) { offset }
  else
    offset_cache = []
    offset = 0

    source.each_char do |char|
      char.bytesize.times { offset_cache << offset }
      offset += 1
    end

    offset_cache << offset
  end
end

build_range(location, offset_cache) click to toggle source

Build a range from a prism location.

# File lib/prism/translation/parser.rb, line 330
def build_range(location, offset_cache)
  ::Parser::Source::Range.new(
    source_buffer,
    offset_cache[location.start_offset],
    offset_cache[location.end_offset]
  )
end

build_tokens(tokens, offset_cache) click to toggle source

Build the parser gem tokens from the prism tokens.

# File lib/prism/translation/parser.rb, line 325
def build_tokens(tokens, offset_cache)
  Lexer.new(source_buffer, tokens, offset_cache).to_a
end

convert_for_prism(version) click to toggle source

Converts the version format handled by Parser to the format handled by Prism.

# File lib/prism/translation/parser.rb, line 353
def convert_for_prism(version)
  case version
  when 33
    "3.3.1"
  when 34
    "3.4.0"
  when 35, 40
    "4.0.0"
  when 41
    "4.1.0"
  else
    "latest"
  end
end

error_diagnostic(error, offset_cache) click to toggle source

Build a diagnostic from the given prism parse error.

# File lib/prism/translation/parser.rb, line 174
def error_diagnostic(error, offset_cache)
  location = error.location
  diagnostic_location = build_range(location, offset_cache)

  case error.type
  when :argument_block_multi
    Diagnostic.new(:error, :block_and_blockarg, {}, diagnostic_location, [])
  when :argument_formal_constant
    Diagnostic.new(:error, :argument_const, {}, diagnostic_location, [])
  when :argument_formal_class
    Diagnostic.new(:error, :argument_cvar, {}, diagnostic_location, [])
  when :argument_formal_global
    Diagnostic.new(:error, :argument_gvar, {}, diagnostic_location, [])
  when :argument_formal_ivar
    Diagnostic.new(:error, :argument_ivar, {}, diagnostic_location, [])
  when :argument_no_forwarding_amp
    Diagnostic.new(:error, :no_anonymous_blockarg, {}, diagnostic_location, [])
  when :argument_no_forwarding_star
    Diagnostic.new(:error, :no_anonymous_restarg, {}, diagnostic_location, [])
  when :argument_no_forwarding_star_star
    Diagnostic.new(:error, :no_anonymous_kwrestarg, {}, diagnostic_location, [])
  when :begin_lonely_else
    location = location.copy(length: 4)
    diagnostic_location = build_range(location, offset_cache)
    Diagnostic.new(:error, :useless_else, {}, diagnostic_location, [])
  when :class_name, :module_name
    Diagnostic.new(:error, :module_name_const, {}, diagnostic_location, [])
  when :class_in_method
    Diagnostic.new(:error, :class_in_def, {}, diagnostic_location, [])
  when :def_endless_setter
    Diagnostic.new(:error, :endless_setter, {}, diagnostic_location, [])
  when :embdoc_term
    Diagnostic.new(:error, :embedded_document, {}, diagnostic_location, [])
  when :incomplete_variable_class, :incomplete_variable_class_3_3
    location = location.copy(length: location.length + 1)
    diagnostic_location = build_range(location, offset_cache)

    Diagnostic.new(:error, :cvar_name, { name: location.slice }, diagnostic_location, [])
  when :incomplete_variable_instance, :incomplete_variable_instance_3_3
    location = location.copy(length: location.length + 1)
    diagnostic_location = build_range(location, offset_cache)

    Diagnostic.new(:error, :ivar_name, { name: location.slice }, diagnostic_location, [])
  when :invalid_variable_global, :invalid_variable_global_3_3
    Diagnostic.new(:error, :gvar_name, { name: location.slice }, diagnostic_location, [])
  when :module_in_method
    Diagnostic.new(:error, :module_in_def, {}, diagnostic_location, [])
  when :numbered_parameter_ordinary
    Diagnostic.new(:error, :ordinary_param_defined, {}, diagnostic_location, [])
  when :numbered_parameter_outer_scope
    Diagnostic.new(:error, :numparam_used_in_outer_scope, {}, diagnostic_location, [])
  when :parameter_circular
    Diagnostic.new(:error, :circular_argument_reference, { var_name: location.slice }, diagnostic_location, [])
  when :parameter_name_repeat
    Diagnostic.new(:error, :duplicate_argument, {}, diagnostic_location, [])
  when :parameter_numbered_reserved
    Diagnostic.new(:error, :reserved_for_numparam, { name: location.slice }, diagnostic_location, [])
  when :regexp_unknown_options
    Diagnostic.new(:error, :regexp_options, { options: location.slice[1..] }, diagnostic_location, [])
  when :singleton_for_literals
    Diagnostic.new(:error, :singleton_literal, {}, diagnostic_location, [])
  when :string_literal_eof
    Diagnostic.new(:error, :string_eof, {}, diagnostic_location, [])
  when :unexpected_token_ignore
    Diagnostic.new(:error, :unexpected_token, { token: location.slice }, diagnostic_location, [])
  when :write_target_in_method
    Diagnostic.new(:error, :dynamic_const, {}, diagnostic_location, [])
  else
    PrismDiagnostic.new(error.message, :error, error.type, diagnostic_location)
  end
end

prism_options() click to toggle source

Options for how prism should parse/lex the source.

# File lib/prism/translation/parser.rb, line 339
def prism_options
  options = {
    filepath: @source_buffer.name,
    version: convert_for_prism(version),
    partial_script: true,
  }
  # The parser gem always encodes to UTF-8, unless it is binary.
  # https://github.com/whitequark/parser/blob/v3.3.6.0/lib/parser/source/buffer.rb#L80-L107
  options[:encoding] = false if @source_buffer.source.encoding != Encoding::BINARY

  options
end

unwrap(result, offset_cache) click to toggle source

If there was a error generated during the parse, then raise an appropriate syntax error. Otherwise return the result.

# File lib/prism/translation/parser.rb, line 274
def unwrap(result, offset_cache)
  result.errors.each do |error|
    next unless valid_error?(error)
    diagnostics.process(error_diagnostic(error, offset_cache))
  end

  result.warnings.each do |warning|
    next unless valid_warning?(warning)
    diagnostic = warning_diagnostic(warning, offset_cache)
    diagnostics.process(diagnostic) if diagnostic
  end

  result
end

valid_error?(error) click to toggle source

This is a hook to allow consumers to disable some errors if they don’t want them to block creating the syntax tree.

# File lib/prism/translation/parser.rb, line 163
def valid_error?(error)
  true
end

valid_warning?(warning) click to toggle source

This is a hook to allow consumers to disable some warnings if they don’t want them to block creating the syntax tree.

# File lib/prism/translation/parser.rb, line 169
def valid_warning?(warning)
  true
end

warning_diagnostic(warning, offset_cache) click to toggle source

Build a diagnostic from the given prism parse warning.

# File lib/prism/translation/parser.rb, line 247
def warning_diagnostic(warning, offset_cache)
  diagnostic_location = build_range(warning.location, offset_cache)

  case warning.type
  when :ambiguous_first_argument_plus
    Diagnostic.new(:warning, :ambiguous_prefix, { prefix: "+" }, diagnostic_location, [])
  when :ambiguous_first_argument_minus
    Diagnostic.new(:warning, :ambiguous_prefix, { prefix: "-" }, diagnostic_location, [])
  when :ambiguous_prefix_ampersand
    Diagnostic.new(:warning, :ambiguous_prefix, { prefix: "&" }, diagnostic_location, [])
  when :ambiguous_prefix_star
    Diagnostic.new(:warning, :ambiguous_prefix, { prefix: "*" }, diagnostic_location, [])
  when :ambiguous_prefix_star_star
    Diagnostic.new(:warning, :ambiguous_prefix, { prefix: "**" }, diagnostic_location, [])
  when :ambiguous_slash
    Diagnostic.new(:warning, :ambiguous_regexp, {}, diagnostic_location, [])
  when :dot_dot_dot_eol
    Diagnostic.new(:warning, :triple_dot_at_eol, {}, diagnostic_location, [])
  when :duplicated_hash_key
    # skip, parser does this on its own
  else
    PrismDiagnostic.new(warning.message, :warning, warning.type, diagnostic_location)
  end
end