class Marshal::Structure::Tokenizer

Constants

MAJOR_VERSION

Supported major Marshal version

MINOR_VERSION

Supported minor Marshal version

TYPE_ARRAY

An Array

TYPE_BIGNUM

A Bignum

TYPE_CLASS

A class (reference, not content)

TYPE_DATA

A wrapped C pointer

TYPE_EXTENDED

An object that has been extended with a module

TYPE_FALSE

false type prefix

TYPE_FIXNUM

Fixnum type prefix

TYPE_FLOAT

A Float

TYPE_HASH

A Hash

TYPE_HASH_DEF

A Hash with a default value (not proc)

TYPE_IVAR

Instance variables for a following object

TYPE_LINK

A reference to a previously-stored Object

TYPE_MODULE

A module (reference, not content)

TYPE_MODULE_OLD

An old-style Module (reference, not content)

I'm not sure what makes this old. The byte stream is identical to TYPE_MODULE

TYPE_NIL

nil type prefix

TYPE_OBJECT

A ruby Object

TYPE_REGEXP

A Regexp

TYPE_STRING

A String

TYPE_STRUCT

A Struct

TYPE_SYMBOL

A Symbol

TYPE_SYMLINK

A reference to a previously Symbol

TYPE_TRUE

true type prefix

TYPE_UCLASS

A subclass of a built-in type

TYPE_USERDEF

An object saved with _dump

TYPE_USRMARSHAL

An object saved with marshal_dump

Public Class Methods

new(stream) click to toggle source

Creates a Tokenizer for a Marshal stream in String stream.

# File lib/marshal/structure/tokenizer.rb, line 174
def initialize stream
  @byte_array = stream.bytes.to_a
  @consumed   = 2
  @state      = [:any]
  @stream     = stream
  @stream.force_encoding Encoding::BINARY
end

Public Instance Methods

byte() click to toggle source

Consumes one byte from the marshal stream

# File lib/marshal/structure/tokenizer.rb, line 185
def byte
  raise Marshal::Structure::EndOfMarshal.new(@consumed, 1) if
    @consumed >= @byte_array.size

  data = @byte_array[@consumed]
  @consumed += 1

  data
end

byte_array(count) click to toggle source

Consumes count bytes from the marshal stream as an Array of bytes

# File lib/marshal/structure/tokenizer.rb, line 198
def byte_array count
  bytes(count).bytes.to_a
end

byte_sequence() click to toggle source

Consumes a sequence of bytes from the marshal stream based on the next integer

# File lib/marshal/structure/tokenizer.rb, line 206
def byte_sequence
  size = long
  bytes size
end

bytes(count) click to toggle source

Consumes count from the marshal stream

# File lib/marshal/structure/tokenizer.rb, line 214
def bytes count
  raise Marshal::Structure::EndOfMarshal.new(@consumed, count) if
    @consumed + count > @stream.size

  data = @stream[@consumed, count]
  @consumed += count
  data
end

character() click to toggle source

Consumes one byte from the marshal stream and returns a character

# File lib/marshal/structure/tokenizer.rb, line 226
def character
  byte.chr
end

check_version() click to toggle source

Checks if the stream starts with a compatible marshal version

# File lib/marshal/structure/tokenizer.rb, line 233
def check_version
  major = @stream[0].ord
  minor = @stream[1].ord

  return if major == MAJOR_VERSION and minor <= MINOR_VERSION

  raise TypeError, "incompatible marshal file format (can't be read)\n\tformat version #{MAJOR_VERSION}.#{MINOR_VERSION} required; #{major}.#{minor} given"
end

long() click to toggle source

Decodes a stored C long

# File lib/marshal/structure/tokenizer.rb, line 245
def long
  c = byte

  return 0 if c == 0

  # convert to signed integer
  c = (c ^ 0x80) - 0x80

  if c > 0 then
    return c - 5 if 4 < c

    x = 0

    c.times do |i|
      x |= byte << (8 * i)
    end
    
    x
  else
    return c + 5 if c < -4

    x = -1

    (-c).times do |i|
      factor = 8 * i
      x &= ~(0xff << factor)
      x |= byte << factor
    end

    x
  end
end

tokens() click to toggle source

Returns an Enumerator that will tokenize the Marshal stream.

# File lib/marshal/structure/tokenizer.rb, line 316
def tokens
  check_version

  Enumerator.new do |yielder|
    until @state.empty? do
      token = next_token

      yielder << token if token
    end
  end
end