Question

我有一个包含JSON哈希的大文件（> 50Mb）。类似的东西：

{ 
  "obj1": {
    "key1": "val1",
    "key2": "val2"
  },
  "obj2": {
    "key1": "val1",
    "key2": "val2"
  }
  ...
}

我不想解析整个文件并说出前十个元素，而是想解析哈希中的每个项目。我实际上并不关心密钥，即obj1。

如果我将上述内容转换为：

  {
    "key1": "val1",
    "key2": "val2"
  }
  "obj2": {
    "key1": "val1",
    "key2": "val2"
  }

我可以使用Yajl流媒体轻松实现我想要的目标：

io = File.open(path_to_file)
count = 10
Yajl::Parser.parse(io) do |obj|
  puts "Parsed: #{obj}"
  count -= 1
  break if count == 0
end
io.close

有没有办法在不必更改文件的情况下执行此操作？在Yajl中可能会有某种回调吗？

Answer 1

我最终使用JSON::Stream来解决这个问题，{{3}}有start_document，start_object等回调。

我给'我的'解析器'一个to_enum方法，它在解析时发出所有'Resource'对象。请注意，除非您完全解析JSON流，否则ResourcesCollectionNode永远不会被使用，ResourceNode只是ObjectNode的子类，仅用于命名目的，尽管我可能只是去除它：< / p>

class Parser
  METHODS = %w[start_document end_document start_object end_object start_array end_array key value]

  attr_reader :result

  def initialize(io, chunk_size = 1024)
    @io = io
    @chunk_size = chunk_size
    @parser = JSON::Stream::Parser.new

    # register callback methods
    METHODS.each do |name|
      @parser.send(name, &method(name))
    end 
  end

  def to_enum
    Enumerator.new do |yielder|
      @yielder = yielder
      begin
        while !@io.eof?
          # puts "READING CHUNK"
          chunk = @io.read(@chunk_size)
          @parser << chunk
        end
      ensure
        @yielder = nil
      end
    end
  end

  def start_document
    @stack = []
    @result = nil
  end

  def end_document
    # @result = @stack.pop.obj
  end

  def start_object
    if @stack.size == 0
      @stack.push(ResourceCollectionNode.new)
    elsif @stack.size == 1
      @stack.push(ResourceNode.new)
    else
      @stack.push(ObjectNode.new)
    end
  end

  def end_object
    if @stack.size == 2
      node = @stack.pop
      #puts "Stack depth: #{@stack.size}. Node: #{node.class}"
      @stack[-1] << node.obj

      # puts "Parsed complete resource: #{node.obj}"
      @yielder << node.obj

    elsif @stack.size == 1
      # puts "Parsed all resources"
      @result = @stack.pop.obj
    else
      node = @stack.pop
      # puts "Stack depth: #{@stack.size}. Node: #{node.class}"
      @stack[-1] << node.obj
    end
  end

  def end_array
    node = @stack.pop
    @stack[-1] << node.obj
  end

  def start_array
    @stack.push(ArrayNode.new)
  end

  def key(key)
    # puts "Stack depth: #{@stack.size} KEY: #{key}"
    @stack[-1] << key
  end

  def value(value)
    node = @stack[-1]
    node << value
  end

  class ObjectNode
    attr_reader :obj

    def initialize
      @obj, @key = {}, nil
    end

    def <<(node)
      if @key
        @obj[@key] = node
        @key = nil
      else
        @key = node
      end
      self
    end
  end

  class ResourceNode < ObjectNode
  end

  # Node that contains all the resources - a Hash keyed by url
  class ResourceCollectionNode < ObjectNode
    def <<(node)
      if @key
        @obj[@key] = node
        # puts "Completed Resource: #{@key} => #{node}"
        @key = nil
      else
        @key = node
      end
      self
    end
  end

  class ArrayNode
    attr_reader :obj

    def initialize
      @obj = []
    end

    def <<(node)
      @obj << node
      self
    end
  end

end

以及使用中的示例：

def json
  <<-EOJ
  {
    "1": {
      "url": "url_1",
      "title": "title_1",
      "http_req": {
        "status": 200,
        "time": 10
      }
    },
    "2": {
      "url": "url_2",
      "title": "title_2",
      "http_req": {
        "status": 404,
        "time": -1
      }
    },
    "3": {
      "url": "url_1",
      "title": "title_1",
      "http_req": {
        "status": 200,
        "time": 10
      }
    },
    "4": {
      "url": "url_2",
      "title": "title_2",
      "http_req": {
        "status": 404,
        "time": -1
      }
    },
    "5": {
      "url": "url_1",
      "title": "title_1",
      "http_req": {
        "status": 200,
        "time": 10
      }
    },
    "6": {
      "url": "url_2",
      "title": "title_2",
      "http_req": {
        "status": 404,
        "time": -1
      }
    }          

  }
  EOJ
end


io = StringIO.new(json)
resource_parser = ResourceParser.new(io, 100)

count = 0
resource_parser.to_enum.each do |resource|
  count += 1
  puts "READ: #{count}"
  pp resource
  break
end

io.close

输出：

READ: 1
{"url"=>"url_1", "title"=>"title_1", "http_req"=>{"status"=>200, "time"=>10}}

Answer 2

我遇到了同样的问题并创建了宝石json-streamer，这将节省您创建自己的回调的需要。

您案例中的用法是（v 0.4.0）：

io = File.open(path_to_file)
streamer = Json::Streamer::JsonStreamer.new(io)
streamer.get(nesting_level:1).each do |object|
  p oject
end
io.close

在你的例子中应用它会产生没有'obj'键的对象：

{
  "key1": "val1",
  "key2": "val2"
}

使用ruby-yajl解析大型JSON哈希？

2 个答案: