Question

我正在读取一个csv文件，以存储在一个不可变的数据结构中。每排都是一个入口。每个入口都有一个车站。每个车站可以有多个入口。有没有办法我可以在一次通过而不是你在下面看到的双通过来做到这一点？

object NYCSubwayEntrances {
  def main(args: Array[String]) = {
    import com.github.tototoshi.csv.CSVReader
    //http://www.mta.info/developers/data/nyct/subway/StationEntrances.csv
    val file = new java.io.File("StationEntrances.csv")
    val reader = CSVReader.open(file)
    reader.readNext //consume headers
    val entranceMap = list2multimap(
      reader.all map {
        case fields: List[String] => 
          // println(fields)
          (
            fields(2), 
            Entrance(
              fields(14).toBoolean,
              Option(fields(15)),
              fields(16).toBoolean,
              fields(17),
              fields(18) match {case "YES" => true case _ => false},
              fields(19) match {case "YES" => true case _ => false},
              fields(20),
              fields(21),
              fields(22),
              fields(23),
              fields(24).toInt,
              fields(25).toInt
            )
          )
        }
      )
    reader.close
    val reader2 = CSVReader.open(file)
    reader2.readNext //consume headers
    val stations = reader2.all map { case fields: List[String] =>
      Station(
        fields(2),
        fields(0),
        fields(1),
        colate(scala.collection.immutable.ListSet[String](
          fields(3),
          fields(4),
          fields(5),
          fields(6),
          fields(7),
          fields(8),
          fields(9),
          fields(10),
          fields(11),
          fields(12),
          fields(13)
        )),
        entranceMap(fields(2)).toList
      )
    }
    reader2.close

    import net.liftweb.json._
    import net.liftweb.json.Serialization.write
    implicit val formats = Serialization.formats(NoTypeHints)
    println(pretty(render(parse(write(stations.toSet)))))
  }

  import scala.collection.mutable.{HashMap, Set, MultiMap}

  def list2multimap[A, B](list: List[(A, B)]) = 
    list.foldLeft(new HashMap[A, Set[B]] with MultiMap[A, B]){(acc, pair) => acc.addBinding(pair._1, pair._2)}

  def colate(set: scala.collection.immutable.ListSet[String]): List[String] =
    ((List[String]() ++ set) diff List("")).reverse
}

case class Station(name: String, division: String, line: String, routes: List[String], entrances: List[Entrance]) {}
case class Entrance(ada: Boolean, adaNotes: Option[String], freeCrossover: Boolean, entranceType: String, entry: Boolean, exitOnly: Boolean, entranceStaffing: String, northSouthStreet: String, eastWestStreet: String, corner: String, latitude: Integer, longitude: Integer) {}

可以在以下位置找到具有所有正确依赖关系的sbt项目 https://github.com/AEtherSurfer/NYCSubwayEntrances

StationEntrances.csv是从http://www.mta.info/developers/sbwy_entrance.html

获得的

Answer 1

我有以下代码段。第一个解决方案使用groupBy对与同一站点相关的入口进行分组。它不假设行已排序。虽然它只读取一次文件，但它确实有3次传递（一次读取所有内存，一次读取groupBy，另一次读取创建工作站）。请参阅最后的Row提取器代码。

val stations = {
  val file = new java.io.File("StationEntrances.csv")
  val reader = com.github.tototoshi.csv.CSVReader.open(file)
  val byStation = reader
    .all     // read all in memory
    .drop(1) // drop header
    .groupBy {
      case List(division, line, station, _*) => (division, line, station)
    }
  reader.close
  byStation.values.toList map { rows =>
    val entrances = rows map { case Row(_, _, _, _, entrance) => entrance }
    rows.head match {
      case Row(division, line, station, routes, _) =>
        Station(
          division, line, station,
          routes.toList.filter(_ != ""),
          entrances)
    }
  }
}

此解决方案假定行已排序且应该更快，因为它只执行一次并在读取文件时构建结果列表。

val stations2  = {
  import collection.mutable.ListBuffer
  def processByChunk(iter: Iterator[Seq[String]], acc: ListBuffer[Station])
           : List[Station] = {
    if (!iter.hasNext) acc.toList
    else {
      val head = iter.next
      val marker = head.take(3)
      val (rows, rest) = iter.span(_ startsWith marker)
      val entrances = (head :: rows.toList) map {
        case Row(_, _, _, _, entrance) => entrance
      }
      val station = head match {
        case Row(division, line, station, routes, _) =>
          Station(
            division, line, station,
            routes.toList.filter(_ != ""),
            entrances)
      }
      processByChunk(rest, acc += station)
    }
  }
  val file = new java.io.File("StationEntrances.csv")
  val reader = com.github.tototoshi.csv.CSVReader.open(file)
  val stations = processByChunk(reader.iterator.drop(1), ListBuffer())
  reader.close
  stations
}

我创建了一个专用的提取器来获取给定线路的路径/入口。我认为它使代码更具可读性，但是如果你正在处理list，那么调用fields(0)到fields(25)并不是最佳的，因为每个调用都必须遍历列表。提取器避免这种情况。对于大多数Java csv解析器，您通常会得到Array[String]，因此通常不会出现问题。最后，csv解析通常不返回空字符串，因此您可能希望使用if (adaNotes == "") None else Some(adaNotes)而不是Option(adaNotes)。

object Row {
  def unapply(s: Seq[String]) = s match {
    case List(division, line, station, rest @ _*) =>
      val (routes,
        List(ada, adaNotes, freeCrossover, entranceType,
          entry, exitOnly, entranceStaffing, northSouthStreet, eastWestStreet,
          corner, latitude, longitude)) = rest splitAt 11 // 11 routes
      Some((
        division, line, station,
        routes,
        Entrance(
          ada.toBoolean, Option(adaNotes),
          freeCrossover.toBoolean, entranceType,
          entry == "YES", exitOnly == "YES",
          entranceStaffing, northSouthStreet, eastWestStreet, corner,
          latitude.toInt, longitude.toInt)))
    case _ => None
  }
}

优化csv文件的非规范化

1 个答案: