我有一张包含320万行的表buildings。我需要将此表扩展到11个不同的时段，以(balanced) Paneldata的形式处理它。这意味着每个物体都有11个不同的年份（从2000年至2010年）进行观察。应该称这些时期为：
2000
2001
...
2009
2010
CREATE TABLE public.buildings
(
gid integer NOT NULL DEFAULT nextval('buildings_gid_seq'::regclass),
osm_id character varying(11),
name character varying(48),
type character varying(16),
geom geometry(MultiPolygon,4326),
centroid geometry(Point,4326),
gembez character varying(50),
gemname character varying(50),
krsbez character varying(50),
krsname character varying(50),
pv boolean,
gr smallint,
capac double precision,
instdate date,
pvid integer,
dist double precision,
gemewz integer,
n500 integer,
ibase double precision,
popden integer,
instp smallint,
b2000 double precision,
b2001 double precision,
b2002 double precision,
b2003 double precision,
b2004 double precision,
b2005 double precision,
b2006 double precision,
b2007 double precision,
b2008 double precision,
b2009 double precision,
b2010 double precision,
ibase_id integer[],
ibase_dist integer[],
CONSTRAINT buildings_pkey PRIMARY KEY (gid)
)
WITH (
OIDS=FALSE
);
ALTER TABLE public.buildings
OWNER TO postgres;
CREATE INDEX build_centroid_gix
ON public.buildings
USING gist
(st_transform(centroid, 31467));
CREATE INDEX buildings_geom_idx
ON public.buildings
USING gist
(geom);
我想在 R 中使用数据进行回归分析。
ibase_id是gid的数组。
ibase_dist是一个相关数组，其中gid与obejct的距离。两个数组的长度始终相同。
数组中的gid属于buildings的记录，这些记录位于centroid周围500米的半径范围内，是对象的中心，并且pv = TRUE （这意味着dist，instdate，instp，capac＆amp; pvid为NOT NULL）。
SELECT a.gid AS buildid, array_agg(b.gid) AS ibase_id, array_agg(round(ST_Distance(ST_Transform(a.centroid, 31467), ST_Transform(b.centroid, 31467))::integer)) AS ibase_dist
FROM buildings a
LEFT JOIN (SELECT * FROM buildings WHERE pv=TRUE) AS b ON ST_DWithin(ST_Transform(a.centroid, 31467), ST_Transform(b.centroid, 31467), 500.0)
AND a.gid <> b.gid
GROUP BY a.gid
示例：
ibase_id: {3075528,409073,322311,226643,833798,322344,226609};
ibase_dist {290,293,398,494,411,381,384}
UPDATE buildings
SET ibase=SUM(1/s)
FROM unnest(SELECT ibasedist FROM buildings WHERE (SELECT instp
FROM buildings
WHERE gid IN unnest(ibase_id))<year) s
对于每个时期，只考虑阵列的编号，其年份在面板数据的观察期之前。 （上面的查询不起作用，但是，因为我需要先调用数组）现在，这两个数组保存了所有年份的信息。这就是为什么我认为它们应该被添加到每个时间段，以便在扩展到面板数据之后，我计算每条记录的ibase（11x3,200万）。
我不需要回归分析的所有列。如果它会显着提高乘法的性能，我们可以坚持行（基本上省略了几何列）：
gid integer NOT NULL DEFAULT nextval('buildings_gid_seq'::regclass),
gembez character varying(50),
gemname character varying(50),
krsbez character varying(50),
krsname character varying(50),
pv boolean,
gr smallint,
capac double precision,
dist double precision,
gemewz integer,
n500 integer,
ibase double precision,
popden integer,
instp smallint,
b2000 double precision,
b2001 double precision,
b2002 double precision,
b2003 double precision,
b2004 double precision,
b2005 double precision,
b2006 double precision,
b2007 double precision,
b2008 double precision,
b2009 double precision,
b2010 double precision,
ibase_id integer[],
ibase_dist integer[],
CONSTRAINT buildings_pkey PRIMARY KEY (gid)
)
WITH (
OIDS=FALSE
我的基本想法是创建一个包含11个不同时期的第二个表periods，并将此表与表buildings相乘。不知道如何实现这一点。不幸的是，我对R没有太多经验，也没有使用Database Interface for R。
使用PostgreSQL 9.5beta2，由Visual C ++ build 1800,64位和R x64 3.2.1编译
答案 0 :(得分：1)
基本上，面板数据集是长格式的数据，每个记录的重复年份为时间列。您当前的结构是宽格式。虽然R可以转换这个非常大的数据集，但PostGreSQL可以将所有年份一起堆叠在一个联合查询中，并使用其引擎并将结果集传递给R.请注意，某些数据类型（如几何对象和数组）可能无法正确转换为R数据类型，因此删除它们或将它们转换为字符串/数字类型。
下面是这样一个堆叠年份的SQL UNION查询。我不太清楚您对ibase_id和ibase_dist或＆＃34;乘以＆＃34;的含义方面，但添加了Year列，其中包含相应的b列。让R脚本通过RPostGreSQL模块调用它。
import("RPostgreSQL")
# CREATE CONNECTION
drv <- dbDriver("PostgreSQL")
con <- dbConnect(drv, dbname = "postgres",
host = "localhost", port = ####,
user = "username", password = "password")
strSQL <- "SELECT '2000' As year, gid, gembez, gemname, krsbez,
krsname, pv, gr, capac, dist, gemewz, n500
popden, instp, b2000 As b, (1/ibase_dist) As ibase
FROM public.buildings
INNER JOIN
(SELECT a.gid AS buildid,
SUM(round(ST_Distance(
ST_Transform(a.centroid, 31467),
ST_Transform(b.centroid, 31467)
)::integer)) AS ibase_dist
FROM buildings a
LEFT JOIN buildings b
ON ST_DWithin(ST_Transform(a.centroid, 31467),
ST_Transform(b.centroid, 31467), 500.0)
AND a.gid <> b.gid
WHERE b.pv=True AND b.instp < a.instp
GROUP BY a.gid) AS distSum
ON public.buildings.gid = distSum.buildid
WHERE public.buildings.instp = 2000
UNION
...other SELECT statements for years 2001-2010..."
# IMPORT QUERY RESULTSET INTO DATAFRAME
df <- dbGetQuery(con, strSQL)
# CLOSE CONNECTION
dbDisconnect(con)
但请确保您拥有大数据集操作所需的RAM。您可能需要相应地分配内存。或者，您可以迭代地将每年的SELECT语句附加到不断增长的数据框对象中，而不是一次性加载所有语句。
# ...SAME CONNECTION SETUP AS ABOVE...
years = c('2000', '2001', '2002', '2003', '2004', '2005',
'2006', '2007', '2008', '2009', '2010')
# CREATES LIST OF YEAR DATA FRAME
dfList = lapply(years,
function(y) {
# NOTICE CONCATENATION OF Y IN SELECT STATEMENT
strSQL <- paste0("SELECT '", y, "' As year, gid, gembez, gemname, krsbez,
krsname, pv, gr, capac, dist, gemewz, n500,
popden, instp, b", y, ", As b, (1/ibase_dist) As ibase,
FROM public.buildings
INNER JOIN
(SELECT a.gid AS buildid,
SUM(round(ST_Distance(
ST_Transform(a.centroid, 31467),
ST_Transform(b.centroid, 31467)
)::integer)) AS ibase_dist
FROM buildings a
LEFT JOIN buildings b
ON ST_DWithin(ST_Transform(a.centroid, 31467),
ST_Transform(b.centroid, 31467), 500.0)
AND a.gid <> b.gid
WHERE b.pv=True AND b.instp < a.instp
GROUP BY a.gid) AS distSum
ON public.buildings.gid = distSum.buildid
WHERE public.buildings.instp =", y)
dbGetQuery(con, strSQL)
})
# APPEND LIST OF DATA FRAMES INTO ONE LARGE DATA FRAME
df <- do.call(rbind, dfList)
# REMOVE PREVIOUS LIST FOR MEMORY RESOURCES
rm(dfList)
# CLOSE CONNECTION
dbDisconnect(con)
答案 1 :(得分：0)
我使用带有临时表t1的Cross JOIN创建了Paneldata表，其中包含句点。
CREATE TABLE public.t1
(
period smallint
)
WITH (
OIDS=FALSE
);
CREATE TABLE paneldata AS
(SELECT *
FROM t1 CROSS JOIN
(SELECT gid,
gemname,
gembez,
krsname,
krsbez,
pv,
gr,
capac,
dist,
gemewz,
n500,
popden,
instp
FROM buildings) AS test
ORDER BY gid)