我正在尝试使用shell脚本从csv文件将数据加载到Oracle表中。我想转置数据并将其加载到我的oracle表中。
CSV文件数据格式：以下是我收到的传入数据。
Date ,Emp_name,Math_zero,Math_max,Math_min,Math_avg,English_zero,English_max,English_min,English_avg
20161005,abc ,0 ,1 ,0 ,0 ,0 ,1 ,0 ,0
20161005,def ,0 ,1 ,0 ,0 ,0 ,1 ,0 ,0
将数据加载到Oracle表时所需的数据格式：
Date ,emp_name,subject,subject_zero,subject_max,subject_min,subject_avg
20161005,abc ,Math ,0 ,1 ,0 ,0
20161005,def ,English,0 ,1 ,0 ,0
我的Oracle表有七列：
Date
Emp_name
Subject
Subject_zero
Subject_max
Subject_min
Subject_avg
请建议一些shell脚本将传入的数据转换为所需的格式。
由于 AGGARWAL
答案 0 :(得分：0)
在将字符串（原样）导入数据库之后，可以在SQL中完成此操作。但是，你的输入字符串有一个可怕的结构，因此解开它很复杂。
您的“所需输出”只有两行;不应该有四行，因为每个emp_name都有两个科目的分数？我的输出有四行。我还稍微更改了测试数据，以确保正确的值最终出现在正确的行和列中;对于emp_name / date和subject的所有组合，得分为“0,1,0,0”的顺序对调试没有帮助。
我假设输入字符串可能包含NULL（如下所示：'1,0,1,1 ,,,,,“ - 两个连续的逗号表示NULL）对于某些emp_name /日期和主题。这就是为什么SUBJ _....值的搜索模式非常复杂。
解决方案：
with
test_data ( str ) as (
select 'Date,Emp_name,Math_zero,Math_max,Math_min,Math_avg,'
|| 'English_zero,English_max,English_min,English_avg' || chr(10)
|| '20161005,abc,0,1,0,3,0,1,0,0' || chr(10)
|| '20161005,def,1,1,0,0,0,1,5,0'
from dual
),
line_split ( line_no, line_str ) as (
select level, regexp_substr(str, '[^' || chr(10) || ']+', 1, level)
from test_data
connect by level <= 1 + regexp_count(str, chr(10))
),
cols ( idx, col ) as (
select level, regexp_substr(line_str, '[^,]+', 1, level)
from ( select line_str from line_split where line_no = 1 )
connect by level <= 1 + regexp_count(line_str, ',')
),
subjects ( idx, subj ) as (
select ceil(idx/4), substr(col, 1, instr(col, '_') - 1)
from cols
where mod(idx, 4) = 3
)
select to_date(regexp_substr(l.line_str, '[^,]+', 1, 1), 'yyyymmdd') as dt,
regexp_substr(l.line_str, '[^,]+', 1, 2) as emp_name,
subj as subject,
to_number(regexp_substr(l.line_str, '([^,]*)(,|$)', 1, 4 * s.idx - 1, null, 1)) as subj_zero,
to_number(regexp_substr(l.line_str, '([^,]*)(,|$)', 1, 4 * s.idx , null, 1)) as subj_max,
to_number(regexp_substr(l.line_str, '([^,]*)(,|$)', 1, 4 * s.idx + 1, null, 1)) as subj_min,
to_number(regexp_substr(l.line_str, '([^,]*)(,|$)', 1, 4 * s.idx + 2, null, 1)) as subj_avg
from line_split l cross join subjects s
where l.line_no > 1
;
输出（包含我的测试数据）：
DT EMP_NAME SUBJECT SUBJ_ZERO SUBJ_MAX SUBJ_MIN SUBJ_AVG
---------- -------- ---------- ---------- ---------- ---------- ----------
2016-10-05 abc Math 0 1 0 3
2016-10-05 abc English 0 1 0 0
2016-10-05 def Math 1 1 0 0
2016-10-05 def English 0 1 5 0
4 rows selected.
答案 1 :(得分：0)
我认为最好的方法是使用http://www.sql-workbench.net/如果你有JAVA（1.6或1.7）那么你可以准备wbExport脚本并从命令行shell执行它。该工具可以通过GUI和命令行以两种方式工作。您也可以将它与shell集成：
LOG_FILE = test.log中
触摸$ {LOG_FILE}
echo "`date +%Y-%m-%d\ %H:%M:%S` STEP020 : Gathering script settings from database (BEGIN)" | tee -a ${LOG_FILE}
## DATABASE CONNECTION
CONFIG_SQL_FILE=${BASE_DIR}/deamon/sql/${DB2_CFG_SCRIPT}
truncate -s0 ${DB2_CFG_LOG} > /dev/null
echo " WbExport -file=${BASE_DIR}/deamon_settings.csv " > ${CONFIG_SQL_FILE}
echo " -type=text " >> ${CONFIG_SQL_FILE}
echo " -encoding='UTF-8' " >> ${CONFIG_SQL_FILE}
echo " -lineEnding='lf' " >> ${CONFIG_SQL_FILE}
echo " -header=false " >> ${CONFIG_SQL_FILE}
echo " -append=false " >> ${CONFIG_SQL_FILE}
echo " -dateFormat='yyyy-MM-dd' " >> ${CONFIG_SQL_FILE}
echo " -delimiter=';' " >> ${CONFIG_SQL_FILE}
echo " -timestampFormat='yyyy-MM-dd HH:mm:ss' " >> ${CONFIG_SQL_FILE}
echo " ; " >> ${CONFIG_SQL_FILE}
echo " " >> ${CONFIG_SQL_FILE}
echo "SELECT " >> ${CONFIG_SQL_FILE}
echo " HOST_ID, HOST_NAME, SCRIPT_PATH, CREATE_TS, UPDATE_TS, " >> ${CONFIG_SQL_FILE}
echo " HOST_DESC, HOST_MONIT_FREQ,HOST_MONIT_ENABLED, " >> ${CONFIG_SQL_FILE}
echo " PROCESS_MONIT_FREQ, PROCESS_MONIT_ENABLED " >> ${CONFIG_SQL_FILE}
echo "FROM " >> ${CONFIG_SQL_FILE}
echo " ${DB2_USERNAME}.SOME_TABLE " >> ${CONFIG_SQL_FILE}
echo "WHERE " >> ${CONFIG_SQL_FILE}
echo " LOWER(HOST_NAME) LIKE LOWER('${HOST_NAME}') AND " >> ${CONFIG_SQL_FILE}
echo " LOWER(SCRIPT_PATH) LIKE LOWER('${BASE_DIR}') " >> ${CONFIG_SQL_FILE}
echo "FETCH FIRST 1 ROWS ONLY; --limit to 1 row " >> ${CONFIG_SQL_FILE}
echo " " >> ${CONFIG_SQL_FILE}
echo " COMMIT; " >> ${CONFIG_SQL_FILE}
echo " " >> ${CONFIG_SQL_FILE}
if [ -f ${CONFIG_SQL_FILE} ]; then
echo "`date +%Y-%m-%d\ %H:%M:%S` STEP020 : Failed to store setting file: ${CONFIG_SQL_FILE}" | tee -a ${LOG_FILE}
echo "`date +%Y-%m-%d\ %H:%M:%S` STEP020 : Gathering script settings from database (FAILED)" | tee -a ${LOG_FILE}
exit 4;
fi
## DATABASE CONNECTION USED HERE
java -Djava.awt.headless=true -Xmx256m -classpath ${WORKBENCH_PATH} \
-jar ${DB2_JAR} \
-driverjar=${DB2_DRIVER} \
-username=${DB2_USERNAME} \
-password=${DB2_PASSWORD} \
-url=${DB2_URL} \
-script=${CONFIG_SQL_FILE} \
-logfile=${DB2_CFG_LOG} \
-fetchSize=100 \
-nosettings > /dev/null 2>&1
cat ${CONFIG_SQL_FILE} | sed "s/^/`date +%Y-%m-%d\ %H:%M:%S` STEP020 : WORKBENCH /" | tee -a ${LOG_FILE}
if [ `cat ${DB2_CFG_LOG} | grep Error | wc -l` -gt 0 ]; then
echo "`date +%Y-%m-%d\ %H:%M:%S` STEP020 : Failed to retrieve meta data (FAILED)" | tee -a ${LOG_FILE}
exit 5
fi
echo "`date +%Y-%m-%d\ %H:%M:%S` STEP020 : Gathering script settings from database (END-OK)" | tee -a ${LOG_FILE}
您必须了解更多SQLWorkbench才能使用它，但肯定它是现在最好的工具。您还必须将jdbc下载到目标数据库。