如何在火花数据框列中保留前导零

时间:2017-11-30 03:30:50

标签: scala apache-spark double spark-dataframe

这是我的数据

Source.organizationId|^|Source.sourceId|^|FilingDateTime|^|SourceTypeCode|^|DocumentId|^|Dcn|^|DocFormat|^|StatementDate|^|IsFilingDateTimeEstimated|^|ContainsPreliminaryData|^|CapitalChangeAdjustmentDate|^|CumulativeAdjustmentFactor|^|ContainsRestatement|^|FilingDateTimeUTCOffset|^|ThirdPartySourceCode|^|ThirdPartySourcePriority|^|SourceTypeId|^|ThirdPartySourceCodeId|^|FFAction|!|
4295858966|^|62|^|2017-08-31T11:00:00Z|^|AUTINR|^|44819867|^|cr06305583|^|PDFNTV|^|2017-06-30T00:00:00Z|^|False|^|False|^|2017-06-30T00:00:00Z|^|1.00000|^|False|^|60|^|SS |^|1|^|3011930|^|1000716240|^|I|!|

CumulativeAdjustmentFactor的值为1.00000。 当我加载并打印它时,我得到了

1.0

如何保留相同的值?

架构在

之下
root
 |-- Source_organizationId: long (nullable = true)
 |-- Source_sourceId: integer (nullable = true)
 |-- FilingDateTime: string (nullable = true)
 |-- SourceTypeCode: string (nullable = true)
 |-- DocumentId: integer (nullable = true)
 |-- Dcn: string (nullable = true)
 |-- DocFormat: string (nullable = true)
 |-- StatementDate: string (nullable = true)
 |-- IsFilingDateTimeEstimated: boolean (nullable = true)
 |-- ContainsPreliminaryData: boolean (nullable = true)
 |-- CapitalChangeAdjustmentDate: string (nullable = true)
 |-- CumulativeAdjustmentFactor: double (nullable = true)
 |-- ContainsRestatement: boolean (nullable = true)
 |-- FilingDateTimeUTCOffset: integer (nullable = true)
 |-- ThirdPartySourceCode: string (nullable = true)
 |-- ThirdPartySourcePriority: integer (nullable = true)
 |-- SourceTypeId: integer (nullable = true)
 |-- ThirdPartySourceCodeId: integer (nullable = true)
 |-- FFAction: string (nullable = true)
 |-- DataPartition: string (nullable = true)

当我打印时,我得到低于输出

+---------------------+---------------+--------------------+--------------+----------+--------+---------+--------------------+-------------------------+-----------------------+---------------------------+--------------------------+-------------------+-----------------------+--------------------+------------------------+------------+----------------------+--------+-------------+
|Source_organizationId|Source_sourceId|      FilingDateTime|SourceTypeCode|DocumentId|     Dcn|DocFormat|       StatementDate|IsFilingDateTimeEstimated|ContainsPreliminaryData|CapitalChangeAdjustmentDate|CumulativeAdjustmentFactor|ContainsRestatement|FilingDateTimeUTCOffset|ThirdPartySourceCode|ThirdPartySourcePriority|SourceTypeId|ThirdPartySourceCodeId|FFAction|DataPartition|
+---------------------+---------------+--------------------+--------------+----------+--------+---------+--------------------+-------------------------+-----------------------+---------------------------+--------------------------+-------------------+-----------------------+--------------------+------------------------+------------+----------------------+--------+-------------+
|           4295880476|            228|2016-00-28T03:00:00Z|        JPNARA|      null|    null|     null|2016-00-31T12:00:00Z|                    false|                  false|       2016-00-31T12:00:00Z|                       1.0|              false|                    540|                 SS |                       1|     3012415|            1000716240|       I|        Japan|
|           4295876886|            312|2014-00-25T12:00:00Z|        JPNARA|      null|    null|     null|2014-00-31T12:00:00Z|                    false|                  false|       2015-00-31T12:00:00Z|                       5.0|              false|                      0|                 SS |                       1|     3012415|            1000716240|       I|        Japan|
|           4295878485|             12|2001-00-14T12:00:00Z|        JPNINR|      null|    null|     null|2001-00-30T12:00:00Z|                    false|                  false|       2001-00-30T12:00:00Z|                       0.5|              false|                      0|                 SS |                       1|     3012418|            1000716240|       I|        Japan|
|           4295878485|             17|2002-00-14T12:00:00Z|        JPNINR|      null|    null|     null|2002-00-30T12:00:00Z|                    false|                  false|       2002-00-30T12:00:00Z|                       0.5|              false|                      0|                 SS |                       1|     3012418|            1000716240|       I|        Japan|
|           4295878485|             46|2006-00-26T12:00:00Z|        JPNINR|      null|    null|     null|2005-00-31T12:00:00Z|                    false|                  false|       2005-00-31T12:00:00Z|                       0.5|               true|                      0|                 SS |                       1|     3012418|            1000716240|       I|        Japan|
|           4295876080|            114|2012-00-10T12:00:00Z|        JPNINR|      null|    null|     null|2012-00-01T12:00:00Z|                    false|                  false|       2012-00-01T12:00:00Z|                       0.5|              false|                      0|                 SS |                       1|     3012418|            1000716240|       I|        Japan|
|           4295880487|              7|2001-00-22T12:00:00Z|        JPNINR|      null|    null|     null|2001-00-30T12:00:00Z|                    false|                  false|       2001-00-30T12:00:00Z|                      null|              false|                      0|                 SS |                       1|     3012418|            1000716240|       I|        Japan|
|           4295880487|              9|2002-00-14T12:00:00Z|        JPNINR|      null|    null|     null|2002-00-30T12:00:00Z|                    false|                  false|       2002-00-30T12:00:00Z|                      null|              false|                      0|                 SS |                       1|     3012418|            1000716240|       I|        Japan|
|           4295880487|             10|2002-00-20T12:00:00Z|        JPNINR|      null|    null|     null|2002-00-30T12:00:00Z|                    false|                  false|       2002-00-30T12:00:00Z|                      null|              false|                      0|                 SS |                       1|     3012418|            1000716240|       I|        Japan|
|           4295880534|            128|2012-00-04T12:00:00Z|        JPNINR|      null|    null|     null|2012-00-01T12:00:00Z|                    false|                  false|       2012-00-01T12:00:00Z|                       1.0|              false|                      0|                 SS |                       1|     3012418|            1000716240|       I|        Japan|
|           4295877413|            173|2014-00-28T03:00:00Z|           20F|  34601135|14723203|  ASFILED|2013-00-31T12:00:00Z|                    false|                  false|       2013-00-31T12:00:00Z|                       1.0|              false|                    540|                 SS |                       1|     3011845|            1000716240|       I|        Japan|
|           4295878485|             21|2003-00-30T12:00:00Z|           20F|   1210145|    null|     null|2003-00-31T12:00:00Z|                    false|                  false|       2003-00-31T12:00:00Z|                       0.5|              false|                      0|                 SS |                       1|     3011845|            1000716240|       I|        Japan|
|           4295878485|             28|2004-00-29T12:00:00Z|           20F|   3866028|    null|     null|2004-00-31T12:00:00Z|                    false|                  false|       2004-00-31T12:00:00Z|                       0.5|              false|                      0|                 SS |                       1|     3011845|            1000716240|       I|        Japan|
|           4295878485|             38|2005-00-29T12:00:00Z|           20F|   5538099|    null|     null|2005-00-31T12:00:00Z|                    false|                  false|       2005-00-31T12:00:00Z|                       0.5|              false|                      0|                 SS |                       1|     3011845|            1000716240|       I|        Japan|
|           4295878485|             49|2006-00-28T12:00:00Z|           20F|   7351897|    null|     null|2006-00-31T12:00:00Z|                    false|                  false|       2006-00-31T12:00:00Z|                       0.5|              false|                      0|                 SS |                       1|     3011845|            1000716240|       I|        Japan|
|           4295878485|             55|2007-00-28T12:00:00Z|           20F|   8834460|    null|     null|2007-00-31T12:00:00Z|                    false|                  false|       2007-00-31T12:00:00Z|                      null|              false|                      0|                 SS |                       1|     3011845|            1000716240|       I|        Japan|
|           4295878485|            177|2015-00-25T03:00:00Z|           20F|  38086474|15950616|  ASFILED|2015-00-31T12:00:00Z|                    false|                  false|       2015-00-31T12:00:00Z|                       1.0|              false|                    540|                 SS |                       1|     3011845|            1000716240|       I|        Japan|
|           4295878485|             11|2001-00-04T12:00:00Z|           20F|      null|    null|     null|2001-00-31T12:00:00Z|                    false|                  false|       2001-00-31T12:00:00Z|                       0.5|               true|                      0|                 SS |                       1|     3011845|            1000716240|       I|        Japan|
|           4295878485|             16|2002-00-20T12:00:00Z|           20F|   3556586|    null|     null|2002-00-31T12:00:00Z|                    false|                  false|       2002-00-31T12:00:00Z|                       0.5|              false|                      0|                 SS |                       1|     3011845|            1000716240|       I|        Japan|
|           4295878485|             57|2007-00-25T12:00:00Z|           6K0|   8923021|    null|     null|2007-00-30T12:00:00Z|                    false|                  false|       2007-00-30T12:00:00Z|                      null|              false|                      0|                 SS |                       1|     3011851|            1000716240|       I|        Japan|
+---------------------+---------------+--------------------+--------------+----------+--------+---------+--------------------+-------------------------+-----------------------+---------------------------+--------------------------+-------------------+-----------------------+--------------------+------------------------+------------+----------------------+--------+-------------+

2 个答案:

答案 0 :(得分:0)

由于您的CumulativeAdjustmentFactor列类型为doubleshow方法只需调用toString的{​​{1}},因此无法完成。

double

> 1.0000.toString 1.0 的固有行为。

您可以转换为double类型以显示完整值。

答案 1 :(得分:0)

如果您想将其转换为字符串,那么我们已经通过示例讨论了here的format_number方法。这个方法需要一个数字来获得你想要的精度,并且会在pyspark实现中保留尾随零,scala spark我还没有测试过它。