Question

我试图在IPython扩展模块中使用TokenInputTransformer，但似乎有多行输入的令牌变换器的标准实现有问题。考虑以下最小扩展名：

from IPython.core.inputtransformer import TokenInputTransformer

@TokenInputTransformer.wrap
def test_transformer(tokens):
    return tokens

def load_ipython_extension(ip):
    for s in (ip.input_splitter, ip.input_transformer_manager): 
        s.python_line_transforms.extend([test_transformer()])
    print "Test activated"

当我在IPython 1.1.0中加载扩展时，我得到一个带有多行输入的非处理异常：

In [1]: %load_ext test
Test activated

In [2]: abs(
   ...: 2
   ...: )
Traceback (most recent call last):
  File "/Applications/anaconda/bin/ipython", line 6, in <module>
    sys.exit(start_ipython())
  File "/Applications/anaconda/lib/python2.7/site-packages/IPython/__init__.py", line 118, in start_ipython
    return launch_new_instance(argv=argv, **kwargs)
  File "/Applications/anaconda/lib/python2.7/site-packages/IPython/config/application.py", line 545, in launch_instance
    app.start()
  File "/Applications/anaconda/lib/python2.7/site-packages/IPython/terminal/ipapp.py", line 362, in start
    self.shell.mainloop()
  File "/Applications/anaconda/lib/python2.7/site-packages/IPython/terminal/interactiveshell.py", line 436, in mainloop
    self.interact(display_banner=display_banner)
  File "/Applications/anaconda/lib/python2.7/site-packages/IPython/terminal/interactiveshell.py", line 548, in interact
    self.input_splitter.push(line)
  File "/Applications/anaconda/lib/python2.7/site-packages/IPython/core/inputsplitter.py", line 620, in push
    out = self.push_line(line)
  File "/Applications/anaconda/lib/python2.7/site-packages/IPython/core/inputsplitter.py", line 655, in push_line
    line = transformer.push(line)
  File "/Applications/anaconda/lib/python2.7/site-packages/IPython/core/inputtransformer.py", line 152, in push
    return self.output(tokens)
  File "/Applications/anaconda/lib/python2.7/site-packages/IPython/core/inputtransformer.py", line 157, in output
    return untokenize(self.func(tokens)).rstrip('\n')
  File "/Applications/anaconda/lib/python2.7/site-packages/IPython/utils/_tokenize_py2.py", line 276, in untokenize
    return ut.untokenize(iterable)
  File "/Applications/anaconda/lib/python2.7/site-packages/IPython/utils/_tokenize_py2.py", line 214, in untokenize
    self.add_whitespace(start)
  File "/Applications/anaconda/lib/python2.7/site-packages/IPython/utils/_tokenize_py2.py", line 199, in add_whitespace
    assert row >= self.prev_row
AssertionError

If you suspect this is an IPython bug, please report it at:
    https://github.com/ipython/ipython/issues
or send an email to the mailing list at ipython-dev@scipy.org

You can print a more detailed traceback right now with "%tb", or use "%debug"
to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via:
    %config Application.verbose_crash=True

我做错了什么或者它真的是一个IPython错误？

Answer 1

我认为这确实是一个IPython错误。具体来说，当涉及括号（()[]{}）的表达式分布在多行上时，我们处理标记化的方式失败。我正试图弄清楚我们能做些什么。

Answer 2

有点迟到但是，我试图在我自己的扩展中使用它，并且遇到了同样的问题。我只是通过从列表中删除NL来解决它（它与NEWLINE令牌不同于哪个结束语句），NL令牌只出现在[]，（），{}内部所以它应该可以安全地移除

from tokenize import NL
@TokenInputTransformer.wrap
def mat_transformer(tokens):
    tokens = list(filter(lambda t: t.type != NL, tokens)) 
    return tokens

如果您正在寻找完整的示例，我已经在那里发布了我的愚蠢代码：https://github.com/Quinzel/pymat

IPython扩展中的多行TokenInputTransformer问题

2 个答案: