不可靠/ Flakey Capybara / AngularJS集成测试时序问题

时间:2016-02-23 02:58:14

标签: rspec phantomjs capybara integration-testing poltergeist

如何让这些测试可靠地通过?

目前这些测试很有趣 有时他们会过去。有时他们会失败 以下是演示此问题的设置,代码和输出 我们非常感谢您克服这个问题的建议,我相信会帮助很多其他人,所以请发表评论!

测试代码环境

  1. Rails 3.2
  2. RSpec 2.x
  3. Capybara
  4. Poltergeist
  5. PhantomJS
  6. AngularJS
  7. Google Chrome版本47.0.2526.106(64位)
  8. 从Gemfile.lock

    测试Gems
    capybara (2.1.0)
    database_cleaner (0.7.1)
    debug_inspector (0.0.2)
    guard-bundler (0.1.3)
    guard-livereload (1.2.0)
    guard-rspec (2.1.2)
    jasminerice (0.0.10)
    pg (0.17.1)
    phantomjs (2.1.1.0)
    poltergeist (1.4.1)
    protractor-rails (0.0.17)
    pry (0.9.12)
    rack (1.4.7)
    rack-test (0.6.3)
    rails (3.2.21)
    rails-assets-angular (1.3.20)
    rspec-rails (2.11.4)
    simplecov (0.8.2)
    sprockets (2.2.3)
    zeus (0.13.3)
    zeus-parallel_tests (0.2.1)
    

    我尝试过的事情

    1. 确保我使用Capybara的等待DSL匹配器
    2. 确保我的数据库清理程序已正确设置
    3. 测试每个页面项,假设它可能不在页面上并且仍然可以加载
    4. 缩小不一致的测试
    5. 单独运行不一致的测试
    6. 识别代码Capybara DSL,它们是不一致的测试结果的触发器。

      • 即。创建新记录并假设页面已重定向,并且记录位于页面click_on

      • 。点击不一致'工作'
    7. 将Capybara升级到最新版本(在单独的分支中)
    8. 将Poltergeist和RSpec升级到最新版本(在一个单独的分支中,仍然在努力)
    9. 我使用的资源

      [1] Capybara The DSL
      [2] Capybara, PhantomJs, Poltergeist, and Rspec Tips
      还有更多...

      如何运行测试

      rspec spec/integration/costings/show_costing_spec.rb --format documentation

      测试代码

      show_costing_spec.rb
      require "spec_helper"
      
      RSpec.describe "Show a new costing in the listing," do
      
        before :each do
          admin_sign_in
          create_costing("test1")
        end
      
        it "shows the costing after creation" do
          within "#costings_table" do
            expect(page).to have_css("#name", text: "test1")
          end
        end
      
        it "shows the details of the new costing after creation" do
          expect(page).to have_content("Costings")
          within "#costings_table" do
            expect(page).to have_content("test1")
            all("#show").last.click
          end
      
          expect(page).to have_content("Costing Details")
          expect(page).to have_css("#name", text: "test1")
        end
      end  
      
      spec_helper.rb
      # This file is copied to spec/ when you run 'rails generate r spec:install'  
      ENV["RAILS_ENV"] ||= 'test'
      require File.expand_path("../../config/environment", __FILE__)
      # Add library functions here so we can test them.
      require File.expand_path(File.dirname(__FILE__) + "/../lib/general")
      require 'rspec/rails'
      require 'rspec/autorun'
      
      # Integration Testing
      require 'capybara/poltergeist'
      Capybara.register_driver :poltergeist_debug do |app|
       Capybara::Poltergeist::Driver.new(app, :inspector => true)  
      end
      Capybara.javascript_driver = :poltergeist_debug
      Capybara.default_driver = :poltergeist_debug
      
      # Capybara Integration Test Helpers
      def admin_sign_in
        visit "/login"
        #Create staff member in database
        Staff.make!(:admin)
        #Log In
        fill_in "staff_username", with: "adminstaff"
        fill_in "staff_password", with: "password"
        click_button "login"
      end
      
      def create_costing(item)
        visit "/api#/costings"
        click_on "new_btn"
        within "#form_costing" do
          find("#name", match: :first).set("#{item}")
          find("#description", match: :first).set("test description")    
          find("#from_date", match: :first).set("15/02/2016")
          find("#cost_hourly_cents", match: :first).set("1.00")
          click_on "create_btn"
        end
      end
      
      RSpec.configure do |config|
        config.before(:suite) do
          # Requires supporting ruby files with custom matchers and macros, etc,
          # in spec/support/ and its subdirectories.
          require File.expand_path(File.dirname(__FILE__) + "/support/blueprints")
          Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}
        end
      
        # Remove this line if you're not using ActiveRecord or ActiveRecord fixtures
        config.fixture_path = "#{::Rails.root}/spec/fixtures"
      
        # Allow a 'focus' tag so that we can run just a few tests which we are currently working on
        config.treat_symbols_as_metadata_keys_with_true_values = true
        config.filter_run focus: true
        config.run_all_when_everything_filtered = true
        config.filter_run_excluding :slow unless ENV["SLOW_SPECS"]
      
        # Defer Garbage Collection
        config.before(:all) { DeferredGarbageCollection.start }
        config.after(:all)  { DeferredGarbageCollection.reconsider }
      
        # If you're not using ActiveRecord, or you'd prefer not to run each of your
        # examples within a transaction, remove the following line or assign false
        # instead of true.
        config.use_transactional_fixtures = false
        # config.infer_spec_type_from_file_location!
      
        # Configure Database Cleaner
        config.include Capybara::DSL
        config.before(:suite) do
          DatabaseCleaner.clean_with(:truncation)
        end
      
        config.before(:each) do
          DatabaseCleaner.strategy = :transaction
        end
      
        config.before(:each, :js => true) do
          DatabaseCleaner.strategy = :truncation
        end
      
        config.before(:each) do
          DatabaseCleaner.start
        end
      
        config.after(:each) do
          DatabaseCleaner.clean
        end
      end
      

      测试结果

      Test Run 1: Failing

        

      运行选项:   包括{:focus => true}   排除{:slow => true}

           

      所有例子都被过滤掉了;忽略{:focus => true}

           

      在商家信息中显示新的费用,     显示创建后的成本核算     显示创建后新成本核算的详细信息(FAILED - 1)

           

      故障:

           

      1)在列表中显示新的成本核算,
           显示创建后新成本核算的详细信息        失败/错误:期望(页面).to have_content(" test1")
               期望#has_content?(" test1")返回true,得到假        #./spec/integration/costings/show_costing_spec.rb:20:in block(3 levels)in
             <./spec/integration/costings/show_costing_spec.rb:19:in块(2级)in

           

      在5.46秒内完成   2个例子,1个失败

      Test Run 2: Passing

        

      运行选项:      包括{:focus =&gt; true}      排除{:slow =&gt; true}

           

      所有例子都被过滤掉了;忽略{:focus =&gt; true}

           

      在列表中显示新的成本核算,
          显示创建后的成本核算     显示创建后新成本核算的详细信息

           

      在3.57秒内完成    2个例子,0个失败

      更新1

      将测试宝石升级到以下版本:
          水豚(2.6.2)来自(2.1.0)
           database_cleaner(1.5.1)from(0.7.1)
          debug_inspector(0.0.2)
          后卫捆绑(0.1.3)
          guard-livereload(1.2.0)
          guard-spec(2.1.2)
          jasminerice(0.0.10)
          第0页(0.17.1)
          phantomjs(2.1.1.0)
           poltergeist(1.9.0)(1.4.1)
          量角器轨道(0.0.17)
           pry(0.10.3)来自(0.9.12)
          机架(1.4.7)
          机架测试(0.6.3)
          铁轨(3.2.21)
           rails-assets-angular(1.4.9)from(1.3.20)
          来自(2.11.4)的 rspec-rails(3.4.2)
          simplecov(0.8.2)
          链轮(2.2.3)
          宙斯(0.13.3)
          zeus-parallel_tests(0.2.1)

      Result 1

        

      不幸的是升级这些宝石似乎没有什么区别,我的测试仍然很好。

      更新2

      我实施了Tom Walpole的建议。确保我的admin_sign_in等待sign_in完成。

      还更新了我建议的database_cleaner设置。

      Result 2

        

      对于我的堆栈,这些更改似乎没有效果。

           

      注意:如果没有使用AngularJS,我觉得这些变化会有所不同。谢谢汤姆的建议。

      更新3

      我需要了解有关测试运行期间发生的事情的更多信息。网上有建议记录,使用屏幕保护宝石等,但我不觉得这些是最有效的时间。我想指定我希望测试暂停的位置,并查看变量和表单字段的内容。在浏览器中是理想的。

      我使用的是什么
      我正在使用&#34; save_and_open_page&#34;和&#34;打印page.html&#34;调试。

      我感动的内容
      当我运行RSpec 3.4.2时,我添加了一个调试辅助方法:

      rails_helper.rb

      def debugit
        puts current_url
        require 'pry'
        binding.pry
      end
      

      Result 3

        

      将在控制台中打印一个URL,测试将暂停。在这个阶段,我将能够导航到URL,登录,导航到测试页面并查看Capybara测试所做的事情。

           

      这使我能够确定在测试使用capybara的fill_in DSL时出现问题的根源。    在某些测试运行中,将正确填充字段并提交表单。   在另一种情况下,表单将被正确填写,但提交按钮会被击中太快。这里的结果是创建了一条记录,但没有保留名称和描述的输入字段。

      更新4

      我发现因为我使用的是AngularJS输入表单和表格,AngularJS需要一点时间来绑定到输入字段。如果此时不允许输入数据将不被保存。

      Capybara提供等待方法,如&#34;在&#34;和&#34;找到&#34;。我使用过这些,但他们没有帮助解决AngularJS绑定时间问题。 我发现ng-if可以用来创建一个if语句来等待表示AngularJS绑定到表单字段完成的特定项目。

      所以我用Capybara等待方法等待我想要填充的字段,并使用了AngularJS&#39; ng-如果在准备好之前不显示字段。

      实施
      index.html.erb

      <div  ng-if="tableParams.data">
        <table id="costings_table ng-table="tableParams" class="table">
          <td id="field1">{{table.field1}}</td>
          <td id="field2">{{table.field2}}</td>
        </table>
      </div>
      

      Result 4

        

      测试终于过去了!但是,我有所有这些使用xpath查找方法确保特定和难以定位的项目等待...

      更新5

      即使在我的gemfile中我运行了gem phantomJS 2.1.1版,我的命令行版本只有1.X.事实证明这很重要。

      我将命令行phantomJS版本更新为2.1.1。同时我确保所有输入框,按钮,表格,标题都有独特的ID。然后,我能够在不破坏测试的情况下删除所有find(:xpath)。

      Result 5

        

      这套测试现在可靠地传递!正是我想要的!是的!

2 个答案:

答案 0 :(得分:2)

跳出来的最直接的事情是你的admin_sign_in实际上并没有等到sign_in完成。这意味着您可以在未在浏览器中设置会话cookie的情况下调用create_costingadmin_sign_in方法中的最后一行应该是

expect(page).to have_text('You are signed in') # whatever message is shown upon sign in

expect(page).to have_current_path('/') # whatever path an admin is redirected to upon signing in

这将确保登录已实际完成,因此会话cookie已在您的浏览器中设置。

此外,您的数据库清理程序配置应使用append_after块而不是之后 - 请参阅https://github.com/DatabaseCleaner/database_cleaner#rspec-with-capybara-example

答案 1 :(得分:1)

问题

当测试使用capybara的fill_in DSL时出现问题。在某些测试运行中,将正确填充字段并提交表单。在另一种情况下,表单将被正确填写,但提交按钮会被击中太快。这里的结果是创建了一条记录,但没有保留名称和描述的输入字段。

1。填写表格时,确保完成AngularJS绑定并使用Capybara等待方法

AngularJS&#39;在准备就绪之前,需要使用ng-if语句不显示表单字段 这需要与使用Capybara等待方法一起完成,以确保只有在表单加载完成后才提交fill_in字段。

index.html.erb或同等资料:

<div  ng-if="tableParams.data">
  <table id="costings_table ng-table="tableParams" class="table">
    <td id="field1">{{table.field1}}</td>
    <td id="field2">{{table.field2}}</td>
   </table>
</div>

2。将PhantomJS的命令行版本更新为最新版本(2.1.1)

这似乎使测试能够在没有尽可能多的Capybara等待方法的情况下运行,以实现可靠的测试。

更新了测试代码
show_costing_spec.rb

require "rails_helper"

RSpec.describe "Show a new costing in the listing,", :type => :feature do

  before :each do
    admin_sign_in
    create_costing("test1")
  end

  it "shows the costing after creation" do
    within "#costings_table" do
      expect(page.find("#code2")).to have_content("2")
      expect(page.find("#name2")).to have_content("test1")
    end
  end

  it "shows the details of the new costing after creation" do
    within "#costings_table" do
      click_on "show2"
    end

    expect(page.find("#page_title")).to have_content("Costing Details")
    expect(page.find("#code")).to have_content("2")
    expect(page.find("#name")).to have_content("test1") 
    expect(page.find("#description")).to have_content("test description")
  end
end

rails_helper.rb

# This file is copied to spec/ when you run 'rails generate rspec:install'
ENV["RAILS_ENV"] ||= 'test'
require File.expand_path("../../config/environment", __FILE__)

# Add library functions here so we can test them.
require File.expand_path(File.dirname(__FILE__) + "/../lib/general")

require 'rspec/rails'
require 'devise'

RSpec.configure do |config|
  config.before(:suite) do
    # Requires supporting ruby files with custom matchers and macros, etc,
    # in spec/support/ and its subdirectories.
    require File.expand_path(File.dirname(__FILE__) + "/support/blueprints")
    Dir[Rails.root.join("spec/support/**/*.rb")].each {|f| require f}

    # Setup Devise before it is used in rails_helper
    config.include Devise::TestHelpers, :type => :controller
    Devise.stretches = 1 # Improves speed.
end

config.include Capybara::DSL, :type => :feature
  config.mock_with :rspec

# Remove this line if you're not using ActiveRecord or ActiveRecord fixtures
config.fixture_path = "#{::Rails.root}/spec/fixtures"

# Allow a 'focus' tag so that we can run just a few tests which we are currently working on
config.filter_run focus: true
config.run_all_when_everything_filtered = true
config.filter_run_excluding :slow unless ENV["SLOW_SPECS"]

# Defer Garbage Collection
config.before(:all) { DeferredGarbageCollection.start }
config.after(:all)  { DeferredGarbageCollection.reconsider }

# Integration Testing
require 'capybara/rspec'
require 'capybara/poltergeist'

Capybara.register_driver :poltergeist_debug do |app|
  Capybara::Poltergeist::Driver.new(app, {:inspector => true, js_errors: false })  
end

Capybara.javascript_driver = :poltergeist_debug
Capybara.default_driver = :poltergeist_debug

# Debugging tools
def debugit
  puts current_url
  require 'pry'
  binding.pry
end

# If you're not using ActiveRecord, or you'd prefer not to run each of your
# examples within a transaction, remove the following line or assign false
# instead of true.
config.use_transactional_fixtures = false

#Show Deprications As Errors with full backtracing
config.raise_errors_for_deprecations!

#rest of the file....
# Final part of Configure Database Cleaner

Capybara.default_max_wait_time = 5
config.use_transactional_fixtures = false

config.before(:suite) do
  if config.use_transactional_fixtures?
    raise(<<-MSG)
      Delete line `config.use_transactional_fixtures = true` from
      rails_helper.rb (or set it to false) to prevent uncommitted
      transactions being used in JavaScript-dependent specs. During 
      testing, the app-under-test that the browser driver connects to 
      uses a different database connection to the database connection 
      used by the spec. The app's database connection would not be 
      able to access uncommitted transaction data setup over the 
      spec's database connection.
     MSG
  end
  DatabaseCleaner.clean_with(:truncation)
end  

config.before(:each) do
  DatabaseCleaner.strategy = :transaction
end

config.before(:each, type: :feature) do
  # :rack_test driver's Rack app under test shares database connection
  # with the specs, so continue to use transaction strategy for speed.
  driver_shares_db_connection_with_specs = Capybara.current_driver == :rack_test

    if !driver_shares_db_connection_with_specs
      # Driver is probably for an external browser with an app
      # under test that does *not* share a database connection with the
      # specs, so use truncation strategy.
      DatabaseCleaner.strategy = :truncation
    end
  end

  config.before(:each) do
    DatabaseCleaner.start
  end

  config.append_after(:each) do
    DatabaseCleaner.clean
  end
end


def admin_sign_in
  visit "/login"

  #Create staff member in database
  Staff.make!(:admin)

  #Log In
  fill_in "staff_username", with: "adminstaff"
  fill_in "staff_password", with: "password"
  click_button "login"

  expect(page).to have_text('Logout')
end

def create_costing(item)
  @item = item
  visit "/api#/costings"

  expect(page).to have_selector("#new_btn")
  click_on "new_btn"

  expect(page).to have_text("New Costing")
  within "#form_costing" do
    fill_in "name", with: "#{@item}"
    fill_in "description", with: "test description"
    fill_in "from_date1", with: "15/02/2015" 
    fill_in "cost_hourly_cents1", with: "12.00"

    expect(page).to have_selector("#create_btn")
    click_on "create_btn"
  end
  expect(page.find("#page_title")).to have_content("Costings")
end