如何在JavaScript调用后获取网页的源代码（模态窗口）

Question

在某些网站上，例如this page当我点击“立即购买”时，将打开一个模态窗口。我试图通过在Java中使用WebClient单击此按钮然后单击模态窗口上的“购买”按钮，但我始终具有相同的页面源结果。在Java中有没有其他方法可以做到这一点？

Answer 1

感谢您的建议。这个模态窗口在代码中。

我只是使用HtmlUnit，但我仍然无法得到我想要的东西：/这是我的代码：

    public String getPageSourceFromBrowser(String url) throws SiteAnalizeException {    
        WebClient webClient = new WebClient(BrowserVersion.FIREFOX_3_6); 

        HtmlPage firstPage = null;
        String result = null;
        try {

            webClient.setJavaScriptEnabled(true);
            webClient.setThrowExceptionOnScriptError(false);
            webClient.setCssEnabled(false);
            webClient.setUseInsecureSSL(false);
            webClient.setRedirectEnabled(true);

            firstPage = webClient.getPage(new URL(url));  
            result = firstPage.getWebResponse().getContentAsString("UTF-8");
            DomNodeList<HtmlElement> button = firstPage.getElementsByTagName("a");
            for (HtmlElement htmlElement : button) {
              if(htmlElement.asText().equals("Buy Now")) {
                 HtmlPage page = htmlElement.click();
                 //HtmlElement button2 = page.getElementById("market_buynow_dialog_addfunds");
                 //HtmlPage page2 = button2.click();
                         String htmlBody = page.getWebResponse().getContentAsString(); 
                 System.out.println(htmlBody);              
              }
            }
 }

这是来自页面来源的按钮代码：

<div class="market_listing_right_cell market_listing_action_buttons">
                <div class="market_listing_buy_button">
                            <a href="javascript:BuyMarketListing('listing', '2306130643366381797', 730, '2', '11320870')" class="item_market_action_button item_market_action_button_green">
                <span class="item_market_action_button_edge item_market_action_button_left"></span>
                <span class="item_market_action_button_contents">
                    Buy now                 </span>
                <span class="item_market_action_button_edge item_market_action_button_right"></span>
                <span class="item_market_action_button_preload"></span>
            </a>
                        </div>
        </div>

JavaScript的：

BuyItemDialog = {
m_bPurchaseClicked: false,
m_bPurchaseSuccess: false,
m_bInitialized: false,

m_oListingOriginalRow: null,
m_sElementPrefix: null,
m_ulListingId: false,
m_item: null,
m_fnDocumentKeyHandler: null,

m_nSubtotal: 0,
m_nFeeAmount: 0,
m_nTotal: 0,

m_sAddFundsReturnURL: null,

Initialize: function() {
    $('market_buynow_dialog_purchase').observe( 'click', this.OnAccept.bindAsEventListener(this) );
    $('market_buynow_dialog_addfunds').observe( 'click', this.OnAddFunds.bindAsEventListener(this) );
    $('market_buynow_dialog_cancel').observe( 'click', this.OnCancel.bindAsEventListener(this) );
    $('market_buynow_dialog_cancel_close').observe( 'click', this.OnCancel.bindAsEventListener(this) );
    $('market_buynow_dialog_close').observe( 'click', this.OnCancel.bindAsEventListener(this) );

    $('market_buynow_dialog').style.visibility = 'hidden';
    $('market_buynow_dialog').show();
    $('market_buynow_dialog').hide();
    $('market_buynow_dialog').style.visibility = '';

    this.m_bInitialized = true;
},

Show: function ( sElementPrefix, listingid, item ) {

    if ( !this.m_bInitialized )
        this.Initialize();

    this.m_bPurchaseClicked = false;
    this.m_bPurchaseSuccess = false;

    $('market_buynow_dialog_error').hide();

    $('market_buynow_dialog_title').update( 'Kup przedmiot' );
    $('market_buynow_dialog_purchasecomplete_message').hide();
    $('market_buynow_dialog_purchase').show();
    $('market_buynow_dialog_purchase_throbber').hide();
    $('market_buynow_dialog_paymentinfo_frame_container').show();
    $('market_buynow_dialog_bottom_buttons').hide();

    $('market_buynow_dialog_cancel').show();
    $('market_buynow_dialog_cancel_close').hide();

    var sCurrentURL = window.location.protocol + "//" + window.location.host + window.location.pathname;
    this.m_sAddFundsReturnURL = encodeURIComponent( sCurrentURL + '#buy' + sElementPrefix + '|' + listingid + '|' + item['appid'] + '|' + item['contextid'] + '|'  + item['id'] );
    this.m_oListingOriginalRow = $(sElementPrefix + '_' + listingid);

    var bNoWallet = g_rgWalletInfo['wallet_currency'] == 0;
    var sWalletCurrencyCode = GetCurrencyCode( g_rgWalletInfo['wallet_currency'] );
    var rgListing = g_rgListingInfo[listingid];
    if ( rgListing['converted_fee'] > 0 )
    {
        this.m_nSubtotal = rgListing['converted_price'];
        this.m_nFeeAmount = rgListing['converted_fee'];
        this.m_nTotal = rgListing['converted_price'] + rgListing['converted_fee'];

        var nFeePublisher = rgListing['converted_publisher_fee'];
        var nFeeSteam = rgListing['converted_steam_fee'];

        if ( this.m_nFeeAmount != nFeePublisher + nFeeSteam || this.m_nTotal != this.m_nSubtotal + nFeePublisher + nFeeSteam )
        {
            alert( "An unexpected error occurred trying to show the purchase dialog. Error: " + listingid + " " + this.m_nTotal + " " + this.m_nSubtotal + " " + this.m_nFeeAmount + " " + nFeePublisher + " " + nFeeSteam );
            return;
        }

        $('market_buynow_dialog_totals_subtotal').update( v_currencyformat( this.m_nSubtotal, sWalletCurrencyCode ) );
        $('market_buynow_dialog_totals_publisherfee').update( v_currencyformat( nFeePublisher, sWalletCurrencyCode ) );
        $('market_buynow_dialog_totals_publisherfee_percent').update( ( rgListing['publisher_fee_percent'] * 100 ).toFixed(1) );
        if ( typeof g_rgAppContextData[rgListing['publisher_fee_app']] != 'undefined' )
        {
            $J('#market_buynow_dialog_totals_publisherfee_gamename').text( g_rgAppContextData[rgListing['publisher_fee_app']].name );
        }
        else
        {
            // No app data for some reason
            // Say "Game fee"
            $J('#market_buynow_dialog_totals_publisherfee_gamename').text( 'GrÄ' );
        }
        $('market_buynow_dialog_totals_transactionfee').update( v_currencyformat( nFeeSteam, sWalletCurrencyCode ) );
        $('market_buynow_dialog_totals_transactionfee_percent').update( ( g_rgWalletInfo['wallet_fee_percent'] * 100 ).toFixed(1) );
        $('market_buynow_dialog_totals_total').update( v_currencyformat( this.m_nTotal, sWalletCurrencyCode ) );
        $('market_buynow_dialog_totals').show();
    }
    else
    {
        this.m_nSubtotal = rgListing['converted_price'];
        this.m_nFeeAmount = 0;
        this.m_nTotal = rgListing['converted_price'];
        $('market_buynow_dialog_totals').hide();
    }

    var elEuSSA = $('market_buynow_dialog_eu_ssa');
    if ( this.m_nTotal > g_rgWalletInfo['wallet_balance'] || bNoWallet )
    {
        $('market_buynow_dialog_purchase').hide();
        $('market_buynow_dialog_addfunds').show();
        $('market_buynow_dialog_accept_ssa_container').hide();
        if ( elEuSSA )
        {
            elEuSSA.hide();
        }
    }
    else
    {
        $('market_buynow_dialog_purchase').show();
        $('market_buynow_dialog_addfunds').hide();
        $('market_buynow_dialog_accept_ssa_container').show();
        if ( elEuSSA )
        {
            elEuSSA.show();
        }
    }

    if ( bNoWallet )
    {
        $('market_buynow_dialog_walletbalance').hide();
    }
    else
    {
        $('market_buynow_dialog_walletbalance').show();
        $('market_buynow_dialog_walletbalance_amount').update( v_currencyformat( g_rgWalletInfo['wallet_balance'], sWalletCurrencyCode ) );
    }

    this.m_sElementPrefix = sElementPrefix;
    this.m_ulListingId = listingid;
    this.m_item = item;

    var oListingRow = Element.clone( this.m_oListingOriginalRow, true );
    var oListingTable = Element.clone( this.m_oListingOriginalRow.up(1), false );
    var oListingTableRowHeader = Element.clone( this.m_oListingOriginalRow.up().down(), true);
    var oListingTableRows = Element.clone( this.m_oListingOriginalRow.up(), false );
    oListingTableRows.appendChild( oListingTableRowHeader );

    oListingRow.id = oListingRow.id + 'Copy';
    var oItemImg = oListingRow.select('.market_listing_item_img').first();
    if ( typeof oItemImg != 'undefined' )
        oItemImg.id = oItemImg.id + 'Copy';

    var oItemActionMenuButton = oListingRow.select('.market_actionmenu_button').first();
    if ( typeof oItemActionMenuButton != 'undefined' )
    {
        oItemActionMenuButton.id = oItemActionMenuButton.id + 'Copy';
        $(oItemActionMenuButton).observe( 'click', function() {
            var rgAsset = g_rgListingInfo[listingid].asset;
            HandleMarketActionMenu( oItemActionMenuButton, g_rgAssets[rgAsset.appid][rgAsset.contextid][rgAsset.id] );

            return false;
        } );
    }

    var oItemName = oListingRow.select('.market_listing_item_name').first();
    oItemName.id = oItemName.id + 'Copy';
    oListingTableRows.appendChild( oListingRow );

    var oAvatarLink = oListingTableRows.select('a').each( function( item ) {
        item.target = '_new';
    });

    oListingTableRows.id = oListingTable.id + 'Copy';
    oListingTable.appendChild( oListingTableRows );
    oListingTable.id = oListingTable.id + 'Copy';

    $('market_buynow_dialog_item').innerHTML = '';
    $('market_buynow_dialog_item').appendChild( oListingTable );
    if ( typeof oItemImg != 'undefined' )
        CreateItemHoverFromContainer( g_rgAssets, oItemImg.id, item.appid, item.contextid, item.id, item.amount );
    CreateItemHoverFromContainer( g_rgAssets, oItemName.id, item.appid, item.contextid, item.id, item.amount );
    $('hover').style.zIndex = 1001;

    this.m_fnDocumentKeyHandler = this.OnDocumentKeyPress.bindAsEventListener( this );
    $(document).observe( 'keydown', this.m_fnDocumentKeyHandler );

    showModal( 'market_buynow_dialog', true );
    $('market_buynow_dialog').focus();
},

DisplayError: function( error ) {
    $('market_buynow_dialog_error').show();
    $('market_buynow_dialog_error_text').update( error );
    $('market_buynow_dialog_error_text').style.color = '#ffffff';
    new Effect.Morph( $('market_buynow_dialog_error_text'), { style: {color: '#ff0000'}, duration: 0.25 } );
},

Dismiss: function() {
    $(document).stopObserving( 'keydown', this.m_fnDocumentKeyHandler );
    hideModal( 'market_buynow_dialog' );
},

OnAddFunds: function( event ) {
    event.stop();

    window.location = 'http://store.steampowered.com/steamaccount/addfunds?marketlisting=' + this.m_ulListingId + '&returnurl=' + this.m_sAddFundsReturnURL;
},

OnAccept: function( event ) {
    event.stop();

    // If already accepted, ignore
    if ( this.m_bPurchaseSuccess || this.m_bPurchaseClicked )
    {
        return;
    }

            if ( !$('market_buynow_dialog_accept_ssa') || !$('market_buynow_dialog_accept_ssa').checked )
    {
        this.DisplayError( 'Musisz przyjÄÄ warunki Umowy UĹźytkownika Steam, aby sfinalizowaÄ transakcjÄ.' );
        return;
    }

    this.m_bPurchaseClicked = true;
    $('market_buynow_dialog_error').hide();

    $('market_buynow_dialog_purchase_throbber').clonePosition( $('market_buynow_dialog_purchase') );
    $('market_buynow_dialog_purchase').fade({ duration: 0.25 });
    $('market_buynow_dialog_purchase_throbber').show();
    $('market_buynow_dialog_purchase_throbber').fade({ duration: 0.25, from: 0, to: 1 });

    var listingid = this.m_ulListingId;

    $J.ajax( {
        url: 'https://steamcommunity.com/market/buylisting/' + listingid,
        type: 'POST',
        data: {
            sessionid: g_sessionID,
            currency: g_rgWalletInfo['wallet_currency'],
            subtotal: this.m_nSubtotal,
            fee: this.m_nFeeAmount,
            total: this.m_nTotal
        },
        crossDomain: true,
        xhrFields: { withCredentials: true }
    } ).done( function ( data ) {
        BuyItemDialog.OnSuccess( { responseJSON: data } );
    } ).fail( function( jqxhr ) {
        // jquery doesn't parse json on fail
        var data = $J.parseJSON( jqxhr.responseText );
        BuyItemDialog.OnFailure( { responseJSON: data } );
    } );
},

OnCancel: function( event ) {
    this.Dismiss();
    event.stop();
},

OnSuccessEffects: function() {
    $('market_buynow_dialog_cancel').hide();
    $('market_buynow_dialog_cancel_close').show();

    $('market_buynow_dialog_purchase_throbber').fade({ duration: 0.25 });
    new Effect.BlindUp( 'market_buynow_dialog_paymentinfo_frame_container', { duration: 0.25 } );
    new Effect.BlindDown( 'market_buynow_dialog_purchasecomplete_message', { duration: 0.25 } );
    new Effect.BlindDown( 'market_buynow_dialog_bottom_buttons', { duration: 0.25 } );

    // Replace the listing row with a message that says this was purchased
    var oOriginalItemName = Element.clone( this.m_oListingOriginalRow.select('.market_listing_item_name').first(), true );
    this.m_oListingOriginalRow.update('');

    var elMessage = new Element( 'div', {'class': 'market_listing_purchase_message' } );
    var sItemNameSpanId = this.m_sElementPrefix + '_purchased_' + this.m_ulListingId;
    elMessage.update(
            'Zakupiono <%1$s></%2$s>. Zobacz go w swoim <%3$s>ekwipunku</%4$s>.'
                .replace( '%1$s', 'span id="' + sItemNameSpanId + '"' )
                .replace( '%2$s', 'span' )
                .replace( '%3$s', 'a href="http://steamcommunity.com/my/inventory/"' )
                .replace( '%4$s', 'a' ) );

    this.m_oListingOriginalRow.appendChild( elMessage );
    $(sItemNameSpanId).appendChild( oOriginalItemName );
},

OnSuccess: function( transport ) {
    this.m_bPurchaseSuccess = true;

    if ( transport.responseJSON )
    {
        var rgNewWalletInfo = transport.responseJSON.wallet_info;
        if ( rgNewWalletInfo && rgNewWalletInfo.success )
        {
            g_rgWalletInfo = rgNewWalletInfo;
        }

        var sWalletCurrencyCode = GetCurrencyCode( g_rgWalletInfo['wallet_currency'] );
        $('marketWalletBalanceAmount').update( v_currencyformat( g_rgWalletInfo['wallet_balance'], sWalletCurrencyCode ) );

        this.OnSuccessEffects();
    }
    else
    {
        this.OnFailureEffects();
        this.DisplayError( 'WystÄpiĹ bĹÄd podczas zakupu przedmiotu. MoĹźliwe, Ĺźe oferta zostaĹa usuniÄta. OdĹwieĹź stronÄ i sprĂłbuj ponownie.' );
    }
},

OnFailureEffects: function() {
    var queue = Effect.Queues.get('global');
    queue.each(function(effect) { effect.cancel(); });

    $('market_buynow_dialog_purchase').show();
    $('market_buynow_dialog_purchase').setOpacity('0');
    $('market_buynow_dialog_purchase').fade({ duration: 0.25, from: 0, to: 1 });
    $('market_buynow_dialog_purchase_throbber').fade({ duration: 0.25 });
},

OnFailure: function( transport ) {
    this.m_bPurchaseClicked = false;
    this.OnFailureEffects();

    if ( transport.responseJSON && transport.responseJSON.message )
    {
        this.DisplayError( transport.responseJSON.message );
    }
    else
    {
        this.DisplayError( 'WystÄpiĹ bĹÄd podczas zakupu przedmiotu. MoĹźliwe, Ĺźe oferta zostaĹa usuniÄta. OdĹwieĹź stronÄ i sprĂłbuj ponownie.' );
    }
},

OnDocumentKeyPress: function( event ) {
    if ( event.keyCode == Event.KEY_ESC )
    {
        this.Dismiss();
        event.stop();
    }
},

OnInputKeyPress: function( event ) {
    if ( event.keyCode == Event.KEY_RETURN )
    {
        if ( this.m_bPurchaseSuccess )
        {
            this.OnConfirmationAccept( event );
        }
        else
        {
            this.OnAccept( event );
        }
    }
},

OnInputKeyUp: function( event ) {

}}


function BuyMarketListing ( sElementPrefix, listingid, appid, contextid, itemid ) 
{

if ( !g_bLoggedIn )

{

    showModal( 'NotLoggedInWarning', true );
    return;
}

if ( !g_rgAssets[appid] || !g_rgAssets[appid][contextid] || !g_rgAssets[appid][contextid][itemid] )
{
    return;
}

BuyItemDialog.Show( sElementPrefix, listingid, g_rgAssets[appid][contextid][itemid]     );
}

对话窗口位于页面源的底部（我只需剪切主标签和应该点击的按钮）：

 <div id="market_buynow_dialog" class="market_modal_dialog" style="display: none;">
<div class="market_dialog_title">
    <span id="market_buynow_dialog_title">
            <a id="market_buynow_dialog_purchase" href="#"class="btn_green_white_innerfade btn_medium_wide btn_uppercase"><span>buy</span></a>

Answer 2

我会研究CasperJS处理动态网页，Ajax加载等问题.CasperJS是一个无头webkit浏览器，允许用户轻松完成这类工作。他们的文档很好，即使您对JavaScript不太熟悉，文档也很棒，并且在您测试完演示后，学习曲线很小。

这里的想法是封装一个CasperJS脚本来做一件非常棒的事情。给它一个特定的名称，如getBuyNowInfoFromSiteX.js或其他类似的名称。您可以稍后使用命令行参数使其更通用。在您的情况下，您尝试从网站获取一些数据。在JavaScript对象中获得数据后，让脚本通过将数据转换为JSON对象（JSON.stringify(object)）将数据写入stdout。

使用这个简单的脚本，我们现在可以从Java调用它并获取结果。我在很长一段时间没有使用过Java，也没有在过去用Java调用子进程，而是在查看it is possible之后（虽然有点麻烦！）。阅读该文章/浏览其他在线示例 - 似乎有几种方法。实际上，它看起来像这样（不是一个完全正常的演示，我抓住它from here）：

// Run our script "getBuyNowInfoFromSiteX.js" using the Runtime exec method:
Process p = Runtime.getRuntime().exec("casperjs getBuyNowInfoFromSiteX.js");

BufferedReader stdInput = new BufferedReader(new 
    InputStreamReader(p.getInputStream()));

String jsonOutput = '';

// read the standard output from the script
while ((s = stdInput.readLine()) != null) {
    jsonOutput += s; // use a StringBuilder, this is for demo purposes.
}

// ... do something with the jsonOutput string

使用该简化示例，您将获得一个从我们的脚本getBuyNowInfoFromSiteX.js的标准输出中收集的字符串。因为该脚本输出JSON，所以我将变量命名为jsonOutput。

现在我们以JSON格式从页面获取数据，我们需要一个库来用Java来处理它。这是StackOverflow上的an answer I stumbled across，它描述了三个可以处理JSON的不同库。

解析后，数据可以在Java中使用，您可以继续使用其他任何操作。

考虑

如果您要解析的网站没有动态内容，这可能会落伍。如果它全部是静态的，或者您尝试获取的信息不需要在该页面上启用Ajax调用/ JavaScript，那么您可以使用HTMLUnit。

另请注意，运行无头WebKit浏览器并不便宜。如果您计划生成多个进程（例如，根据客户端的请求进行动态抓取等），这将不可行，因为您可能会快速运行内存不足。如果您只是收集数据，将其存储在数据库中，然后将其丢弃，那么这样就可以了。

作为mentioned in the comments，有时模态对话框实际上只是页面上隐藏的div元素，并且根本没有任何动态的Ajax-y内容。查看页面的来源并确定最佳的行动方案。谁知道，您可能只是找到一个可以直接插入的无证JSON API，并完全避免整个网络抓取业务！