我目前正在为一个业余爱好项目开发一个小型爬虫应用。我在这里使用Qt 5.10。这是代码的相关部分:
std::string result = "";
url = QUrl(inURL.c_str());
jar->setCookiesFromUrl(list,url);
while(true){
QNetworkRequest request = QNetworkRequest(url);
QEventLoop loop;
connect(manager, SIGNAL(finished(QNetworkReply*)),&loop, SLOT(quit()));
netReply = manager->get(request);
loop.exec();
if(netReply->error() == QNetworkReply::NoError){
result = QByteArray(netReply->readAll()).constData();
break;
}
else{
DebugMsg << "Error loading: " << inURL << ". Retrying";
}
}
_usedWProxy++;
if(checkForIdentity() && torified)renewIdentity();
return result;
这里jar,url,manager和netReply是我的对象的成员。 这段代码适用于大多数情况,但最终崩溃了&#34; gdb其中&#34;输出如下。它不会在某个URL或服务器上崩溃,因此目前它对我来说并不是真正可重现的。我正在迭代很多网址,并且不时崩溃。 它最后一次在这个网站上崩溃了:
https://sofifa.com/players?v=18&e=159061&set=true&offset=9120
GDB out:
线程10&#34; QNetworkAccessM&#34;收到信号SIGSEGV,分段故障。
[切换到线程0x7fffe2492700(LWP 22676)]
来自/lib/x86_64-linux-gnu/libcrypto.so.1.0.0的lh_insert()中的0x00007fffee007540
(gdb)其中
来自/lib/x86_64-linux-gnu/libcrypto.so.1.0.0的lh_insert()中的0 0x00007fffee007540
1 0x00007fffedf51314在? ()来自/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
2 0x00007fffedf51917在? ()来自/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
来自/lib/x86_64-linux-gnu/libcrypto.so.1.0.0的BIO_free()中的3 0x00007fffedffd962
来自/lib/x86_64-linux-gnu/libcrypto.so.1.0.0的BIO_free_all()中的4 0x00007fffedffe2e4
来自/lib/x86_64-linux-gnu/libssl.so.1.0.0的SSL_free()中的5 0x00007fffee370f3f 6 0x00007ffff5cda6ea ?? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5
7 0x00007ffff5cbc578在? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5
8 0x00007ffff5cc2451在? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5
QMetaObject中的9 0x00007ffff72a3bc9 ::来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5的(QObject *,int,int,void **)()
来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5的QAbstractSocket :: disconnectFromHost()()中的10 0x00007ffff5c92078
来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5的QSslSocket :: close()()中的11 0x00007ffff5cc27e1
12 0x00007ffff5bf69e9在? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5
13 0x00007ffff5bf96c5在? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5
14 0x00007ffff5bf9525在? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5
15 0x00007ffff5bf793a在? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5
QBetaObject中的16 0x00007ffff72a3bc9 ::来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5的(QObject *,int,int,void **)()
17 0x00007ffff5cddb48 in ?? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5
18 0x00007ffff5cc24b1在? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5
QMetaObject中的19 0x00007ffff72a3bc9 ::来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5的(QObject *,int,int,void **)()
20 0x00007ffff5c926f3 in ?? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5
21 0x00007ffff5c927ac在? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5
22 0x00007ffff5c9b123在? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5
23 0x00007ffff5ce9209在? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Network.so.5
来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5的QObject :: event(QEvent *)()中的24 0x00007ffff72a4b21
来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5的QCoreApplication :: notifyInternal2(QObject *,QEvent *)()中的25 0x00007ffff72789e3
来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5的QCoreApplicationPrivate :: sendPostedEvents(QObject *,int,QThreadData *)()中的26 0x00007ffff727b48b
27 0x00007ffff72cd563 ?? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
来自/lib/x86_64-linux-gnu/libglib-2.0.so.0的g_main_context_dispatch()中的28 0x00007ffff2766197
29 0x00007ffff27663f0 in ?? ()来自/lib/x86_64-linux-gnu/libglib-2.0.so.0
来自/lib/x86_64-linux-gnu/libglib-2.0.so.0的g_main_context_iteration()中的30 0x00007ffff276649c
来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5的QEventDispatcherGlib :: processEvents(QFlags)()中的31 0x00007ffff72ccbaf
来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5的QEventLoop :: exec(QFlags)()中的32 0x00007ffff727726a
来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5的QThread :: exec()()中的33 0x00007ffff70a16dc
34 0x00007ffff70a660c在? ()来自/data/local/Qt/5.10.0/gcc_64/lib/libQt5Core.so.5
在p_read_create.c:333 的start_thread(arg = 0x7fffe2492700)中35 0x00007ffff59946ba
36 0x00007ffff627541d in clone()at ../sysdeps/unix/sysv/linux/x86_64/clim.S:109
由于GDB输出中没有提到我自己代码的一行,我怀疑这里有OpenSSL和/或Qt的问题。 这是我的OpenSSL版本:
user @ machine:〜$ openssl version
OpenSSL 1.0.2g 2016年3月1日
顺便说一句,我正在运行Ubuntu 16.04.03。
据我了解,它在断开连接时崩溃了。
我完全不知道采取什么措施来解决这个问题。 我会感谢你的所有建议。 提前致谢 马库斯
使用BJAM编译的最小示例:
main.cpp中:
#include <iostream>
#include <QtCore/QCoreApplication>
#include "loader.hpp"
int main(int argc, char *argv[])
{
QCoreApplication a(argc, argv);
TestLoader *load = new TestLoader();
for(unsigned int i = 0; i < 5000; i++){
std::cout << load->getContent("https://sofifa.com/players?v=18&e=159061&set=true&offset=9120").length() << std::endl;
}
load->deleteLater();
a.exec();
}
loader.hpp
#define TESTLOADER_H
#include <QObject>
#include <QNetworkAccessManager>
#include <QNetworkRequest>
#include <QNetworkReply>
#include <QUrl>
#include <QEventLoop>
#include <iostream>
#include <unistd.h>
#include <stdio.h> //printf
#include <string.h> //strlen
#include <string> //string
#include <sys/socket.h> //socket
#include <arpa/inet.h> //inet_addr
#include <netdb.h> //hostent
class TestLoader : public QObject
{
Q_OBJECT
private:
QNetworkAccessManager *manager;
QNetworkReply *netReply;
QUrl url;
public:
explicit TestLoader(QObject *parent = 0){
manager = new QNetworkAccessManager(this);
};
std::string getContent(std::string inURL){
std::string result = "";
url = QUrl(inURL.c_str());
while(true){
QNetworkRequest request = QNetworkRequest(url);
QEventLoop loop;
connect(manager, SIGNAL(finished(QNetworkReply*)),&loop, SLOT(quit()));
netReply = manager->get(request);
loop.exec();
if(netReply->error() == QNetworkReply::NoError){
result = QByteArray(netReply->readAll()).constData();
break;
}
else{
std::cout << "Error loading: " << inURL << ". Retrying";
}
}
return result;
};
signals:
public slots:
private slots:
};
#endif // TESTLOADER_H
的Jamfile:
alias install : install-bin ;
explicit install ;
install install-bin : LoaderTestApp
: <location>bin
<install-dependencies>on
<install-type>EXE
;
project :
requirements
<include>./
<cxxflags>-std=c++11
<cxxflags>-I/usr/include
<cxxflags>-I/usr/local/include
<cxxflags>-Wall
<cxxflags>-Wno-deprecated
<cxxflags>-g
<linkflags>-L/usr/lib
<linkflags>-L/usr/local/lib
<linkflags>-L/usr/lib/x86_64-linux-gnu
;
exe LoaderTestApp : loadertestapp.cpp
/qt5//QtCore
/qt5//QtNetwork
loader.hpp
:
;
这3个文件应放在名为utils的目录中(这是我的结构) 然后你需要Jamroot :(放在顶层目录,请改变Qt的路径)
using gcc ;
using qt5 : /data/local/Qt/5.10.0/gcc_64/ ;
project :
requirements
;
build-project utils
答案 0 :(得分:0)
我已经想通了,我项目的问题是什么。我使用QNAM来抓取网站,而另一个线程我使用CURL将抓取结果放到couchDB中。在OpenSSL 1.1.0中,多线程功能得到了改进(参见here),因此更新OpenSSL对我来说非常有用。 很抱歉这个问题,甚至没有包含这个问题,感谢@eyllanesc,他在我的机器上测试了我的代码。