用scrapy从药物报告中的表格中刮取数据

Question

我试图使用Scrapy从Erowid.org（药物旅行报告的精选数据库）中提取数据。我使用http://github.com/DavidYi1/Drug-Forum-Data-Mining的修改版本。这允许我抓取网站并为每个物质导出一个文件夹，每个旅行报告都在一个单独的.txt文件中。

我希望获得有关药物剂量，摄入方法和体重的元数据，这些元数据通常在报告前的表格中显示（例如http://www.erowid.org/exp/69866）。

但是，我不太清楚如何做到这一点，以便我可以在每个报告对应的每个.txt文件的开头都有这个元数据。例如：

  

物质：LSD

     剂量：200ug

     

方法：口服

     重量：160磅

     

旅行报告的文本

提前致谢！

Answer 1

您可以使用xpaths选择dara：

In [49]: sub =  response.xpath("//td[@class='dosechart-substance']/a/text()").extract_first()

In [50]: dose =  response.xpath("//td[@class='dosechart-amount']/text()").extract_first()

In [51]: form =  response.xpath("//td[@class='dosechart-form']/b/text()").extract_first()

In [52]: weight =  response.xpath("//td[@class='bodyweight-amount']/text()").extract_first()

In [53]: sub, dose, form, weight
Out[53]: (u'LSD', u'6 hits', u'(blotter / tab)', u'170 lb')

文本位于包含report-text-surround类的div中：

In [64]: response.xpath("//div[@class='report-text-surround']/text()").extract()
Out[64]: 
[u'\n',
 u'\n\n',
 u'\n\t',
 u'\n',
 u'\n\n\t',
 u'\n\n',
 u'\n',
 u'\n',
 u'\nI became interested in psychedelic drugs in the winter of 2005, and having had multiple experiences with mushrooms and LSD at lower doses, I decided during mid March of 2006 that I was ready for a step up.  I spent a few hours on Saturday morning cleaning up, I ate a light meal (I believe it was cold pizza) and at 2:40 in the afternoon I ate my last 6 hits of blotter acid. \r',
 u'\n\r',
 u'\nI considered myself a fairly seasoned psychonaut at the time, and I recognized the initial onset of the condition roughly fifteen minutes later.  Colors became sharper, and sounds took on a strange ethereal quality.  I walked into the den, encouraged.  It was a large room, with a hardwood floor and a fireplace.  I felt certain that this was the right room in which to begin my journey, and so I reclined on the couch and fell into a relaxed, mellow state.  That was when I began to hear singing.  It started as a sweet humming sound, like a barbershop quartet tuning up, only at very low volume.  The sound began to rise, and the tone began to change.   The windows were slightly open, and a cool breeze began blowing the curtains from the window,   It seemed to me at the time that there was a whole invisible choir in the room, and each change of the light wrought its own change upon the tune.  It grew and grew, and soon instruments were thrown into this beautiful choral mix.  I began to search in vain for the source of the sound, but as I moved the music grew around me.  Soon I recognized that the sound was coming from me, and from all the life around me.  The music was the embodiment of life, and spirit, and joy.  Everything I saw added to this beautiful melody.  I went into another room, a hallway this time, and a few soaring violin notes accompanied me.  The mellow feeling gave way to a kind of curious energy, as if I was using awe itself as fuel.  \r',
 u'\n\r',
 u"\nFor the next thirteen hours, I was enthralled by the beauty of life as I had not been in over a decade.  Color seemed to thrive in this new, alien world! The entirety of the rainbow glared at me from every corner.  I saw colors I couldn't name, colors that I've never seen replicated.  It was as if I had never seen color.  I  looked out the window towards the bay, and grasped the size of the universe in a new way.  A vast map formed inside my head, earth, and the planets, and the sun and stars and space beyond.  It grew ever larger, and an arrow appeared over the grain of sand the earth had become, an arrow with the words 'You are Here' hovering almost comically above.  I snapped back to reality, or what my reality had become, and took a few moments to clear my head.  While I was recovering from the sheer size of the void, a white swing caught my eye.   A red cardinal had perched on the right hand arm-rest and the colors were so sharp and clean that they almost hurt to look at. I thought it seemed very realistic, then laughed to myself.  After all, it was real life!   My mental processes seemed remarkably clear, and I began to 'toy' with ideas in my head.  I was fascinated by my new found ability to take a blueprint, and picture it as an actual object.  I am not nor will I ever be a mechanically-    inclined person, but today it seemed not to matter.  I built entire houses out of thought,  being able to cross-section and dissect them with ease.   Even such complex instruments as jet engines could be effortlessly mentally re-constructed.  After playing in the sandbox that my brain had become for some time, I opened the back door and headed out onto the patio.\r",
 u'\n\r',
 u"\nTrees and grass were moving in a way I had never seen before, above the surface of the ocean.  Branches and leaves seemed to burn with motion, each becoming a hazy, insubstantial blur as they moved.  'Like kelp, but not' I said out loud.  It occurred to me that I was talking to myself, and had been for quite some time, and a fresh gale of laughter overtook me.  Each old and familiar sight was altered, changed somehow as to make it appear completely new and wonderful.  I looked up at the sky, curious, and was greeted by the sight of a vast cerulean ocean.  It was a very clear day, and both the moon and the sun were visible, and they both screamed and competed for my attention.  A jet soared overhead, and it left a great plume of rainbow exhaust all the way across the sky.  As I watched, the jet passed cleanly through the moon and out the other side, creating momentary chaos.  The sky changed entirely, becoming first darker and then lighter.  Colors sprang from nowhere, and disappeared just as quickly.  I knew how easy it would be to forget everything and watch the sky patiently until the end of time, but I had other things planned for the day.  I looked at an oak, and saw that each leaf was in fact a diffuse, green light.  The green gave way to blue, which in turn gave way to yellow.  These changes continued at a fairly predictable rate, and I realized that I had control over them!  \r",
 u'\n\r',
 u"\nSuddenly it seemed that the outside world was on my side, and I was safe and in control.  I felt completely at peace.  It was a warm day, but even if it had been freezing I doubt if I would have felt it.  Eventually I decided to return to the house before I forgot how to move.  It sounds silly to me looking back now, but at the time it seemed to be very possible.  Given the sensory over stimulation of the situation, I'm not entirely surprised.     \r",
 u'\n\r',
 u'\nI ventured back inside and turned the t.v on, browsing at large.  It started to become complicated as the buttons began to change, sometimes as part of a pattern, sometimes at random.  A short time later, they began to drift from the face of the remote itself!  Laughing like a loon, I tossed the oddity of a remote back onto the table, where the clean grain of the cherry wood swam over it and devoured it.  I smirked at this odd sight, completely at ease, and refocused my attention on the movie.   It was an old favorite of mine, but I found vision being routinely drawn away to a large, white wall on my right.  I eventually gave up watching the t.v to focus my full attention on the wall.  Complicated fractal and runic patterns would arise spontaneously on its surface, only to be seconds later subducted into the ether.  Chaos would destroy order, which would instantly arise in another form.  I saw planets rise and fall, stars form, reason melt and the very fabric of the universe be torn open, to spontaneously be pulled back into the plasma of the wall, and then spit back out in another form.  At times I could only interpret the colors as noise, the sounds as color.  I could visibly see the music from the clock radio on the counter traveling from point to destination, but I could not for the life of me make out the digital readout of the clock.  I became very preoccupied with time, and dimension.  Such intangible concepts to me were like solid objects, visible things that could be touched and seen.  \r',
 u'\n\r',
 u"\nIf it were possible to break an acid trip down into words, the sheer volume of words would be staggering.  Hundreds of ideas cross the brain in seconds, weaving a complicated spider's web that seem to fit perfectly together, although they share few common characteristics.  Color and light seem as different sides of the same coin, striving in some cases for balance, and in other cases for contrast.  I decided to listen to some music without the radio, and in seconds was conducting full symphonies in my head, adding and changing instruments on the fly to produce intricate and pleasing melodies.  All the electronic gadgets in the world seemed inferior to that wonderful internal toolkit I was only now receiving full access to.  Soon the audible mental orchestra gave way to a visible one, and I imagined myself to be in a concert hall, watching a group of celestial beings play instruments unlike any I had ever seen before.  It was purely a mental picture, but something in its aspect reminded me so strongly of real life that I began to wonder if I had been granted a true vision of some other dimension.  Realizing the answer to this question would always escape me, I endeavored to think it over anyway, with disappointing results  After some time of this, I realized it was quickly approaching sunset, a sight I did not intend to miss.    \r",
 u'\n\r',
 u"\nI ventured outside again and felt my breath leave my throat.  I was enraptured.  The sky that had been so perfectly, absolutely blue only a few hours ago was now a violet so gorgeous and benign that the strength left my knees and I sat loudly on the ground with a loud 'oomph!' sound.  This violet color was reflected on the surface of the water as an intense, fiery orange.  The sun itself had turned a red so deep it seemed a burning ball of blood in the sky.   All around, waves of colored smoke-light (it had the characteristics of both) began to caper and dance.  They distinctly seemed to be sentient, and benevolent.  I reached my hand into one of them and at that point I had an epiphany.  I realized that nothing is unimportant, and everything matters.  For some people this might seem obvious, but the idea struck me at the time as being crucial, and central to my own personal beliefs.  The sun finished its performance, and the stars made their way onto the stage.  The sky became a vast, black painted sheet of canvas, and the stars gleamed like cold jewels in the night sky.  They rippled and undulated, as if they were pleased with their own beauty. \r",
 u'\n\r',
 u"\nI sat there, totally unable to move for an indeterminate amount of time, watching the night silently explode with color and life.  Beauty enveloped me, rolled over me like a wave.  The clouds began to race across the shining orange surface of the moon, surrounding it in a halo of the purest gold.  This gold light bounced off the surface of the water, lending an silver gleam to every moist blade of grass.  The night itself seemed to beckon me, it was totally engrossing.  I watched and I walked, but I didn't speak, not wanting the spoken word to mar the beauty of the night world.  I wandered for a while, oblivious to both the growing coolness in the air and the emptiness of my stomach.  I didn't seem to need food, the night provided more than enough to sustain me.  I wandered for hours, thinking of many things, and basking in the beauty of the night.  \r",
 u'\n\r',
 u'\nEventually, the maniac energy which had possessed me throughout the day gave way to an exalted, happy weariness, much like the one that accompanies a a day of hard but ultimately successful labor.  I retired to my bedroom, and began to stare at the ceiling.  Chunks of drywall began to bulge and drip from the walls and ceiling, only to dissipate into a thin smoke.  The smoke would drift a few feet from its point of origin, and then lazily make its way back into the gap.  Soon enough,  I began to ponder the nature of the universe in a general way, which led me specifically to thoughts of black holes.  Not a moment after I had begun this train of thought, a small black vortex opened up in the center of the ceiling with an audible stretching noise.  It grew larger and larger, and upon reaching the size of a basketball began to exude ribbons of the colored smoke-light that I had seen earlier.  They were very bright, primary colors, the exact shades on a kindergarten color chart.  They began to revolve around the room, picking up speed as they went.  Eventually the colors began to band together, and soon the room was a circus of colors all racing through each other and intertwining.  All at once, the mystery of the tie-dyed t shirt was no mystery to me.  The night ended with me laughing myself to sleep.  \r',
 u'\n\r',
 u"\nI awoke the next morning with a sense of exultation, I felt like I'd lived eons and seen things that no one else would ever see.  The relief of stress was enormous, and for a few months the minor problems we all have every day failed to dampen my new found spirits.  It has affected me in more ways that I can put into words, but overall I'm a much more tolerant person, and I like to think of myself as being more understanding.  I feel as if the mystery has returned to my life, and with it my will to continue moving forward.  After all, there's no telling what might be around the next bend, and whether or not it'll be able to top that day as the most thrilling and awe-inspiring of my life.",
 u'\n',
 u'\n\n\n\n',
 u'\n\n']

如果您查看来源，您可以看到数据所在的标签，您只需根据以下内容进行选择：

您可能还想阅读xpath tutorial