{"id":923141,"date":"2024-05-27T13:37:33","date_gmt":"2024-05-27T13:37:33","guid":{"rendered":"https:\/\/proxyelite.info\/?p=923141"},"modified":"2025-06-04T03:04:30","modified_gmt":"2025-06-04T03:04:30","slug":"scrapegraphai-revolutionizing-web-scraping","status":"publish","type":"post","link":"https:\/\/proxyelite.info\/cn\/scrapegraphai-revolutionizing-web-scraping\/","title":{"rendered":"ScrapeGraphAI\uff1a\u5f7b\u5e95\u6539\u53d8\u7f51\u7edc\u6293\u53d6\uff1f"},"content":{"rendered":"

\u7f51\u9875\u6293\u53d6\uff08Web \u6293\u53d6\u4ee3\u7406<\/a>\u968f\u7740\u5f3a\u5927\u5de5\u5177\u548c\u5e93\u7684\u51fa\u73b0\uff0c\u6570\u636e\u6293\u53d6\uff08Web \u6570\u636e\u6293\u53d6\uff09\u9886\u57df\u4e5f\u53d1\u751f\u4e86\u663e\u8457\u7684\u53d8\u5316\u3002ScrapeGraphAI \u662f\u8be5\u9886\u57df\u6700\u65b0\u3001\u6700\u5177\u5f71\u54cd\u529b\u7684\u5de5\u5177\u4e4b\u4e00\u3002\u4f46 ScrapeGraphAI \u7a76\u7adf\u662f\u4ec0\u4e48\uff1f\u5b83\u53c8\u5c06\u5982\u4f55\u5f7b\u5e95\u6539\u53d8\u60a8\u7684\u7f51\u9875\u6293\u53d6\u4efb\u52a1\uff1f\u8ba9\u6211\u4eec\u6df1\u5165\u4e86\u89e3\u5b83\u7684\u7ec6\u8282\uff0c\u63a2\u7d22\u5b83\u7684\u7279\u6027\u3001\u529f\u80fd\u4ee5\u53ca\u5982\u4f55\u5229\u7528\u5b83\u6ee1\u8db3\u60a8\u7684\u6570\u636e\u63d0\u53d6\u9700\u6c42\u3002<\/p>\n\n\n\n

\u4ec0\u4e48\u662f ScrapeGraphAI\uff1f<\/h2>\n\n\n\n

ScrapeGraphAI \u662f\u4e00\u4e2a\u521b\u65b0\u7684\u7f51\u9875\u6293\u53d6\u5e93\uff0c\u5728\u5f00\u53d1\u4eba\u5458\u548c\u6570\u636e\u7231\u597d\u8005\u4e2d\u8fc5\u901f\u6d41\u884c\u8d77\u6765\u3002\u53d1\u5e03\u540e\u77ed\u77ed\u51e0\u5468\u5185\uff0c\u5b83\u5728 GitHub \u4e0a\u5c31\u83b7\u5f97\u4e86 8,000 \u591a\u9897\u661f\uff0c\u8bc1\u660e\u4e86\u5b83\u7684\u5b9e\u7528\u6027\u548c\u6709\u6548\u6027\u3002\u8be5\u5e93\u7b80\u5316\u4e86\u4ece\u5404\u79cd\u6765\u6e90\uff08\u5305\u62ec HTML\u3001XML \u548c JSON\uff09\u6293\u53d6\u6570\u636e\u7684\u8fc7\u7a0b\uff0c\u4f7f\u5176\u6210\u4e3a\u4efb\u4f55\u6570\u636e\u63d0\u53d6\u4efb\u52a1\u7684\u591a\u529f\u80fd\u5de5\u5177\u3002<\/p>\n\n\n\n

\u4e3a\u4ec0\u4e48\u4f7f\u7528 ScrapeGraphAI\uff1f<\/h2>\n\n\n\n

\u5982\u679c\u60a8\u5b9a\u671f\u4ece\u4e92\u8054\u7f51\u4e0a\u6293\u53d6\u6570\u636e\uff0cScrapeGraphAI \u53ef\u4ee5\u663e\u8457\u7b80\u5316\u60a8\u7684\u5de5\u4f5c\u6d41\u7a0b\u3002\u4ee5\u4e0b\u662f\u8003\u8651\u4f7f\u7528\u6b64\u5e93\u7684\u4e00\u4e9b\u4ee4\u4eba\u4fe1\u670d\u7684\u7406\u7531\uff1a<\/p>\n\n\n\n

    \n
  1. \u4f7f\u7528\u65b9\u4fbf<\/strong>\uff1aScrapeGraphAI \u63d0\u4f9b\u4e86\u4e00\u4e2a\u7b80\u5355\u7684\u754c\u9762\u6765\u8bbe\u7f6e\u548c\u6267\u884c\u7f51\u9875\u6293\u53d6\u4efb\u52a1\u3002\u5176\u7528\u6237\u53cb\u597d\u7684\u8bbe\u8ba1\u8ba9\u90a3\u4e9b\u7f16\u7801\u7ecf\u9a8c\u6709\u9650\u7684\u4eba\u4e5f\u80fd\u8f7b\u677e\u4f7f\u7528\u3002<\/li>\n\n\n\n
  2. \u591a\u529f\u80fd\u6027<\/strong>\uff1a\u65e0\u8bba\u60a8\u9700\u8981\u4ece\u7f51\u9875\u3001XML \u6587\u4ef6\u8fd8\u662f JSON \u6e90\u6293\u53d6\u6570\u636e\uff0cScrapeGraphAI \u90fd\u80fd\u5904\u7406\u3002\u8fd9\u79cd\u591a\u529f\u80fd\u6027\u4f7f\u5176\u6210\u4e3a\u6ee1\u8db3\u5404\u79cd\u6570\u636e\u63d0\u53d6\u9700\u6c42\u7684\u4e00\u7ad9\u5f0f\u89e3\u51b3\u65b9\u6848\u3002<\/li>\n\n\n\n
  3. \u4e0e\u5927\u578b\u8bed\u8a00\u6a21\u578b (LLM) \u96c6\u6210<\/strong>\uff1a\u8be5\u5e93\u652f\u6301\u4e0e OpenAI \u7684 ChatGPT \u7b49\u6d41\u884c LLM \u96c6\u6210\uff0c\u4ece\u800c\u5b9e\u73b0\u9ad8\u7ea7\u6570\u636e\u5904\u7406\u529f\u80fd\u3002<\/li>\n\n\n\n
  4. \u793e\u533a\u652f\u6301\u548c\u6587\u6863<\/strong>\uff1a\u968f\u7740\u7528\u6237\u548c\u8d21\u732e\u8005\u793e\u533a\u7684\u4e0d\u65ad\u58ee\u5927\uff0cScrapeGraphAI \u63d0\u4f9b\u4e86\u5927\u91cf\u6587\u6863\u548c\u793a\u4f8b\uff0c\u5e2e\u52a9\u60a8\u5165\u95e8\u548c\u89e3\u51b3\u95ee\u9898\u3002<\/li>\n<\/ol>\n\n\n\n

    ScrapeGraphAI \u5165\u95e8<\/h2>\n\n\n\n

    \u8ba9\u6211\u4eec\u901a\u8fc7\u4e00\u4e2a\u57fa\u672c\u793a\u4f8b\u6765\u4e86\u89e3\u5982\u4f55\u5f00\u59cb\u4f7f\u7528 ScrapeGraphAI \u8fdb\u884c\u7f51\u9875\u6293\u53d6\u9879\u76ee\u3002<\/p>\n\n\n\n

    \u8bbe\u7f6e\u4f60\u7684\u73af\u5883<\/strong> <\/p>\n\n\n\n

    \u9996\u5148\uff0c\u521b\u5efa\u4e00\u4e2a\u865a\u62df\u73af\u5883\u6765\u7ba1\u7406\u60a8\u7684\u9879\u76ee\u4f9d\u8d56\u9879\u3002\u6b64\u6b65\u9aa4\u53ef\u786e\u4fdd\u60a8\u7684\u9879\u76ee\u4fdd\u6301\u72ec\u7acb\u5e76\u907f\u514d\u4e0e\u5176\u4ed6 Python \u9879\u76ee\u53d1\u751f\u51b2\u7a81\u3002<\/p>\n\n\n\n

    python -m venv .venv\nsource .venv\/bin\/activate  # On Windows, use .venv\\Scripts\\activate<\/code><\/pre>\n\n\n\n
    \u5b89\u88c5 ScrapeGraphAI \u548c\u4f9d\u8d56\u9879<\/strong><\/p>\n\n\n\n
    \u63a5\u4e0b\u6765\uff0c\u5b89\u88c5 ScrapeGraphAI \u4ee5\u53ca\u5176\u4ed6\u5fc5\u8981\u7684\u5e93\uff0c\u5982 pandas \u548c dotenv<\/p>\n\n\n\n
    pip install scrapegraphai pandas python-dotenv<\/code><\/pre>\n\n\n\n
    \u521b\u5efa\u57fa\u672c\u6293\u53d6\u811a\u672c<\/strong><\/p>\n\n\n\n
    \u73b0\u5728\uff0c\u8ba9\u6211\u4eec\u521b\u5efa\u4e00\u4e2a\u7b80\u5355\u7684 Python \u811a\u672c\u6765\u4ece\u7f51\u7ad9\u6293\u53d6\u6570\u636e\u3002\u5728\u6b64\u793a\u4f8b\u4e2d\uff0c\u6211\u4eec\u5c06\u4ece\u793a\u4f8b\u7f51\u7ad9\u6293\u53d6\u6587\u7ae0\u6807\u9898\u548c\u4f5c\u8005\u3002<\/p>\n\n\n\n
    import scrapegraphai as sgai\nimport pandas as pd\nimport os\nfrom dotenv import load_dotenv\n\n# Load environment variables\nload_dotenv()\n\n# Initialize ScrapeGraphAI\napi_key = os.getenv(\"OPENAI_API_KEY\")\nscraper = sgai.Scraper(api_key=api_key)\n\n# Define the scraping task\nprompt = \"List me all the articles from example.com with titles and authors.\"\nconfig = {\n    \"source\": \"https:\/\/example.com\",\n    \"fields\": [\"title\", \"author\"]\n}\n\n# Execute the scraping task\nresult = scraper.scrape(prompt, config)\n\n# Convert result to a DataFrame\ndf = pd.DataFrame(result[\"articles\"])\n\n# Save the result to an Excel file\ndf.to_excel(\"articles.xlsx\", index=False)<\/code><\/pre>\n\n\n\n
    \u5728\u6b64\u811a\u672c\u4e2d\uff0c\u6211\u4eec\uff1a<\/p>\n\n\n\n
      \n
    • \u5bfc\u5165\u5fc5\u8981\u7684\u5e93\u3002<\/li>\n\n\n\n
    • \u4ece\u73af\u5883\u6587\u4ef6\u52a0\u8f7d OpenAI API \u5bc6\u94a5\u3002<\/li>\n\n\n\n
    • \u4f7f\u7528 API \u5bc6\u94a5\u521d\u59cb\u5316 ScrapeGraphAI \u6293\u53d6\u5de5\u5177\u3002<\/li>\n\n\n\n
    • \u5b9a\u4e49\u6293\u53d6\u4efb\u52a1\u7684\u63d0\u793a\u548c\u914d\u7f6e\u3002<\/li>\n\n\n\n
    • \u6267\u884c\u6293\u53d6\u4efb\u52a1\u5e76\u5c06\u7ed3\u679c\u4fdd\u5b58\u5230Excel\u6587\u4ef6\u3002<\/li>\n<\/ul>\n\n\n\n
      \u5904\u7406 JSON \u6570\u636e<\/strong> <\/p>\n\n\n\n
      ScrapeGraphAI \u8fd8\u64c5\u957f\u4ece JSON \u6587\u4ef6\u4e2d\u63d0\u53d6\u6570\u636e\u3002\u4ee5\u4e0b\u662f\u5982\u4f55\u4ece JSON \u6587\u4ef6\u4e2d\u63d0\u53d6\u4e66\u7c4d\u4fe1\u606f\u7684\u793a\u4f8b\u3002<\/p>\n\n\n\n
      import scrapegraphai as sgai\nimport json\n\n# Initialize ScrapeGraphAI\nscraper = sgai.Scraper(api_key=api_key)\n\n# Define the JSON scraping task\njson_data = \"\"\"\n[\n    {\"title\": \"Book One\", \"author\": \"Author A\", \"genre\": \"Fiction\"},\n    {\"title\": \"Book Two\", \"author\": \"Author B\", \"genre\": \"Non-Fiction\"}\n]\n\"\"\"\nprompt = \"Given the JSON, list all the titles, authors, and genres of the books.\"\n\n# Execute the JSON scraping task\nresult = scraper.scrape_json(prompt, json_data)\n\n# Print the result\nprint(result)<\/code><\/pre>\n\n\n\n
      \u8be5\u811a\u672c\u6f14\u793a\u5982\u4f55\u4f7f\u7528 ScrapeGraphAI \u4ece JSON \u5b57\u7b26\u4e32\u4e2d\u63d0\u53d6\u7279\u5b9a\u5b57\u6bb5\u3002<\/p>\n\n\n\n
      \u9ad8\u7ea7\u529f\u80fd\u548c\u7528\u4f8b<\/h2>\n\n\n\n
      ScrapeGraphAI \u5305\u542b\u8bb8\u591a\u9ad8\u7ea7\u529f\u80fd\uff0c\u53ef\u6ee1\u8db3\u66f4\u590d\u6742\u7684\u6293\u53d6\u573a\u666f\u3002\u4ee5\u4e0b\u662f\u4e00\u4e9b\u60a8\u53ef\u80fd\u4f1a\u89c9\u5f97\u6709\u7528\u7684\u9644\u52a0\u529f\u80fd\uff1a<\/p>\n\n\n\n
        \n
      • \u81ea\u5b9a\u4e49\u63d0\u793a\u548c\u914d\u7f6e<\/strong>\uff1a\u4f7f\u7528\u81ea\u5b9a\u4e49\u63d0\u793a\u548c\u914d\u7f6e\u5b9a\u5236\u60a8\u7684\u6293\u53d6\u4efb\u52a1\uff0c\u4ee5\u7cbe\u786e\u63d0\u53d6\u60a8\u9700\u8981\u7684\u6570\u636e\u3002<\/li>\n\n\n\n
      • \u4e0e\u5176\u4ed6\u5de5\u5177\u96c6\u6210<\/strong>\uff1a\u5c06ScrapeGraphAI\u4e0e\u5176\u4ed6\u6570\u636e\u5904\u7406\u5de5\u5177\uff08\u5982pandas\u548cNumPy\uff09\u7ed3\u5408\u8d77\u6765\uff0c\u5bf9\u6293\u53d6\u7684\u6570\u636e\u8fdb\u884c\u8fdb\u4e00\u6b65\u5206\u6790\u3002<\/li>\n\n\n\n
      • \u672c\u5730\u6cd5\u5b66\u7855\u58eb<\/strong>\uff1a\u4f7f\u7528\u672c\u5730\u5927\u578b\u8bed\u8a00\u6a21\u578b\u8fdb\u884c\u6570\u636e\u63d0\u53d6\uff0c\u4ee5\u589e\u5f3a\u6027\u80fd\u548c\u9690\u79c1\u3002<\/li>\n<\/ul>\n\n\n\n
        \u793a\u4f8b\uff1a\u4ece Wired.com \u6293\u53d6\u5e76\u5206\u6790\u6570\u636e<\/h2>\n\n\n\n
        \u4e3a\u4e86\u8bf4\u660e ScrapeGraphAI \u7684\u5f3a\u5927\u529f\u80fd\uff0c\u8ba9\u6211\u4eec\u8003\u8651\u4e00\u4e2a\u4ece Wired.com \u6293\u53d6\u5e76\u5206\u6790\u6587\u7ae0\u7684\u4f8b\u5b50\u3002<\/p>\n\n\n\n
        \u8bbe\u7f6e\u6293\u53d6\u4efb\u52a1<\/strong><\/p>\n\n\n\n
        \u5b9a\u4e49\u63d0\u793a\u548c\u914d\u7f6e\u4ee5\u4ece Wired.com \u6293\u53d6\u6587\u7ae0\u6807\u9898\u548c\u4f5c\u8005\u3002<\/p>\n\n\n\n
        prompt = \"List all the articles from Wired.com with titles and authors.\"\nconfig = {\n    \"source\": \"https:\/\/www.wired.com\",\n    \"fields\": [\"title\", \"author\"]\n}<\/code><\/pre>\n\n\n\n
        \u6267\u884c\u4efb\u52a1\u5e76\u4fdd\u5b58\u6570\u636e<\/strong><\/p>\n\n\n\n
        \u8fd0\u884c\u811a\u672c\u6765\u6293\u53d6\u6570\u636e\u5e76\u5c06\u5176\u4fdd\u5b58\u5230 Excel \u6587\u4ef6\u3002<\/p>\n\n\n\n
        result = scraper.scrape(prompt, config)\ndf = pd.DataFrame(result[\"articles\"])\ndf.to_excel(\"wired_articles.xlsx\", index=False)<\/code><\/pre>\n\n\n\n
        \u5206\u6790\u6570\u636e<\/strong><\/p>\n\n\n\n
        \u4fdd\u5b58\u6570\u636e\u540e\uff0c\u60a8\u53ef\u4ee5\u4f7f\u7528 pandas \u6267\u884c\u5404\u79cd\u5206\u6790\uff0c\u4f8b\u5982\u8ba1\u7b97\u6bcf\u4e2a\u4f5c\u8005\u7684\u6587\u7ae0\u6570\u91cf\u6216\u8bc6\u522b\u6587\u7ae0\u6807\u9898\u4e2d\u7684\u8d8b\u52bf\u3002<\/p>\n\n\n\n
        author_counts = df[\"author\"].value_counts()\nprint(author_counts)<\/code><\/pre>\n\n\n\n
        \u7ed3\u8bba<\/h2>\n\n\n\n
        ScrapeGraphAI \u662f\u4e00\u6b3e\u529f\u80fd\u5f3a\u5927\u7684\u5de5\u5177\uff0c\u53ef\u7b80\u5316\u7f51\u9875\u6293\u53d6\u548c\u6570\u636e\u63d0\u53d6\u4efb\u52a1\u3002\u5b83\u7684\u6613\u7528\u6027\u3001\u591a\u529f\u80fd\u6027\u4ee5\u53ca\u4e0e\u5927\u578b\u8bed\u8a00\u6a21\u578b\u7684\u96c6\u6210\u4f7f\u5176\u6210\u4e3a\u5f00\u53d1\u4eba\u5458\u548c\u6570\u636e\u7231\u597d\u8005\u7684\u5b9d\u8d35\u8d44\u4ea7\u3002\u65e0\u8bba\u60a8\u662f\u4ece\u7f51\u9875\u3001XML \u6587\u4ef6\u8fd8\u662f JSON \u6e90\u6293\u53d6\u6570\u636e\uff0cScrapeGraphAI \u90fd\u80fd\u6ee1\u8db3\u60a8\u7684\u9700\u6c42\u3002\u901a\u8fc7\u9075\u5faa\u672c\u6587\u4e2d\u6982\u8ff0\u7684\u793a\u4f8b\u548c\u6700\u4f73\u5b9e\u8df5\uff0c\u60a8\u53ef\u4ee5\u5145\u5206\u5229\u7528 ScrapeGraphAI \u7684\u6f5c\u529b\u6765\u7b80\u5316\u6570\u636e\u63d0\u53d6\u6d41\u7a0b\u3002<\/p>\n\n\n\n
        \u8868\u683c\uff1a\u7f51\u9875\u6293\u53d6\u5de5\u5177\u6bd4\u8f83<\/h3>\n\n\n\n
        \u7279\u5f81<\/th>ScrapeGraphAI<\/th>\u7f8e\u4e3d\u6c64<\/th>\u522e\u75e7<\/th><\/tr><\/thead>
        \u4f7f\u7528\u65b9\u4fbf<\/td>\u9ad8\u7684<\/td>\u4e2d\u7b49\u7684<\/td>\u4e2d\u7b49\u7684<\/td><\/tr>
        \u591a\u529f\u80fd\u6027<\/td>\u9ad8\u7684<\/td>\u4e2d\u7b49\u7684<\/td>\u9ad8\u7684<\/td><\/tr>
        \u4e0e\u6cd5\u5b66\u7855\u58eb (LLM) \u7684\u6574\u5408<\/td>\u662f\u7684<\/td>\u4e0d<\/td>\u4e0d<\/td><\/tr>
        \u793e\u533a\u652f\u6301<\/td>\u9ad8\u7684<\/td>\u9ad8\u7684<\/td>\u9ad8\u7684<\/td><\/tr>
        \u6587\u6863<\/td>\u5e7f\u6cdb\u7684<\/td>\u597d\u7684<\/td>\u597d\u7684<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
        \u8be5\u8868\u5c06 ScrapeGraphAI \u4e0e\u5176\u4ed6\u6d41\u884c\u7684\u7f51\u7edc\u6293\u53d6\u5de5\u5177\u8fdb\u884c\u4e86\u6bd4\u8f83\uff0c\u7a81\u51fa\u4e86\u5176\u5728\u6613\u7528\u6027\u3001\u591a\u529f\u80fd\u6027\u548c\u4e0e\u5927\u578b\u8bed\u8a00\u6a21\u578b\u96c6\u6210\u65b9\u9762\u7684\u4f18\u52bf\u3002<\/p>\n\n\n\n
        \u901a\u8fc7\u91c7\u7528 ScrapeGraphAI\uff0c\u60a8\u53ef\u4ee5\u589e\u5f3a\u60a8\u7684\u7f51\u7edc\u6293\u53d6\u529f\u80fd\u5e76\u7b80\u5316\u60a8\u7684\u6570\u636e\u63d0\u53d6\u6d41\u7a0b\uff0c\u4ece\u800c\u66f4\u8f7b\u677e\u5730\u6536\u96c6\u548c\u5206\u6790\u6240\u9700\u7684\u4fe1\u606f\u3002<\/p>","protected":false},"excerpt":{"rendered":"
        Web scraping (proxy for Web Scraping) has evolved significantly with the advent of powerful tools and libraries. One of the latest and most impactful tools in this space is ScrapeGraphAI. But what exactly is ScrapeGraphAI, and how can it revolutionize your web scraping tasks? Let’s dive into the details, exploring its features, capabilities, and how […]<\/p>\n","protected":false},"author":1,"featured_media":923140,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"inline_featured_image":false,"footnotes":""},"categories":[1],"tags":[],"class_list":["post-923141","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-articles"],"acf":[],"_links":{"self":[{"href":"https:\/\/proxyelite.info\/cn\/wp-json\/wp\/v2\/posts\/923141","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/proxyelite.info\/cn\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/proxyelite.info\/cn\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/proxyelite.info\/cn\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/proxyelite.info\/cn\/wp-json\/wp\/v2\/comments?post=923141"}],"version-history":[{"count":1,"href":"https:\/\/proxyelite.info\/cn\/wp-json\/wp\/v2\/posts\/923141\/revisions"}],"predecessor-version":[{"id":926431,"href":"https:\/\/proxyelite.info\/cn\/wp-json\/wp\/v2\/posts\/923141\/revisions\/926431"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/proxyelite.info\/cn\/wp-json\/wp\/v2\/media\/923140"}],"wp:attachment":[{"href":"https:\/\/proxyelite.info\/cn\/wp-json\/wp\/v2\/media?parent=923141"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/proxyelite.info\/cn\/wp-json\/wp\/v2\/categories?post=923141"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/proxyelite.info\/cn\/wp-json\/wp\/v2\/tags?post=923141"}],"curies":[{"name":"\u53ef\u6e7f\u6027\u7c89\u5242","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}