pixiv downloader 20141004c

Change Log for 20141004c:

  • Update parser.
  • Update download logic for manga page (20141004b)
  • Update parser for the new 2 page manga mode (20141004c)

Download link for pixiv downloader 20141004c, source code in GitHub.

Tested working properly on 2014-10-03 for download by member id (normal, manga, and ugoira), unless Pixiv change their css again…

You guys better donate so I can buy more  figures, got 2 on preorder for this month 😛

Donation Link on the side bar ==> 😀

UPDATED!!!!!

25 thoughts on “pixiv downloader 20141004c”

  1. So the program will be downloading fine for a long while, then for a reason that I’m not aware of something breaks.

    Like this: http://puu.sh/c0mbS.png

    It seems it goes through the rest of the artists on the list, but it fails for all of them.

    Also the error pages don’t seem to work. When I click them I get this: http://puu.sh/c0iWv.png

    Clicking “here” just ends up on that same page.

    1. Looks like you got network issue for this case, for whatever reason, the page is redirected to 1.1.1.1.

      By right, the dump page should contains pixiv data.

      Check if your firewall/av?

  2. Hi!
    There are some rare problem on certain pages. I download by tags and have no problem except on those three pics : 46336451, 46297438 and 46360787
    Here are the messages :

    Image #81
    Image Id: 46336451
    Bookmark Count: 7
    Processing Image Id: 46336451
    Espace insuffisant pour traiter cette commande.
    Title: ????
    Tags : R-18G, ??????? ????? ????? ??? ??? ???? ???
    Date : 2014-10-04 03:01:00
    Mode : manga
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 684, in process_image
    File “PixivModel.pyc”, line 371, in ParseImages
    File “PixivModel.pyc”, line 406, in CheckMangaType
    File “PixivModel.pyc”, line 428, in ParseMangaImagesNew
    AttributeError: ‘NoneType’ object has no attribute ‘findAll’
    Error at process_image(): (, AttributeError(“‘
    NoneType’ object has no attribute ‘findAll'”,),
    )
    Cannot medium dump page for image_id: 46336451
    Error at process_tags(): (, UnboundLocalErr
    or(“local variable ‘parse_medium_page’ referenced before assignment”,), )
    Dumping html to: Error page for search tags %E8%AA%BF%E6%95%99+OR+%E8%86%A3%E5%8
    6%85%E6%94%BE%E5%B0%BF+OR+%E8%82%89%E4%BE%BF%E5%99%A8+OR+%E8%82%89%E5%A5%B4%E9%9
    A%B7+OR+%E8%99%9A%E3%82%8D%E7%9B%AE+OR+%E7%B2%BE%E6%B6%B2%E4%BE%BF%E6%89%80+OR+%
    E9%A6%96%E8%BC%AA+OR+%E8%86%A3%E9%8F%A1+OR+%E4%B8%89%E7%A9%B4+OR+%E4%BA%8C%E7%A9
    %B4+OR+%E8%BC%AA%E5%A7%A6+OR+%E6%BA%A2%E3%82%8C%E7%B2%BE%E6%B6%B2+OR+%E6%B1%9A%E
    3%81%84%E3%81%8A%E3%81%A3%E3%81%95%E3%82%93+OR+%E3%83%93%E3%83%83%E3%83%81+OR+%E
    5%B7%A8%E6%A0%B9+OR+%E7%97%B4%E6%BC%A2+OR+%E9%99%B5%E8%BE%B1+OR+%E3%81%9A%E3%82%
    89%E3%81%97%E6%8C%BF%E5%85%A5+OR+%E3%83%AD%E3%83%AA%E3%83%93%E3%83%83%E3%83%81+O
    R+%E7%AC%91%E9%A1%94%E5%BC%B7%E5%88%B6+OR+%E8%BA%AB%E4%BD%93%E3%81%AB%E8%90%BD%E
    6%9B%B8%E3%81%8D+OR+%E6%AD%A3%E3%81%AE%E5%AD%97+OR+%E3%81%84%E3%81%98%E3%82%81+O
    R+%E7%AC%91%E9%A1%94%E5%BC%B7%E5%88%B6+OR+%E5%A3%B2%E6%98%A5+OR+%E7%97%B4%E5%A5%
    B3+OR+%E6%8F%B4%E5%8A%A9%E4%BA%A4%E9%9A%9B+OR+%E6%8F%B4%E4%BA%A4+OR+%E5%A3%B2%E6
    %98%A5.html
    Error at process_tags_list(): (, UnboundLoc
    alError(“local variable ‘parse_medium_page’ referenced before assignment”,), )
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 1813, in main
    File “PixivUtil2.py”, line 1621, in main_loop
    File “PixivUtil2.py”, line 1496, in menu_download_from_tags_list
    File “PixivUtil2.py”, line 952, in process_tags_list
    File “PixivUtil2.py”, line 897, in process_tags
    File “PixivUtil2.py”, line 785, in process_image
    UnboundLocalError: local variable ‘parse_medium_page’ referenced before assignme
    nt
    press enter to exit.

    Image #236
    Image Id: 46297438
    Bookmark Count: 6
    Processing Image Id: 46297438
    Espace insuffisant pour traiter cette commande.
    Title: ???????????????
    Tags : R-18G, ?????? ??? SM, ??
    Date : 2014-10-02 00:07:00
    Mode : manga
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 684, in process_image
    File “PixivModel.pyc”, line 371, in ParseImages
    File “PixivModel.pyc”, line 406, in CheckMangaType
    File “PixivModel.pyc”, line 428, in ParseMangaImagesNew
    AttributeError: ‘NoneType’ object has no attribute ‘findAll’
    Error at process_image(): (, AttributeError(“‘
    NoneType’ object has no attribute ‘findAll'”,),
    )
    Cannot medium dump page for image_id: 46297438
    Error at process_tags(): (, UnboundLocalErr
    or(“local variable ‘parse_medium_page’ referenced before assignment”,), )
    Dumping html to: Error page for search tags %E3%83%9C%E3%83%B3%E3%83%87%E3%83%BC
    %E3%82%B8+OR+%E5%8F%A3%E6%9E%B7+OR+%E3%82%AF%E3%83%AA%E8%B2%AC%E3%82%81+OR+%E3%8
    3%90%E3%82%A4%E3%83%96+OR+%E7%B7%8A%E7%B8%9B+OR+%E7%B2%BE%E6%B6%B2+OR+%E3%83%90%
    E3%82%A4%E3%83%96IN%E3%83%91%E3%83%B3%E3%83%84+OR+%E3%83%AD%E3%83%BC%E3%82%BF%E3
    %83%BC+OR+%E8%87%AD%E3%81%84+OR+%E9%80%86%E3%83%AA%E3%83%A7%E3%83%8A+OR+%E3%83%9
    2%E3%83%88%E3%82%A4%E3%83%8C+OR+%E3%83%A9%E3%83%86%E3%83%83%E3%82%AF%E3%82%B9+OR
    +%E3%83%A9%E3%83%90%E3%83%BC+OR+%E3%83%9E%E3%82%B9%E3%82%AF%E3%82%AE%E3%83%A3%E3
    %82%B0+OR+%E3%83%9D%E3%83%8B%E3%83%BC%E3%82%AC%E3%83%BC%E3%83%AB+OR+%E6%8B%B7%E5
    %95%8F+OR+%E5%85%A8%E9%A0%AD%E3%83%9E%E3%82%B9%E3%82%AF+OR+%E6%8B%98%E6%9D%9F+OR
    +%E5%8C%82%E3%81%84.html
    Error at process_tags_list(): (, UnboundLoc
    alError(“local variable ‘parse_medium_page’ referenced before assignment”,), )
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 1813, in main
    File “PixivUtil2.py”, line 1621, in main_loop
    File “PixivUtil2.py”, line 1496, in menu_download_from_tags_list
    File “PixivUtil2.py”, line 952, in process_tags_list
    File “PixivUtil2.py”, line 897, in process_tags
    File “PixivUtil2.py”, line 785, in process_image
    UnboundLocalError: local variable ‘parse_medium_page’ referenced before assignme
    nt
    press enter to exit.

    Image #27
    Image Id: 46360787
    Bookmark Count: 8
    Processing Image Id: 46360787
    Espace insuffisant pour traiter cette commande.
    Title: ?????????
    Tags : R-18, ?????? ????? ?
    Date : 2014-10-05 09:12:00
    Mode : manga
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 684, in process_image
    File “PixivModel.pyc”, line 371, in ParseImages
    File “PixivModel.pyc”, line 406, in CheckMangaType
    File “PixivModel.pyc”, line 428, in ParseMangaImagesNew
    AttributeError: ‘NoneType’ object has no attribute ‘findAll’
    Error at process_image(): (, AttributeError(“‘
    NoneType’ object has no attribute ‘findAll'”,),
    )
    Cannot medium dump page for image_id: 46360787
    Error at process_tags(): (, UnboundLocalErr
    or(“local variable ‘parse_medium_page’ referenced before assignment”,), )
    Dumping html to: Error page for search tags %E5%8C%85%E8%8C%8E+OR+%E3%81%B5%E3%8
    1%9F%E3%81%AA%E3%82%8A+OR+%E7%94%B7%E3%81%AE%E5%A8%98+OR+%E5%BD%A2%E3%81%8F%E3%8
    1%A3%E3%81%8D%E3%82%8A%E3%83%9A%E3%83%8B%E3%82%B9+OR+%E3%83%AC%E3%82%BA+OR+%E3%8
    3%95%E3%82%BF%E3%83%8A%E3%83%AA.html
    Error at process_tags_list(): (, UnboundLoc
    alError(“local variable ‘parse_medium_page’ referenced before assignment”,), )
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 1813, in main
    File “PixivUtil2.py”, line 1621, in main_loop
    File “PixivUtil2.py”, line 1496, in menu_download_from_tags_list
    File “PixivUtil2.py”, line 952, in process_tags_list
    File “PixivUtil2.py”, line 897, in process_tags
    File “PixivUtil2.py”, line 785, in process_image
    UnboundLocalError: local variable ‘parse_medium_page’ referenced before assignme
    nt
    press enter to exit.

    Thanks in advance 😉

  3. Yeah, still getting problems with the manga reader. When it encounters these types of manga, it tries a few times, then gives up on everything afterwards.

    Processing Image Id: 32431170
    Title: 幼馴染（♀）に何があった？（ビッチ化）
    Tags : R-18, 漫画, 3Dカスタム少女, ビッチ, ピアス, タトゥー
    Date : 2012-12-28 17:19:00
    Mode : manga
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 684, in process_image
    File “PixivModel.pyc”, line 371, in ParseImages
    File “PixivModel.pyc”, line 406, in CheckMangaType
    File “PixivModel.pyc”, line 428, in ParseMangaImagesNew
    AttributeError: ‘NoneType’ object has no attribute ‘findAll’
    Error at process_image(): (, AttributeError(“‘
    NoneType’ object has no attribute ‘findAll'”,),
    )
    Cannot medium dump page for image_id: 32431170
    Stuff happened, trying again after 2 second ( 1 )
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 506, in process_member
    File “PixivUtil2.py”, line 785, in process_image
    UnboundLocalError: local variable ‘parse_medium_page’ referenced before assignme
    nt

    ….[immediately after attempt #5]

    Giving up image_id: 32431170
    PixivDownloader2 version 20141004c
    https://nandaka.wordpress.com/tag/pixiv-downloader/
    1. Download by member_id

  4. It’s my first time using this. I get this error after “Login using form”

    Error at pixiv_login(): (, <
    httperror_seek_wrapper (mechanize._http.RobotExclusionError instance) at 0x28437
    30 whose wrapped object = <closeable_response at 0x2876620 whose fp = >>, )
    failed
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 1805, in main
    File “PixivUtil2.py”, line 260, in pixiv_login
    File “mechanize_mechanize.pyc”, line 203, in open
    File “mechanize_mechanize.pyc”, line 255, in _mech_open
    httperror_seek_wrapper: HTTP Error 403: request disallowed by robots.txt
    press enter to exit.

    I’ve already put “userobots = False” in config.ini

    1. Thats weird? Are you running in Windows? Check if the application pick the correct config. If you are using shortcut, make sure the working directory is the same with the application location.

      1. Yes, I’m running on Windows, 7 to be precise. I just run PixivUtil2.exe from the extracted folder. I also tried Run as Admin.

  5. I just noticed – this change might have happened some time ago – that the new default INI file now defaults to “overwrite = false” which I assume captures the same variable as the old “Overwrite Old File” variable. Defaulting to “false” is a much appreciated change.

    I will keep a close eye on new downloads – I believe that about the time the Pixiv backend changed some single-image illustrations might have been uploaded with “_0.img” at the end. I think this is something Pixiv Downloader already handles well, however.

    1. Wait, I was confused. I was thinking of “backupoldfile” which is unfortunately set to False by default (I’d rather prune by hand than have the software overwrite unique old versions of files). I also note that using robots is still on by default, but I’ve never had success using the program unless this is set to False. Are there users for which using robots works?

        1. I can understand the reasoning that whatever default settings are selected is your business, and up to the user to set them, but this doesn’t answer my question. whinette confirms it works but why it does is mysterious (and the readme doesn’t shed light on why).

          1. Robots.txt? This is set from the pixiv webserver to avoid robots (e.g. webcrawler, site ripper) to index/download the whole site. You can look the purpose of robots.txt on google.

      2. Yeazh I use robots.txt and the program works nicely (until pixiv break the layout, haha).

  7. Tried downloading new illustrations from bookmark and kept getting this error every single time:

    “UnboundLocalError: local variable ‘parse_medium_page’ referenced before assignment”

    http://pastebin.com/v7FfuZYc

    Not sure what I should do about it. :

  8. There is a problem on (old) manga mode:
    It downloads only pair pages. I’ll put an issue on github.
    Log :

    Image ids: 30507937
    Processing Image Id: 30507937
    Title: 紅楼夢８新刊「共食い禁止令！」

      1. This was already fixed in the git repo by the time nandaka made the comment at #3. Update your install.

        1. Thanks, willdo !
          My juron was about my inability to paste @ windows interpreting control character in my clipboard; not the bug by the way. 😉

          1. You might have less frustration with a proper terminal emulator for cygwin (such as konsole), a virtual machine running linux, a raspberry pi accessed through putty, or barring all that, powershell. 😉

  9. Hello,
    Thanks for updating the Pixiv downloader. It must be frustrating keeping up with all the changes Pixiv have made in the last week.

    It now seems to work fine for everything apart from older manga posts that were made before Pixiv changed. For these, the program will only download even numbered pages (see below). Many thanks.

    Input: 2
    Image ids: 26743783
    Processing Image Id: 26743783
    Title: ?????????????
    Tags : ????? ?????? ??????? ???????? ????????? ?????????????? ??????? ???? ?????
    ??????????? ??????
    Date : 2012-04-23 00:57:00
    Mode : manga
    Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
    llust_id=26743783&page=0
    Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
    llust_id=26743783&page=1
    Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
    llust_id=26743783&page=2
    Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
    llust_id=26743783&page=3
    Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
    llust_id=26743783&page=4
    Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
    llust_id=26743783&page=5
    Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
    llust_id=26743783&page=6
    Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
    llust_id=26743783&page=7
    Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
    llust_id=26743783&page=8
    Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
    llust_id=26743783&page=9
    Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
    llust_id=26743783&page=10
    Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
    llust_id=26743783&page=11
    Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
    llust_id=26743783&page=12
    Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
    llust_id=26743783&page=13
    Page Count : 14
    Image URL : http://i1.pixiv.net/img65/img/kamachi52/26743783_big_p0.png
    Filename : ??????????????????????? (2544637)26743783_big_p0.png
    Using Referer: http://www.pixiv.net/member_illust.php?mode=medium&illust_id=2674
    3783
    Start downloading… 275980 of 275980 Bytes Completed in 2.06s (130.83 Ki
    B/s)
    done.

    Image URL : http://i1.pixiv.net/img65/img/kamachi52/26743783_big_p2.png
    Filename : ??????????????????????? (2544637)26743783_big_p2.png
    Using Referer: http://www.pixiv.net/member_illust.php?mode=medium&illust_id=2674
    3783
    Start downloading… 317646 of 317646 Bytes Completed in 2.25s (137.87 Ki
    B/s)
    done.

    Image URL : http://i1.pixiv.net/img65/img/kamachi52/26743783_big_p4.png
    Filename : ??????????????????????? (2544637)26743783_big_p4.png
    Using Referer: http://www.pixiv.net/member_illust.php?mode=medium&illust_id=2674
    3783
    Start downloading… 253672 of 253672 Bytes Completed in 2.001s (123.80 K
    iB/s)
    done.

    Image URL : http://i1.pixiv.net/img65/img/kamachi52/26743783_big_p6.png
    Filename : ??????????????????????? (2544637)26743783_big_p6.png
    Using Referer: http://www.pixiv.net/member_illust.php?mode=medium&illust_id=2674
    3783
    Start downloading… 142728 of 142728 Bytes Completed in 1.5s (92.92 KiB/
    s)
    done.

    Image URL : http://i1.pixiv.net/img65/img/kamachi52/26743783_big_p8.png
    Filename : ??????????????????????? (2544637)26743783_big_p8.png
    Using Referer: http://www.pixiv.net/member_illust.php?mode=medium&illust_id=2674
    3783
    Start downloading… 211058 of 211058 Bytes Completed in 1.82s (113.25 Ki
    B/s)
    done.

    Image URL : http://i1.pixiv.net/img65/img/kamachi52/26743783_big_p10.png
    Filename : ??????????????????????? (2544637)26743783_big_p10.png
    Using Referer: http://www.pixiv.net/member_illust.php?mode=medium&illust_id=2674
    3783
    Start downloading… 210756 of 210756 Bytes Completed in 1.76s (116.94 Ki
    B/s)
    done.

    Image URL : http://i1.pixiv.net/img65/img/kamachi52/26743783_big_p12.png
    Filename : ??????????????????????? (2544637)26743783_big_p12.png
    Using Referer: http://www.pixiv.net/member_illust.php?mode=medium&illust_id=2674
    3783
    Start downloading… 151201 of 151201 Bytes Completed in 1.5s (98.44 KiB/
    s)
    done.

    1. Actually, I find that there’s a new type of manga view (right to left) that trips up the program, like image ids 46322053, 46320252 and 46320003.

Comments are closed.