pixiv downloader 20130222c

Change Log:

  • Update parser to the latest pixiv layout.
  • Update last page detection (20130222b).
  • Update sanitizeFilename: remove new line marker ‘rn’ (20130222c).

Download link for pixiv downloader 20130222c, source code in GitHub.

If you are using the source code version and encounter ValueError: invalid literal for int() with base 10: '<something>', please modify _html.py from mechanize library with this pastebin.

47 thoughts on “pixiv downloader 20130222c”

  1. Three questions:

    How does one modify _html.py? Is it the same thing as _html.pyc?

    I appended %R-18% to the filenameformat. It looks like this: %urlFilename% – %title% %R-18%
    However, it leaves a space for non-R-18 images.
    How do I retain a space for R-18 and leave it out for non-R-18?

    Lastly, can I exclude the thumbnails for manga downloads?

    Thanks! :3

    1. How does one modify _html.py? Is it the same thing as _html.pyc?

      No, .pyc is the compiled .py script.
      Search for the file. If you are using Windows, usually it is on C:Python27Libmechanize.
      Search for def unescape_charref(data, encoding): and replace the code from the pastebin.

      I appended %R-18% to the filenameformat. It looks like this: %urlFilename% – %title% %R-18%
      However, it leaves a space for non-R-18 images.
      How do I retain a space for R-18 and leave it out for non-R-18?

      Lastly, can I exclude the thumbnails for manga downloads?

      Should be fixed in the latest version.

      1. Thanks.

        Thumbnail issue is fixed, but the filename one still persists.

        The filenames return as:

        123 – abc R-18.jpg (for R-18, as it should be)
        456 – def .jpg (for non-R-18, giving an unwanted space)

        Alternatively I put an underscore in place of the space, but ideally I’d like to eliminate any other character (including spaces) that appears after %title% _if_ the image is non-R-18.

  2. Not bug report or such, but something I’d like to know if it’s possible, there are some options that are still unclear to me.

    Let’s say, I’m without connection for a while (may happen to me in the future) and when I’m back online, I want to download all the pics my bookmarked artists published in the meantime; option 8 would be useless because the new pics page keeps “only” the latest 2000 (yes, I’ve bookmarked a lot of artists :p)

    Using the “e” option I can generate a list of all the member_ids of my bookmarks but the main question is, can I set the program to not download pictures older than X days before the system date?

    If that is not possible, I’d like to suggest it as a future feature. I speak as a “profane” of programming but, noticing how the downloader also reads the works’ dates (in fact, you can set it so that the filenames also include the dates), the implementation _may_ be possible.

  3. I can’t seem to get he manga mode to work properly. It saves each page in a different folder, instead of having the entire manga in one folder.
    How do I fix this?

        1. Weird. Tried in my pc, each manga will create a directory containing the pages as configured.

          D:TempPixiv鳴滝しん - 誰か助けて~! 34072502_big
          - _p0.jpg
          - _p1.jpg
          D:TempPixivトイレ籠 - みっくみくな反応171 33928250_big
          - _p0.jpg
          - _p1.jpg

      1. I tried the configuration I posted here again (after trying several other things) and for some mysterious reason it suddenly works fine even though the same configuration didn’t work yesterday.
        Strange, but if everything’s working then it’s fine.

        Thanks for the great app. I’m pretty much using it exclusively now for my art needs.
        It’s slow but I rather save art from pixiv than danbooru or the likes.

  4. This is a godsend. I was using an earlier version a while ago, which I noticed stopped working recently. Now I’ve moved onto this one everything works fine. Thanks!

  5. Oh and I’ve just encountered this error while downloading:

    Image #579
    Processing Image Id: 32999235
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 568, in processImage
    File “PixivModel.pyc”, line 213, in __init__
    File “PixivModel.pyc”, line 44, in __init__
    File “PixivModel.pyc”, line 61, in ParseInfo
    AttributeError: ‘NoneType’ object has no attribute ‘find’
    Error at processImage(): (, AttributeError(“‘N
    oneType’ object has no attribute ‘find'”,), )
    Dumping html to: Error Medium Page for image 32999235.html
    Error at processNewIllustFromBookmark(): (, At
    tributeError(“‘NoneType’ object has no attribute ‘find'”,), )
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 1482, in main
    File “PixivUtil2.py”, line 1280, in menuDownloadNewIllustFromBookmark
    File “PixivUtil2.py”, line 974, in processNewIllustFromBookmark
    File “PixivUtil2.py”, line 568, in processImage
    File “PixivModel.pyc”, line 213, in __init__
    File “PixivModel.pyc”, line 44, in __init__
    File “PixivModel.pyc”, line 61, in ParseInfo
    AttributeError: ‘NoneType’ object has no attribute ‘find’
    press enter to exit.

    1. can you retry again? I’ve tried to download the individual image (32999235) and it working. Looks like you are logged out when downloading the image (check the dump file => Error Medium Page for image 32999235.html)

      1. Yep seems like I’ve logged out while downloading. Was able to finish downloading when retried.

        Thanks for the help.

  6. Hey thanks for the update. I’ve got two questions:

    – It this problem caused by me or from the server? I’ve got this error for couple of images while downloading new new images from bookmark
    Image #267
    Processing Image Id: 33502819
    Title: イケこーりん
    Tags : だが代金は払わない, どんぐり, なにこれイケメン, イケメン, 東方, 森近霖之

    Mode : big
    Image URL : http://i2.pixiv.net/img18/img/kuronekone_ro/33502819.jpg
    Filename : C:UsersEricDesktoppixivutil20130222c一日一枚(4ヶ月目)ネーロ (
    335493)33502819.jpg
    Using Referer: http://www.pixiv.net/member_illust.php?mode=big&illust_id=3350281
    9
    [downloadImage()] HTTP Error 504: Gateway Time-out (http://i2.pixiv.net/img18/im
    g/kuronekone_ro/33502819.jpg)
    1 2 3 4
    Using Referer: http://www.pixiv.net/member_illust.php?mode=big&illust_id=3350281
    9
    [downloadImage()] HTTP Error 504: Gateway Time-out (http://i2.pixiv.net/img18/im
    g/kuronekone_ro/33502819.jpg)
    1 2 3 4
    Using Referer: http://www.pixiv.net/member_illust.php?mode=big&illust_id=3350281
    9
    [downloadImage()] HTTP Error 504: Gateway Time-out (http://i2.pixiv.net/img18/im
    g/kuronekone_ro/33502819.jpg)
    1 2 3 4
    Using Referer: http://www.pixiv.net/member_illust.php?mode=big&illust_id=3350281
    9
    [downloadImage()] HTTP Error 504: Gateway Time-out (http://i2.pixiv.net/img18/im
    g/kuronekone_ro/33502819.jpg)
    Giving up url: http://i2.pixiv.net/img18/img/kuronekone_ro/33502819.jpg

    – Is there any ways to speed up the download? as it takes a fair amount of time to download in pages (i.e when downloading
    For example, it takes around approximately 8~9 minutes to download 50 images (some includes manga) – i.e. 5~12 second for each picture with the size between 200KB~1MB.
    I don’t know if this is normal or not but if possible is there any ways to improve the speed?

    Thanks,

    1. 504: usually network error, either from your side or their side.

      No way to speed up the download speed, except if you want to code it yourself.

      1. [*re-send*
        Sorry, i this gets posted 2-3x; my reply wasn’t showing up above so i decided to make a new post since i wasn’t sure if my comment got posted correctly. I am the same kiumi in the post, please just ctrl+f to find ..OTL]

        My version is newly download and extracted at my desktop so i think nothing should be missing? =3
        i have also not edited anything in config.ini yet

        http://i444.photobucket.com/albums/qq170/arunade/1_zps03ba5ddf.png
        http://i444.photobucket.com/albums/qq170/arunade/2_zpsd0ac447c.png

        Thanks again for helping!

        1. Are you running from batch file? Can you upload your batch file to pastebin?

          If not, then just enter the start page (e.g.:1) and later the end page (e.g.: 0 for all pages)

  7. Hey^^
    I seem to have problem getting the program to run.

    2013-02-28 14:09:10,342 – PixivUtil20130222c – INFO – ###############################################################
    2013-02-28 14:09:10,354 – PixivUtil20130222c – INFO – Starting…
    2013-02-28 14:09:10,357 – PixivUtil20130222c – ERROR – Error at loadConfig()
    Traceback (most recent call last):
    File “PixivConfig.pyc”, line 71, in loadConfig
    File “PixivHelper.pyc”, line 223, in OpenTextFile
    File “codecs.pyc”, line 881, in open
    IOError: [Errno 2] No such file or directory: u’C:\Users\User\Desktop\pixivutil20130222c\config.ini’
    2013-02-28 14:09:10,447 – PixivUtil20130222c – INFO – Only process member where day last updated >= 7
    2013-02-28 14:09:15,661 – PixivUtil20130222c – INFO – Log in using form.
    2013-02-28 14:09:36,693 – PixivUtil20130222c – INFO – Logged in
    2013-02-28 14:09:52,885 – PixivUtil20130222c – INFO – Member id mode.
    2013-02-28 14:17:10,473 – PixivUtil20130222c – INFO –

    I want to download using member_id , but whenever i get to the point where i enter the # for the id it doesn’t do anything.
    Any suggestions or am i doing something wrong? ;3;)//

      1. [ gosh XD i’m really having a bit of a difficulty replying to posts here. Sorry for any inconvenience ]

        in answer to your last question: i don’t think i’m running from batch file.
        Entering the start page & end page worked for my problem though! Thank you so much!

  8. Hi,
    I have a ‘little’ bug since the latest update (I don’t know why), i have a error like #13 (aguoloaguolo), but not exactly the same. (http://pastebin.com/X5EbSBrb)

    Setting of Pivix is on English (tried to switch in Japanese on Pixiv / regional language), but same problem as above (paste)

    Info:
    OS : Windows 7 SP1 (Home Prenium) (*Fresh Install and fully update*)
    Pixiv : English or Japanese
    Regional Language : English (Bug) or Japanese (99% of the time for play VNs) but bug too.

    1. Looks like the returned page still in French (u'end_span a xe9txe9 approuvxe9e.n{{#if more}}' ==> end_span a été approuvée.
      {{#if more}}
      .

      Double check the language setting in pixiv.net and try to remove the cookie from config.ini and relogin?

  9. Hi, I’ve been getting this error every time I’ve been using 8: http://pastebin.com/6ac8tTg3

    I always remove the %title% bit from the filenameformat and manga sections so I’m not sure if that has something to do with it now.

    That said, thank you so much for constantly updating this handy program.

  10. Hello,
    First, thank you for this great piece of program i’ve been already using for quite some time. I trully appreciate the regular updates.

    for the 1st time in 2 years, i’m meeting this error:

    Traceback (most recent call last):
    File “PixivUtil2.py”, line 414, in processMember
    File “BeautifulSoup.pyc”, line 1519, in __init__
    File “BeautifulSoup.pyc”, line 1144, in __init__
    File “BeautifulSoup.pyc”, line 1186, in _feed
    File “sgmllib.pyc”, line 104, in feed
    File “sgmllib.pyc”, line 143, in goahead
    File “sgmllib.pyc”, line 320, in parse_endtag
    File “sgmllib.pyc”, line 358, in finish_endtag
    UnicodeEncodeError: ‘ascii’ codec can’t encode character u’xe9′ in position 11:
    ordinal not in range(128)
    Error at processing Artist Info: (, Unicod
    eEncodeError(‘ascii’, u’end_span a xe9txe9 approuvxe9e.n{{#if more}}’, 11, 1
    2, ‘ordinal not in range(128)’), )
    1 2 3 4

    Any idea ? I’ve tried changing regional setting or using something like applocale, but same results.

        1. It is working in my pc using the 20130222c.

          Are you running from source code or exe? If from script, try to update the library. If not, try to set up you computer regional setting to japanese.
          In pixiv website, what is the language you are using? Only English and Japanese are supported.

    1. That was the solution. Thank you Nandaka.
      My language setting inside my pixiv profile was set on French. Had to set it to english and the problem disappeared.
      As for the regional settings on Windows (Windows 7), I’m still on french and it has never been a problem.

  11. I am getting issues when trying to update from member ID 3104565 (R-18 warning) who has worked previously, and also this specific picture 33725282 by a different artist. Occurs with current Git version, I can post the log if necessary.

      1. I’m using the Github version of PixivUtil2 on Debian Testing. Applying that change did fix it, but I’m curious why that’s necessary. Mechanize hasn’t been updated for at least a year and has never caused issues with PixivUtil2 before that.

  12. Good to have it working again 🙂

    I notice that now the download progress for each pic is almost never visibie during the download, only when finished, it’s not vital but would be nice if it could be fixed. Also, it seems a bit slower to me because it must elaborate longer (due to the necessary changes to adapt to the new layout), or is it just a false impression?

    1. I assume you are running from the source code? There is a bug on the mechanize library on the _html.py file. You need to modify the file to handle this.

  13. while using 8. Download new illust from bookmark
    from page 1 to page 10, it will only process page 1,then said Limit or last page reached.
    Seems it only process 1 page…
    Please fix. Thank you!

      1. 2013-02-23 01:15:46,845 – PixivUtil20130222 – INFO – ###############################################################
        2013-02-23 01:15:46,845 – PixivUtil20130222 – INFO – Starting…
        2013-02-23 01:15:46,855 – PixivUtil20130222 – INFO – Using proxy: 127.0.0.1:8087
        2013-02-23 01:15:46,861 – PixivUtil20130222 – INFO – Only process member where day last updated >= 7
        2013-02-23 01:15:46,862 – PixivUtil20130222 – INFO – Using Username: rogerlhc
        2013-02-23 01:15:46,867 – PixivUtil20130222 – INFO – logging in with saved cookie
        2013-02-23 01:15:46,869 – PixivUtil20130222 – INFO – Trying to log with saved cookie
        2013-02-23 01:15:49,401 – PixivUtil20130222 – INFO – Logged in using cookie
        2013-02-23 01:15:51,676 – PixivUtil20130222 – INFO – New Illust from Bookmark mode.
        =====

        PixivDownloader2 version 20130222
        https://nandaka.wordpress.com/tag/pixiv-downloader/
        Reading F:pixivwatch downloaderconfig.ini …
        done.
        Using proxy: *****
        Creating database… done.
        Only process member where day last updated >= 7
        Using Username:******
        logging in with saved cookie
        Trying to log with saved cookie
        done.
        PixivDownloader2 version 20130222
        https://nandaka.wordpress.com/tag/pixiv-downloader/
        1. Download by member_id
        2. Download by image_id
        3. Download by tags
        4. Download from list
        5. Download from online user bookmark
        6. Download from online image bookmark
        7. Download from tags list
        8. Download new illust from bookmark
        9. Download by Title/Caption
        10. Download by Tag and Member Id
        11. Download Member Bookmark
        ————————
        d. Manage database
        e. Export online bookmark
        x. Exit
        Input: 8
        Start Page (default=1):
        End Page (default=0, 0 for no limit): 20
        Processing New Illust from bookmark
        Page #1
        Image #1
        Processing Image Id: 33771273
        Already downloaded: 33771273
        Image #2
        Processing Image Id: 33770482
        Already downloaded: 33770482
        Image #3
        Processing Image Id: 33770026
        Already downloaded: 33770026
        Image #4
        Processing Image Id: 33769368
        Already downloaded: 33769368
        Image #5
        Processing Image Id: 33766281
        Already downloaded: 33766281
        Image #6
        Processing Image Id: 33765565
        Already downloaded: 33765565
        Image #7
        Processing Image Id: 33764693
        Already downloaded: 33764693
        Image #8
        Processing Image Id: 33764129
        Already downloaded: 33764129
        Image #9
        Processing Image Id: 33761936
        Already downloaded: 33761936
        Image #10
        Processing Image Id: 33761591
        Already downloaded: 33761591
        Image #11
        Processing Image Id: 33761345
        Already downloaded: 33761345
        Image #12
        Processing Image Id: 33756458
        Already downloaded: 33756458
        Image #13
        Processing Image Id: 33755349
        Already downloaded: 33755349
        Image #14
        Processing Image Id: 33754619
        Already downloaded: 33754619
        Image #15
        Processing Image Id: 33753584
        Already downloaded: 33753584
        Image #16
        Processing Image Id: 33753143
        Already downloaded: 33753143
        Image #17
        Processing Image Id: 33751108
        Already downloaded: 33751108
        Image #18
        Processing Image Id: 33749731
        Already downloaded: 33749731
        Image #19
        Processing Image Id: 33749298
        Already downloaded: 33749298
        Image #20
        Processing Image Id: 33747989
        Already downloaded: 33747989
        Limit or last page reached.
        Done.
        PixivDownloader2 version 20130222
        https://nandaka.wordpress.com/tag/pixiv-downloader/
        1. Download by member_id
        2. Download by image_id
        3. Download by tags
        4. Download from list
        5. Download from online user bookmark
        6. Download from online image bookmark
        7. Download from tags list
        8. Download new illust from bookmark
        9. Download by Title/Caption
        10. Download by Tag and Member Id
        11. Download Member Bookmark
        ————————
        d. Manage database
        e. Export online bookmark
        x. Exit
        Input:

Comments are closed.