pixiv downloader 20120724

Change log:

  • Update makeFilename
    • Clean up double space
  • Update filenameFormat
    • %urlFilename% ==> the actual filename stored in server without the file extensions
  • Add filenameMangaFormat
    • %page_index%  ==> for manga mode, add big in the filename.
    • %page_number% ==> for manga mode, add page number with 0-index.
    • %page_big% ==> for manga mode, add page number with 1-index.

Please note that the createmangadir option needs to have %urlFilename% in the filenameMangaFormat to create the directory, or you can update the format from %image_id% to %image_id%%page_index%.

For issues or new feature request, you can put it in the comment or through GitHub tracker.

Download link for PixivUtil20120724, and source code.

21 thoughts on “pixiv downloader 20120724”

  1. Got some errors just now, seems like I can no longer download pictures. Log file gave:

    2012-08-03 03:00:15,756 – PixivUtil20120724 – ERROR – Error at processImage(): (, AttributeError(“‘NoneType’ object has no attribute ‘string'”,), )
    2012-08-03 03:00:15,773 – PixivUtil20120724 – ERROR – Dumping html to: Error Medium Page for image 19010647.html
    2012-08-03 03:00:19,506 – PixivUtil20120724 – ERROR – Error at processImage(): (, AttributeError(“‘NoneType’ object has no attribute ‘string'”,), )
    2012-08-03 03:00:19,523 – PixivUtil20120724 – ERROR – Dumping html to: Error Medium Page for image 19010647.html
    2012-08-03 03:00:23,240 – PixivUtil20120724 – ERROR – Error at processImage(): (, AttributeError(“‘NoneType’ object has no attribute ‘string'”,), )
    2012-08-03 03:00:23,256 – PixivUtil20120724 – ERROR – Dumping html to: Error Medium Page for image 19010647.html
    2012-08-03 03:00:26,927 – PixivUtil20120724 – ERROR – Error at processImage(): (, AttributeError(“‘NoneType’ object has no attribute ‘string'”,), )
    2012-08-03 03:00:26,944 – PixivUtil20120724 – ERROR – Dumping html to: Error Medium Page for image 19010647.html
    2012-08-03 03:00:30,887 – PixivUtil20120724 – ERROR – Error at processImage(): (, AttributeError(“‘NoneType’ object has no attribute ‘string'”,), )
    2012-08-03 03:00:30,911 – PixivUtil20120724 – ERROR – Dumping html to: Error Medium Page for image 19010647.html
    2012-08-03 03:00:30,913 – PixivUtil20120724 – ERROR – Giving up image_id: 19010647
    2012-08-03 03:00:30,913 – PixivUtil20120724 – INFO – Processing Member Id: 557399
    2012-08-03 03:00:34,029 – PixivUtil20120724 – ERROR – Error at processImage(): (, AttributeError(“‘NoneType’ object has no attribute ‘string'”,), )
    2012-08-03 03:00:34,046 – PixivUtil20120724 – ERROR – Dumping html to: Error Medium Page for image 28020592.html
    2012-08-03 03:00:37,709 – PixivUtil20120724 – ERROR – Error at processImage(): (, AttributeError(“‘NoneType’ object has no attribute ‘string'”,), )
    2012-08-03 03:00:37,726 – PixivUtil20120724 – ERROR – Dumping html to: Error Medium Page for image 28020592.html

    and so on. Haven’t been able to download anything since 3:00 AM for some reason

    1. Likewise. Maybe some recent changes in Pixiv? I was trying to download from my bookmarks and I got this in the logfile.

      2012-08-03 10:38:53,171 – PixivUtil20120724 – INFO – Starting…
      2012-08-03 10:38:53,342 – PixivUtil20120724 – INFO – Only process member where day last updated >= 7
      2012-08-03 10:39:06,140 – PixivUtil20120724 – INFO – Starting…
      2012-08-03 10:39:06,171 – PixivUtil20120724 – INFO – Only process member where day last updated >= 7
      2012-08-03 10:39:06,171 – PixivUtil20120724 – INFO – Using Username: cpgendo
      2012-08-03 10:39:06,171 – PixivUtil20120724 – INFO – logging in with saved cookie
      2012-08-03 10:39:06,171 – PixivUtil20120724 – INFO – Trying to log with saved cookie
      2012-08-03 10:39:25,265 – PixivUtil20120724 – INFO – Logged in using cookie
      2012-08-03 10:39:30,608 – PixivUtil20120724 – INFO – New Illust from Bookmark mode.
      2012-08-03 10:39:53,265 – PixivUtil20120724 – ERROR – Error at processImage(): (, AttributeError(“‘NoneType’ object has no attribute ‘string'”,), )
      2012-08-03 10:39:53,265 – PixivUtil20120724 – ERROR – Dumping html to: Error Medium Page for image 29089153.html
      2012-08-03 10:39:53,265 – PixivUtil20120724 – ERROR – Error at processNewIllustFromBookmark(): (, AttributeError(“‘NoneType’ object has no attribute ‘string'”,), )
      2012-08-03 10:39:53,265 – PixivUtil20120724 – ERROR – Unknown Error: ‘NoneType’ object has no attribute ‘string’
      Traceback (most recent call last):
      File “PixivUtil2.py”, line 1423, in main
      File “PixivUtil2.py”, line 1225, in menuDownloadNewIllustFromBookmark
      File “PixivUtil2.py”, line 939, in processNewIllustFromBookmark
      File “PixivUtil2.py”, line 554, in processImage
      File “PixivModel.pyc”, line 169, in __init__
      File “PixivModel.pyc”, line 217, in ParseInfo
      AttributeError: ‘NoneType’ object has no attribute ‘string’
      2012-08-03 10:40:10,312 – PixivUtil20120724 – INFO – EXIT
      2012-08-03 10:40:10,312 – PixivUtil20120724 – INFO –

  2. I get this error when trying to download from a specific user:

    PixivDownloader2 version 20120724
    https://nandaka.wordpress.com/tag/pixiv-downloader/
    Reading config file… Error at loadConfig(): (, No option ‘filenamemangaformat’ in section: ‘Settings’, )
    Failed to read configuration.
    Writing config file… done.
    done.
    Creating database… done.
    Importing list… Found 2822 items done.
    Updated 2822 items.
    Using Username: stevej018
    logging in with saved cookie
    Trying to log with saved cookie
    done.
    PixivDownloader2 version 20120724
    https://nandaka.wordpress.com/tag/pixiv-downloader/
    1. Download by member_id
    2. Download by image_id
    3. Download by tags
    4. Download from list
    5. Download from online user bookmark
    6. Download from online image bookmark
    7. Download from tags list
    8. Download new illust from bookmark
    9. Download by Title/Caption
    10. Download by Tag and Member Id
    ————————
    d. Manage database
    e. Export online bookmark
    x. Exit
    Input: 1
    Member id: 298409
    Start Page (default=1):
    End Page (default=0, 0 for no limit):
    Processing Member Id: 298409
    Reading config file… done.
    Page 1
    Member Name : dai
    Member Avatar: http://i1.pixiv.net/img17/profile/04061102/4389316.png
    Member Token : 04061102
    HTTP Error 504: Gateway Time-out
    504
    1 2 3 4
    HTTP Error 504: Gateway Time-out
    504
    1 2 3 4
    HTTP Error 504: Gateway Time-out
    504
    1 2 3 4
    HTTP Error 504: Gateway Time-out
    504
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 450, in processMember
    File “PixivUtil2.py”, line 221, in downloadImage
    File “PixivUtil2.py”, line 221, in downloadImage
    File “PixivUtil2.py”, line 221, in downloadImage
    File “PixivUtil2.py”, line 120, in downloadImage
    File “mechanize_mechanize.pyc”, line 203, in open
    File “mechanize_mechanize.pyc”, line 255, in _mech_open
    httperror_seek_wrapper: HTTP Error 504: Gateway Time-out
    Error at processMember(): (,
    <httperror_seek_wrapper (urllib2.HTTPError instance) at 0x28ae730 whose wrapped
    object = <closeable_response at 0x28b9508 whose fp = <response_seek_wrapper at
    0x28b5dc8 whose wrapped object = <closeable_response at 0x28b5da0 whose fp = >>>>, )
    Dumping html to: Error page for member 298409.html
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 1408, in main
    File “PixivUtil2.py”, line 1076, in menuDownloadByMemberId
    File “PixivUtil2.py”, line 450, in processMember
    File “PixivUtil2.py”, line 221, in downloadImage
    File “PixivUtil2.py”, line 221, in downloadImage
    File “PixivUtil2.py”, line 221, in downloadImage
    File “PixivUtil2.py”, line 120, in downloadImage
    File “mechanize_mechanize.pyc”, line 203, in open
    File “mechanize_mechanize.pyc”, line 255, in _mech_open
    httperror_seek_wrapper: HTTP Error 504: Gateway Time-out
    press enter to exit.

  3. I have a question regarding to manga.

    When using option number 1, is there a way to:
    – Exclude manga and download pictures only?
    – Download each manga in separate sub folders?

    If possible, I want to download manga and picture separately or download pictures only excluding manga.

    Once again, thank you for the help.

    1. > – Exclude manga and download pictures only?
      This one is not supported.

      > – Download each manga in separate sub folders?
      Either you set createmangadir = True, or use different filenameMangaFormat from filenameFormat.
      For example: if you set filenameFormat = %member_token% (%member_id%)%urlFilename%, then you can set the filenameMangaFormat = manga%member_token% (%member_id%)%urlFilename% or %member_token% (%member_id%)Manga%urlFilename%

      1. Yep, it worked. I’ve set the createmanga to true and filenamemanga. The manga images saves in separate sub folders.

        One more thing, is there way to not download “folder” image? you know, the artists profile pics.

        Anyway thank you very much.

  4. Hi,nandaka,I’m here again to report something caused by the internet connection.

    Downloaded image file is broken while you’re using a not stable link (I’m in China,GFW),but the downloader said it’s already downloaded.

    After update to 20120724 from 20120704,I can’t log in,back to old version and that’s okay.

    Thank you!

    2012-07-26 21:11:36,605 – PixivUtil20120724 – INFO – Starting…
    2012-07-26 21:11:36,615 – PixivUtil20120724 – INFO – Only process member where day last updated >= 7
    2012-07-26 21:11:36,615 – PixivUtil20120724 – INFO – Using Username: *************
    2012-07-26 21:11:36,625 – PixivUtil20120724 – INFO – logging in with saved cookie
    2012-07-26 21:11:36,630 – PixivUtil20120724 – INFO – Trying to log with saved cookie
    2012-07-26 21:11:56,072 – PixivUtil20120724 – INFO – Cookie already expired/invalid.
    2012-07-26 21:11:56,078 – PixivUtil20120724 – INFO – Log in using form.
    2012-07-26 21:13:55,092 – PixivUtil20120724 – ERROR – Error at pixivLogin(): (, timeout(‘timed out’,), )
    2012-07-26 21:13:55,128 – PixivUtil20120724 – ERROR – Unknown Error: timed out
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 1392, in main
    File “PixivUtil2.py”, line 287, in pixivLogin
    File “mechanize_mechanize.pyc”, line 541, in submit
    File “mechanize_mechanize.pyc”, line 203, in open
    File “mechanize_mechanize.pyc”, line 230, in _mech_open
    File “mechanize_opener.pyc”, line 204, in open
    File “mechanize_urllib2_fork.pyc”, line 457, in http_response
    File “mechanize_opener.pyc”, line 221, in error
    File “mechanize_urllib2_fork.pyc”, line 332, in _call_chain
    File “mechanize_urllib2_fork.pyc”, line 571, in http_error_302
    File “mechanize_mechanize.pyc”, line 203, in open
    File “mechanize_mechanize.pyc”, line 230, in _mech_open
    File “mechanize_opener.pyc”, line 204, in open
    File “mechanize_http.pyc”, line 201, in http_response
    File “mechanize_http.pyc”, line 169, in parse_head
    File “mechanize_response.pyc”, line 195, in read
    File “socket.pyc”, line 380, in read
    File “httplib.pyc”, line 541, in read
    File “httplib.pyc”, line 592, in _read_chunked
    File “httplib.pyc”, line 647, in _safe_read
    File “socket.pyc”, line 380, in read
    timeout: timed out
    2012-07-26 21:21:08,960 – PixivUtil20120724 – INFO – EXIT

      1. using ssl,problem sloved.

        But is there a way to fix downloaded image broken?

        Thanks

      2. opps,but the file info has been already written into the db.sqlite,it told you the file has been downloaded,so I can’t download new illustrations from bookmarks which image file is broken…

        1. Either you set overwrite = True in config.ini, or you delete the corrupted download and do db cleaning (d. Manage database => c. Clean Up Database) to remove the entried. Please note the db cleaning will try to detect if the file is exists/not moved/not renamed in the hdd.

  5. Hi, thanks for the update, but i have one questions. I’m trying to download the images from Pixiv by a tag name, and whenever i choose #3 option, which is download by a tag, it asks me bookmark counts. What do i have to put in the bookmark counts ? Also, when I successfully download the images by a tag, images somehow always separated by their own to different folders with the artists name on it. As a result, I always get a hundreds of different folders with a few pictures that are related to one tag. is there any way to put every images downloaded by tags to one folder ?

    1. put 0 to ignore the bookmark count, it is the little blue star if you go to the search page at pixiv.

      For the filename, please check the filename format, by default it will split based on artist/member_id (check for directory separator or ”), just remove it or change to your taste. Please refer to readme.txt for more information.

  6. Nice update with the manga mode! However, ever thought of implementing a numbering system that starts from “00” or “01” rather than 0 and 1? I mean…

    123456_big_01 rather than pic123456_big_1
    123456_big_02 rather than pic123456_big_2
    123456_big_03 rather than pic123456_big_3
    and so on.

    This would be useful with mangas that are more than 10 pages long, since programs tend to list them wrongly (first 1, then 10-19, then 2, then 20-29…)

    1. which program do you use? The file sorting is showed correctly in explorer. The downloading is not affected by the numbering.

      1. Thanks so much for this update. I look forward to trying this out. I know you didn’t want to add support for leading zeros, but there are all sorts of apps that have poor support for non-leading-zero file name sequences, so a sequence with 20 file would show up as:

        1
        10
        11
        12
        13
        14
        15
        16
        17
        18
        19
        2
        20
        3
        4
        5
        6
        7
        8
        9

        …which is why I feel adding leading zeros is a much better default. It always works.

Comments are closed.