pixiv downloader 20130330

Change Log:

  • Add Ctrl+C handler for download by member id and download by tags to allow user to skip to next image or abort the whole process.
  • Add option to keep signed in, set keepsignedin = 1 in config.ini on [Authentication] section.
  • Add login error message parser.
  • Add download time and speed.
  • Backup config.ini when saving to config.ini.bak.
  • Fix command line option for download by member id and tags to allow single tag search.

Download link for pixiv downloader 20130330, source code in GitHub.

67 thoughts on “pixiv downloader 20130330”

  1. >> I don’t know if img-inf have something to do (temporary?). Maybe I’ll add a check, but worst case if all the images are using img-inf, then no member token can be parsed.

    Did you ran some test?

    ————

    As a feature request, I’d like to be able to download only page xx from my bookmark (or from xx to yy). An option in the choice 5 would do I think (start at x end at y, like the dialog for download by member_id).

    1. >> Did you ran some test?
      What test :D, I do checking after someone ask about it.

      >> I’d like to be able to download only page xx from my bookmark
      Need to add more image in my bookmark then, I’ll see about it.

  2. Is there a way to do writeimageinfo for already downloaded images? I only turned it on recently but I’d like the info for past downloads too, without having to re-download everything again.

    Thank you!

    1. After a quick test, looks like the priority still lies in the filenameformat. I’ll sleep for now and look into it tommorow. 🙂

        1. This is because of the way they structure the images. E.g.: in 199669, the avatar is empty, so I check the first image in the list (illust_id=34846113) to get the token based on the thumbnail url. Somehow, for this images, the url format, http://i2.pixiv.net/img-inf/img/2013/04/07/03/08/21/34846113_s.jpg, is not the usual one. This cause the token to be parsed to 21

          If you compare to the 2nd image, the thumb url is http://i2.pixiv.net/img14/img/blue16/28407891_s.jpg, which parse to blue16.

          I don’t know if img-inf have something to do (temporary?). Maybe I’ll add a check, but worst case if all the images are using img-inf, then no member token can be parsed.

  5. Interesting enough, after I manually added my cookie in the config file it works again. I wonder what causes this, the system is running Ubuntu so the chance is that one of the libs got updated causing to break some things. (Bleeding edge, eh.)

    I’m assuming it is going to break again once the cookie expires. Which is a shame as I tend to run PixivUtil every 15 minutes or so with a cronjob.

      1. Alright, fair enough. I suppose that solves the problem for now. Thank you very much for your support, this program has been a big help for almost a year now. I love it.

  6. Since a couple of days ago PixivUtil doesn’t download any images anymore and just returns me this when trying to download a image http://pastebin.com/wLd3hJWF

    I’m running the latest source from GitHub on Linux. It used to work perfect until a few days ago.

    1. Can you upload the dump file: Error Medium Page for image 35196562.html? I tried on my pc using option 1 (member_id:3816549) and option 2 (image_id:35196562), but it is working properly.

        1. Try to open the page on the browser, it says you are not logged in.

          Can you upload the full log file (delete it first, try to replicate the error, and then upload the log file)? See where is the return url (Logging in, return url: http://www.pixiv.net/mypage.php)

        1. I think I need to update the post-login checking. Anyway, for your case, can you check your login information? Try to delete the cookies and use ssl.

      3. If I delete the cookie the same thing happens, however. It doesn’t write any new cookie information into the config file. SSL was off, turning it on didn’t make any difference. If I turn r18mode on it says it can’t find any more images, even after making a new database. Perhaps it’s because I’m running it on Linux, though it used to work fine for months.

        1. try to change your password to simpler one (e.g.: 1234) and turn on the debughttp, see whether the login information is passed or not.

          Worst case, copy you cookie from browser to config.ini.

    2. I made a new Pixiv account and kept the password to 12346, here’s the output from the console (Username replaced with USRNAME for safety measures) http://pastebin.com/XXqYGktw

      I’d like to try using the cookie from my browser, however. Could you provide me the name of the cookie I should copy the value from?

      1. PHPSESSID, I’ll check the log, somehow it redirect you back to index.php. I don’t know if the p_ab_id (A/B testing?) cookie have any effect.

        There also a difference when the data being sent to login.php, somehow the PHPSESSID is not sent as one of the cookie. Maybe you need to update the library?

  7. Great to see the config.ini backed up now!

    However, when I upgraded my files, it did not detect ‘keepSignedIn’, so it backed up my config file. Then it said my login was expired, and created a new cookie value (which was exactly the same value as the manual backup I made prior to upgrading any files), and immediately overwrote the backup with a new backup. You can see it overwritten here: http://i36.tinypic.com/21c72io.jpg

    My filename and filenamemanga formats were intact, however! Thank you for that! I can also see a ctrl+c handler being very useful in my future! Major thanks!

      1. Perhaps, or maybe just one backup per instance run? Also, I wonder why it thought the cookie was different. It doesn’t seem to think so anymore, however.

  8. Thanks very much! This Program is Wonderful. Next version is add a r-18g mode in config.ini if you can.
    then this tool is more great 🙂

    1. R-18 should include R-18G, except u want only to get the R-18G? Technically, you can use download by member id and tags (option 10).

  10. bad english , sorry!

    Download from image bookmark
    this function can add download my private bookmark_tag optional ? not my bookmark all tag

    1. You means, using option 6 and download from your own bookmarked image (member_id = 0), you want to use your own tags rather than then original image tags for the filename? This is not possible as the information is not on the image page itself.

  11. Would it be possible to download images from your own bookmarks. However could they downloaded and sorted into folders with the tag name, instead of the member’s name and ID? This would be incredibly useful for myself and other people who love to organise.

    1. >> sorted into folders with the tag name

      You can auto create folder based on tag name (see filenameformat in readme.txt), the problem is: it will create a folder with all the tags inside the image post.
      Is it not possible to create a folder only containing the tags you want automatically, except you use the usesuppresstags = true and provide the tags in the suppress_tags.txt file (blacklist).

      There is a filenameformat for %searchTags%, but this only for Download by Tags method, as you provide which tag to use as the folder name.

      1. Sorry I have to clarify myself, when I said tags I meant tags that I had created myself to describe the image, not the artist’s tags.

        1. Never tried it before, so I don’t have the sample page to test. Can you save the sample image page and upload it to Mediafire?
          Checking the description, I think it will add your user id in the tags added by you, is this correct?
          I’ll see what I can do.

      2. Not sure what you wanted me to save, so I saved a few webpages.

        I tried using the option of “Download from image bookmark (http://www.pixiv.net/bookmark.php), including
        private/hidden bookmarks.”

        For the filenameformat (in config.txt) I tried for both the folder name and filename using:
        %bookmark% – did nothing (left a blank space where writing should be)
        %tags% – Tags saved by artist and not myself were incorporated into the folder/file name. (i’ve included download result of this).

        Also, Is it possible to seperate bookmarks by page number? Because I notice that once 1st page is finished it goes on to try download images on the 2nd page (“importing page 2”).

        Mediafire link: http://www.mediafire.com/?63ez5aa27d58fka

        1. >> I tried using the option of “Download from image bookmark (http://www.pixiv.net/bookmark.php), including
          private/hidden bookmarks.
          %bookmark% format currently only work from option 11. Use that one, as this is the updated version. I’ll remove the ability to download another user image bookmark from option 6.

          >> Also, Is it possible to seperate bookmarks by page number
          I don’t think this is useful as the image might be shifted to another page if user add/remove the bookmarks.
          If you mean is limiting the download page range, then it is implemented on option 11.

          >> tags that I had created myself to describe the image, not the artist’s tags.
          Not possible, as it is doesn’t show up in the image page.

  13. Hello there, thanks for this tool. I have a specific feature request: Original datestamp on the downloaded file as found on Pixiv. Using a download manager like DownThemAll!, for example, will give the file and the original datestamp (I don’t know if it’s the datestamp from when a file was uploaded to Pixiv, or the original datestamp of the file as found on the uploader’s local copy). This is useful to have.

    Actually, there is one other thing I noticed, and it suggests another (hopefully quite simple) feature request. I downloaded a file and noticed it was broken (only partially downloaded), and I had to temporarily rename the sqlite database file in order to convince the program it had never downloaded the file and therefore attempt a fresh download. This worked, but would it be possible to include a menu entry which forces redownload (using download from a specific image tag should be all that is needed for the vast majority of cases; otherwise the database should probably be temporarily renamed or even scrapped anyway)?

    1. >> Original Timestamp
      Dunno how to get it, why it is useful?

      >> forces redownload
      Check alwaysCheckFileSize for config.ini in readme.txt, or use the DB Manager (press d on main menu) and select delete image by image id, or you can delete the file and do DB clean up from DB Manager.

      1. >> Original Timestamp
        Useful because, well, it shows the original date of the file. Useful for historical purposes, i.e. to track an artist’s development, to compare different versions of files (i.e. the Pixiv version against a version posted on a blog) which might have minor differences, and so on.

        RE redownloading, I’ll take note of the DB Manager, thanks. It sounds like there are some other useful features of the DB Manager I’ve been neglecting, too 😀

  14. ok, you’ll be angry for this, i just used it a few times and this error appears:

    2013-04-02 23:32:49,260 – PixivUtil20130330 – INFO – EXIT
    2013-04-02 23:32:49,260 – PixivUtil20130330 – INFO – ###############################################################
    2013-04-02 23:32:52,671 – PixivUtil20130330 – INFO – ###############################################################
    2013-04-02 23:32:52,671 – PixivUtil20130330 – INFO – Starting…
    2013-04-02 23:32:52,677 – PixivUtil20130330 – INFO – Only process member where day last updated >= 7
    2013-04-02 23:32:59,181 – PixivUtil20130330 – INFO – Log in using form.
    2013-04-02 23:33:09,174 – PixivUtil20130330 – INFO – Logging in, return url: http://www.pixiv.net/mypage.php
    2013-04-02 23:33:09,176 – PixivUtil20130330 – INFO – Logged in
    2013-04-02 23:33:10,684 – PixivUtil20130330 – INFO – Member id mode.
    2013-04-02 23:33:27,221 – PixivUtil20130330 – INFO – Processing Member Id: 93360
    2013-04-02 23:33:27,221 – PixivUtil20130330 – INFO – End Page: 2
    2013-04-02 23:33:27,378 – PixivUtil20130330 – INFO – Member Url: http://www.pixiv.net/member_illust.php?id=93360&p=1
    2013-04-02 23:33:42,253 – PixivUtil20130330 – ERROR – Error at processing Artist Info: (, AttributeError(“‘NoneType’ object has no attribute ‘ul'”,), )
    2013-04-02 23:33:42,253 – PixivUtil20130330 – ERROR – Error at processing Artist Info: 93360
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 420, in processMember
    File “PixivModel.pyc”, line 42, in __init__
    File “PixivModel.pyc”, line 101, in ParseImages
    AttributeError: ‘NoneType’ object has no attribute ‘ul’
    2013-04-02 23:33:46,259 – PixivUtil20130330 – INFO – Member Url: http://www.pixiv.net/member_illust.php?id=93360&p=1

    Any idea?

        1. no dump file 🙁 inside. Anyway, can you go to Pixiv website and set to english language, then edit config.ini and remove the cookie information, then try again? I’ll update the application to generate dump file if encounter any error on the weekend.

  15. Maaaaaaan, you saved my lifeeee, i had like 10 artists waiting for a gallery download xDDDDD, Thank you!

  16. So i have a problem every start. (・ヘ・?）

    ” 2013-03-31 19:08:58,663 – PixivUtil20130330 – ERROR – Error at processMember(): (, ValueError(“time data ‘2.3.2013 20:13’ does not match format ‘%Y-%m-%d %H:%M'”,), ) Member Id: 369356
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 492, in processMember
    File “PixivUtil2.py”, line 591, in processImage
    File “PixivModel.pyc”, line 226, in __init__
    File “PixivModel.pyc”, line 313, in ParseWorksData
    File “_strptime.pyc”, line 325, in _strptime
    ValueError: time data ‘2.3.2013 20:13’ does not match format ‘%Y-%m-%d %H:%M’ “

        1. delete your cookie entry in config.ini, then retry after you set the language to english from pixiv website. If still have error, please upload the page to mediafire.

  17. Yup, this fixed the logging in problem I had before. Thanks.
    Will keep you updated if I hit any more snags.

    1. Whoops, sorry. I’m an idiot with these things, and I need your help once more.

      I’m trying to download images from my own bookmarks (images I bookmarked myself), and I don’t think I have any private images. I got this error:

      http://prntscr.com/yhf0g

      Am I using the wrong command here?

        1. Weird, I tried to parse the page and it is working.
          It is returning [20609344, 27923863, 17550202, 33490758, 25221094, 23215877, 28324516, 1336118, 32586414, 31524392, 30255682, 29938244, 27321420, 26303661, 25162485, 22076151, 11648522, 32230541, 14155233, 32221699], which is the list of image id in your bookmark.

          Are you using exe version or source code version? If you are using source code version, update your library for mechanize and BeautifulSoup. No changes on the parse method from v20130330 version.

      5. Alright, so what I did was try to simply download ONE image, using one of the image IDs you provided.
        http://prntscr.com/yyt1n

        And this happened.
        Hmm.

        I tried disabling all the userscripts and junk, and it still doesn’t work…
        This is depressing…

      6. When I tried for my bookmarks, same error as my first post.

        When I tried for single image, (used 31524392), I got same error as second screenshot.
        However, it’s worth noting that this .html file appeared in the same directory, after the single image error.
        http://www.mediafire.com/view/?y233cyyqlt9ybfw

        Also, does userscripts even affect anything? I doubt it does.

          1. upload pixivutil.log to mediafire, or try to find this line: INFO - Logging in, return url:. It should not return login.php.

          3. That’s weird, if you open the html file, it says you need to log-in to Pixiv. Is R-18 is enabled in your account settings in Pixiv Website?

Comments are closed.