64 thoughts on “pixiv downloader 20131005”

  1. getting an error like this everytime i logging in my account

    Error at pixivLogin(): (, <h
    ttperror_seek_wrapper (mechanize._http.RobotExclusionError instance) at 0x1d9629
    8 whose wrapped object = <closeable_response at 0x1d99828 whose fp = >>, )
    failed
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 1548, in main
    File “PixivUtil2.py”, line 261, in pixivLogin
    File “mechanize_mechanize.pyc”, line 203, in open
    File “mechanize_mechanize.pyc”, line 255, in _mech_open
    httperror_seek_wrapper: HTTP Error 403: request disallowed by robots.txt
    press enter to exit.

    any solution??

  3. 2013-11-19 18:44:14,184 – PixivUtil20131005 – ERROR – Unknown Error: HTTP Error 403: request disallowed by robots.txt
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 1733, in main
    File “PixivUtil2.py”, line 258, in pixiv_login
    File “mechanize_mechanize.pyc”, line 203, in open
    File “mechanize_mechanize.pyc”, line 255, in _mech_open
    httperror_seek_wrapper: HTTP Error 403: request disallowed by robots.txt

        1. Hey Nandaka, could you create a Google+ community for this project? I believe the platform could become something useful for you at sorting out problems like these.

          Furthermore, thank you. I was suffering from this issue today.

  5. I cannot download using tags. Whenever I do I get the following error:

    2013-11-15 04:15:30,796 – PixivUtil20131005 – INFO – ###############################################################
    2013-11-15 04:15:30,796 – PixivUtil20131005 – INFO – Starting…
    2013-11-15 04:15:30,796 – PixivUtil20131005 – INFO – Setting log level to: DEBUG
    2013-11-15 04:15:30,796 – PixivUtil20131005 – INFO – No default cookie jar available, creating…
    2013-11-15 04:15:30,812 – PixivUtil20131005 – INFO – Only process member where day last updated >= 7
    2013-11-15 04:15:30,812 – PixivUtil20131005 – INFO – Using Username: xxxxxxxx
    2013-11-15 04:15:30,812 – PixivUtil20131005 – INFO – logging in with saved cookie
    2013-11-15 04:15:30,812 – PixivUtil20131005 – INFO – Trying to log with saved cookie
    2013-11-15 04:15:34,187 – PixivUtil20131005 – INFO – Logged in using cookie
    2013-11-15 04:15:38,296 – PixivUtil20131005 – INFO – Taglist mode.
    2013-11-15 04:15:42,640 – PixivUtil20131005 – INFO – Searching for: (東方Project10000users入り) %E6%9D%B1%E6%96%B9Project10000users%E5%85%A5%E3%82%8A
    2013-11-15 04:15:42,655 – PixivUtil20131005 – INFO – Looping… for http://www.pixiv.net/search.php?s_mode=s_tag&p=1&word=%E6%9D%B1%E6%96%B9Project10000users%E5%85%A5%E3%82%8A
    2013-11-15 04:15:42,655 – PixivUtil20131005 – ERROR – Error at process_tags(): (, KeyError(u’xe6′,), )
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 877, in process_tags
    File “mechanize_mechanize.pyc”, line 203, in open
    File “mechanize_mechanize.pyc”, line 230, in _mech_open
    File “mechanize_opener.pyc”, line 188, in open
    File “mechanize_http.pyc”, line 320, in http_request
    File “robotparser.pyc”, line 139, in can_fetch
    File “urllib.pyc”, line 1250, in quote
    KeyError: u’xe6′
    2013-11-15 04:15:42,655 – PixivUtil20131005 – ERROR – Error at process_tags_list(): (, KeyError(u’xe6′,), )
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 959, in process_tags_list
    File “PixivUtil2.py”, line 877, in process_tags
    File “mechanize_mechanize.pyc”, line 203, in open
    File “mechanize_mechanize.pyc”, line 230, in _mech_open
    File “mechanize_opener.pyc”, line 188, in open
    File “mechanize_http.pyc”, line 320, in http_request
    File “robotparser.pyc”, line 139, in can_fetch
    File “urllib.pyc”, line 1250, in quote
    KeyError: u’xe6′
    2013-11-15 04:15:42,655 – PixivUtil20131005 – ERROR – Unknown Error: u’xe6′
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 1761, in main
    File “PixivUtil2.py”, line 1498, in menu_download_from_tags_list
    File “PixivUtil2.py”, line 959, in process_tags_list
    File “PixivUtil2.py”, line 877, in process_tags
    File “mechanize_mechanize.pyc”, line 203, in open
    File “mechanize_mechanize.pyc”, line 230, in _mech_open
    File “mechanize_opener.pyc”, line 188, in open
    File “mechanize_http.pyc”, line 320, in http_request
    File “robotparser.pyc”, line 139, in can_fetch
    File “urllib.pyc”, line 1250, in quote
    KeyError: u’xe6′
    2013-11-15 04:15:44,733 – PixivUtil20131005 – INFO – EXIT
    2013-11-15 04:15:44,733 – PixivUtil20131005 – INFO – ###############################################################

    My config.ini file looks like this:

    [Settings]
    proxyaddress =
    useproxy = False
    useragent = Mozilla/5.0 (X11; U; Unix i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9 Firefox/3.0.1
    debughttp = False
    userobots = True
    filenameformat = %R-18%%member_id%_-_%image_id%
    filenamemangaformat = %R-18%%member_id%_-_%image_id%_%page_index%
    timeout = 60
    uselist = False
    processfromdb = True
    overwrite = False
    tagsseparator = ,
    daylastupdated = 7
    rootdirectory = images
    retry = 3
    retrywait = 5
    createdownloadlists = False
    downloadlistdirectory = .
    irfanviewpath = C:Program FilesIrfanView
    startirfanview = False
    startirfanslide = False
    alwayscheckfilesize = False
    checkupdatedlimit = 0
    downloadavatar = False
    createmangadir = False
    usetagsasdir = False
    useblacklisttags = False
    usesuppresstags = False
    tagslimit = -1
    writeimageinfo = False
    datediff = 0
    backupoldfile = False
    loglevel = DEBUG
    enabledump = True
    skipdumpfilter =
    dumpmediumpage = False

    [Pixiv]
    numberofpage = 0
    r18mode = False

    [Authentication]
    username = xxxxxxxx
    password = xxxxxxxx
    cookie =
    usessl = True
    keepsignedin = 0

    I hope you can help me out.

  6. Sorry to bother you but i am getting

    Error at process_image(): (, AttributeError
    (“‘NoneType’ object has no attribute ‘find'”,)

    And reading the Readme said this a problem cause by pixiv changing part of the layout and to tell you about it..

    1. Sure, what is the member/image id? Can you upload the dump html file to mediafire?

      Can you try this:
      1. Open Firefox.
      2. Go to Pixiv website and login, remember to enable [Remember Me] check box.
      3. Right click the page and select View Page Info.
      4. Click the Security tab.
      5. Click the View Cookies button.
      6. Look for Cookie named = PHPSESSID.
      7. Copy the content value.
      8. Open config.ini, go to [Authentication] section, paste the value to cookie, set keepsignedin = 1.

        1. Changing the password? Go to the pixiv website and change your password to the simpler one (e.g. :1234) for testing.

          If still failed, you might need to copy the session id:
          1. Open Firefox.
          2. Go to Pixiv website and login, remember to enable [Remember Me] check box.
          3. Right click the page and select View Page Info.
          4. Click the Security tab.
          5. Click the View Cookies button.
          6. Look for Cookie named = PHPSESSID.
          7. Copy the content value.
          8. Open config.ini, go to [Authentication] section, paste the value to cookie, set keepsignedin = 1.

          1. My password is pretty simple so i’ll skip onto the second method. I will post back if it worked.

    1. Hi, can you enabled this setting in config.ini:
      loglevel = DEBUG
      enabledump = True
      skipdumpfilter =
      dumpmediumpage = False
      and upload the pixivutil.log and the dump html to mediafire?

      Looks like got issue with the password, can you try to use simple password for testing?
      I’ve tried in my pc and it is working.

          1. For some reason i could download some times but when i tried again i couldn’t download anymore even from the same artist i could download just a while ago

          2. Is there any html file generated in the application folder with name similar like: “Dump for member_id Error Code errorCode.html”? Can you upload it to mediafire?

            The error is because of the returned html page from pixiv server, unless I can get the page from you, I cannot check the error.

            Also try to set this in config.ini:
            usessl = True
            keepsignedin = 1

  10. Hey! Incredible work, I love this program.

    One question though, is it possible to pause/resume downloads if I choose download from online user bookmark? I have several hundred bookmarked users and the download would probably end up taking a few weeks. I can’t keep my computer running for that long, so I’d like to be able to pause it, shutdown, and resume when I start my computer back up.

    2. You could also use the “Start Page” and “End Page” option when using the “Download from online user bookmark” to download only one page at a time.

  11. Is there some special sauce that the downloader does when creating the database? I’ve tried to schedule PixivDownloader to periodically download new images from my bookmarked images, but it never gets to creating the database. Here’s what my log is full of.

    PixivDownloader2 version 20131005
    https://nandaka.wordpress.com/tag/pixiv-downloader/
    Reading /Users/—/.pixivutil/config.ini …
    done.

    I’ve tried dumping all of my environmental variables to the launchd document and giving 777 permissions to all the files in my PixivUtil folder and yet, it never gets past that when scheduled. The bash script that the launchd document runs works flawlessly when invoked manually from a login shell.

        1. Try to delete/move the db.sqlite from the application’s folder and run it again. You should see ‘Creating database… done.’ line.

          If not, try to upload the pixivutil.log and config.ini to pastebin

          1. Thanks, but I actually just found the solution to running the downloader with launchd (and probably with cron by extension).

            I initially thought that I could just do “export LANG=en_US.UTF-8” for it to work fine and dandy, but it didn’t. Then after some googling, I found that I could do “export PYTHONIOENCODING=utf-8”, which ultimately did the trick.

          3. An empty SQLite database file was created if I moved my actual database file out of the directory, but it wouldn’t write “Creating database…” to the stdout log nor would the script actually get any further along. Also, if I had to remove the database between runs, it would kinda defeat the purpose of scheduling automatic download of new images from followed artists.

            Now with PYTHONIOENCODING, it works fine. There’s no such environmental variable set when I invoke the script manually though, since it doesn’t come up if I run “env”. If it did, I would’ve probably managed to fix the automated script much sooner.

  12. im getting

    Log in using form.
    Error at pixiv_login(): (, URLError(gaierro
    etaddrinfo failed’),), )
    failed
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 1733, in main
    File “PixivUtil2.py”, line 258, in pixiv_login
    File “mechanize_mechanize.pyc”, line 203, in open
    File “mechanize_mechanize.pyc”, line 230, in _mech_open
    File “mechanize_opener.pyc”, line 193, in open
    File “mechanize_urllib2_fork.pyc”, line 344, in _open
    File “mechanize_urllib2_fork.pyc”, line 332, in _call_chain
    File “mechanize_urllib2_fork.pyc”, line 1142, in http_open
    File “mechanize_urllib2_fork.pyc”, line 1118, in do_open
    URLError:
    press enter to exit.

  13. I’m sorry to asking here this, but I don’t know any other english site who knows about Pixiv. It’s a question about Pixiv site, not the Pixiv Downloader.

    This morning I was browsing some works on Pixiv, went to the 35th page of the search results and then left. Now in the evening I tried to start browsing results from where I stopped and in the same tag, the number of works drastically reduced to no more than 12 pages.
    Even some works I favorited now doesn’t show on the search results anymore, even by looking for the tags.
    Why a lot of works now doesn’t show up anymore in my searches? Is this some Pixiv problem? (Was looking for シテヤンヨ tagged works)

    I’m not fluent enough with Japanese to ask on other places, I hope someone can help me.

    1. When you do the searching, ensure you are choosing the right mode. Pixiv can do search by exact(http://www.pixiv.net/search.php?word=%E3%82%B7%E3%83%86%E3%83%A4%E3%83%B3%E3%83%A8&s_mode=s_tag_full&order=date_d&p=35) or partial matching (http://www.pixiv.net/search.php?word=%E3%82%B7%E3%83%86%E3%83%A4%E3%83%B3%E3%83%A8&order=date_d&p=35). If you check the url, there is additional parameter(&s_mode=s_tag_full) for the exact match.

      Also check on you Pixiv Settings on the website, see if you disable the adult content or not (R-18/R-18G)

  17. Oops, I’ve got hundreds of files with “pixiv Premium is the best way to enjoy what pixiv has to offer!” in their names… any recommendation on how to fix this or should I just delete and redownload with the updated pixiv downloader?

    1. if you want to get the correct title, then you need to delete the image and clean the db first (manage db-> clean), then redownload the images.

      Only the filename got problem, the image is ok.

  18. Thank you sir for this awesome app.

    Btw, are you planning to make a GUI for this? It’s not necessary but i’m just curious.

    1. Sure, just install the required python and the library, and get the source code from GitHub (not the mediafire one).
      I know some people run it in mac/linux.
      Some changes might be required for in the config.ini (path separator).

      1. [hikaru@hikaru-x01 PixivUtil2]$ sudo python setup.py install
        Traceback (most recent call last):
        File “setup.py”, line 5, in
        import py2exe
        ImportError: No module named ‘py2exe’

        1. The setup.py is only for windows, that is the script to compile and bundle the scripts.

          You can run the script directly from PixivUtil2.py.

          1. oooh! thnks!, but amm what’s command for run in terminal for linux? sorry >.<!
            this?:
            [hikaru@hikaru-x01 PixivUtil2]$ sudo python PixivUtil2.py
            [sudo] password for hikaru:
            File "PixivUtil2.py", line 80
            print "Using Referer:", str(referer)
            ^
            SyntaxError: invalid syntax
            [hikaru@hikaru-x01 PixivUtil2]$ python PixivUtil2.py
            File "PixivUtil2.py", line 80
            print "Using Referer:", str(referer)
            ^
            SyntaxError: invalid syntax
            [hikaru@hikaru-x01 PixivUtil2]$ python PixivUtil2.py install
            File "PixivUtil2.py", line 80
            print "Using Referer:", str(referer)
            ^
            SyntaxError: invalid syntax
            [hikaru@hikaru-x01 PixivUtil2]$ ./PixivUtil2.py
            bash: ./PixivUtil2.py: Permiso denegado
            [hikaru@hikaru-x01 PixivUtil2]$ ./PixivUtil2.py
            File "./PixivUtil2.py", line 80
            print "Using Referer:", str(referer)
            ^
            SyntaxError: invalid syntax
            [hikaru@hikaru-x01 PixivUtil2]$

            amm help plss :'(

          2. Looks like you are using python 3.x. Please note PixivUtil only running using Python 2.7.x, check the readme for the required library.

            Also, I believe you don’t need to sudo everything. It doesn’t use su function, running under current user should be enough, as long you have read/write access to the folder.

Comments are closed.