pixiv downloader 20120831

Change Log:

  • Update error detection for new error messages.

Download link for pixiv downloader 20120831 updated 20120831, source code in GitHub. Donate link on your right πŸ™‚

I’m thinking to port the code to Python 3 because of better Unicode support, but not really familiar with it. Probably also need to rewrite some part of the current codes considering this was developed since June 2010 (2+years). Any thought?

41 thoughts on “pixiv downloader 20120831”

  1. Me again, this time, with a new problem. I think Pixiv may have have changed their log-on page or something. I’m getting the following message.

    PixivDownloader2 version 20120831
    https://nandaka.wordpress.com/tag/pixiv-downloader/
    Reading C:StoragePixivconfig.ini …
    done.
    Creating database… done.
    Only process member where day last updated >= 7
    Using Username: *edited out*
    Log in using form.
    Error at pixivLogin(): (, ControlN
    otFoundError(“no control matching name ‘pixiv_id'”,), )
    failed
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 1414, in main
    File “PixivUtil2.py”, line 292, in pixivLogin
    File “mechanize_form.pyc”, line 2780, in __setitem__
    File “mechanize_form.pyc”, line 3101, in find_control
    File “mechanize_form.pyc”, line 3185, in _find_control
    ControlNotFoundError: no control matching name ‘pixiv_id’
    press enter to exit.

    As you can see it get’s an error as soon as it loads. I’ve already tried to re-download the program, and run it as is (both with and without the username/password), but I keep getting the same error message. Do you have any idea what I can do to get around this, or is it just a recode issue, meaning all we can do is wait for the next release to fix the problem?

    1. I have similar issue, every time I try to login it just ends here:
      Error at pixivLogin(): (, ControlN
      otFoundError(“no control matching name ‘pixiv_id'”,), )

      Help vould be appreciated

  2. Okay, so I tried running the utility today and continued to get this error:

    2012-09-20 08:57:07,336 – PixivUtil20120806b – ERROR – Error at pixivLogin(): (, ControlNotFoundError(“no control matching name ‘pixiv_id'”,), )
    2012-09-20 08:57:07,339 – PixivUtil20120806b – ERROR – Unknown Error: no control matching name ‘pixiv_id’
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 1398, in main
    File “PixivUtil2.py”, line 286, in pixivLogin
    File “mechanize_form.pyc”, line 2780, in __setitem__
    File “mechanize_form.pyc”, line 3101, in find_control
    File “mechanize_form.pyc”, line 3185, in _find_control
    ControlNotFoundError: no control matching name ‘pixiv_id’

    Any way to fix this?

  3. Sorry for trouble you again… This time it says: VontrolNotFound Error: no control matching name’pixiv_id’… Please help me. Thanks>///<

  4. 2012-09-20 21:26:49,345 – PixivUtil20120831 – ERROR – Unknown Error: no control matching name ‘pixiv_id’
    Traceback (most recent call last):
    File “PixivUtil2.py”, line 1414, in main
    File “PixivUtil2.py”, line 292, in pixivLogin
    File “mechanize_form.pyc”, line 2780, in __setitem__
    File “mechanize_form.pyc”, line 3101, in find_control
    File “mechanize_form.pyc”, line 3185, in _find_control
    ControlNotFoundError: no control matching name ‘pixiv_id’

    i think pixiv got some changes

  5. I am asking for majpr design change, but I do not know how else to solve this problem.
    Usually I crawl pixiv daily with download members from the list. This is long process, because I got 1100 lines in said list.txt Usually about 50 or 60 of those artists got a new image, about 300 total per day.
    Only sane solution that I see is to separate page crawler from image downloader, so it would check for skippable artists while dowloading updates? Or could you recommend any other way to accelerate process and make it faster that 4 hours? I am unwilling to remove artists from memberlist.

  7. Thanks for your awesome program.
    Could you add the novel download function?
    Please consider this. Thanks.

  8. Hi, could you add the function of filtering based on number of bookmarks for list downloads as a command line option? Thanks.

  9. Hi! I currently updated from using 20120505 to this latest Pixiv Downloader and was wondering why the %works_date_only% changed from having dashes “-” to underscores “_”? For example, it used to be like “03-21-2012” but it is now “03_21_2012”. This causes doubles and wonder if I could change it back somehow.

    1. I think I forgot to replace ‘/’ from the date (07/22/2011). Previously, pixiv display the date using ‘-‘. I’ll update it in the next version.

  10. Just thought I’d say big thanks for this. I’ve been using it from time to time ever since 1st Jan 2012 (not that long ago but still). You are THE MAN for making this and updating it! Keep up the good work!

  11. nandaka, thanks again for your wonderful program, but can you please add a feature of informing a user of new images?

    Using “PixivUtil2 -s 4 list.txt” in .bat works great, but commandline window gets so much text, that it erases a lot of it. Once you have more than 10 or so artists in your list.txt it gets cumbersome to look if one of them got new stuff. Would be great if it would output all new filenames at the end of it’s job or save them into some text file.

    —

    Regarding Python 3.

    Strings are now great to work with, because of unicode support, no doubt.
    Some small api changes would have to be made. Like urllib2.urlopen() will become urllib.request.urlopen(), HTMLParser.HTMLParser becomes html.parser.HTMLParser, etc. No biggie, also HTMLParser now have argument to tell it not be strict with invalid htmls, so it will behave with them more like BeutifulSoup and such.

    One very sad thing about Python 3 at the moment is that there is no good tool to turn your scripts into .exe. I presume you used py2exe on your script, but it doesn’t support Python 3. There is cx_freeze, but I didn’t had a great time with it, it just didn’t work for serious scripts, something about UnicodeError when i tried generate exe with it.
    So it’s pretty bad for making distributions and that can be a huge deciding factor for migration.

    —

    Regarding GUI.

    There is a very good Python library PySide, which is Qt for Python.
    It has fantastic official documentation at http://www.pyside.org/docs/pyside/index.html
    There is also very good and easy to follow beginner tutorials at http://zetcode.com/gui/pysidetutorial/
    Another great thing about it is PySide.QtWebKit module. With it you can browse url pages, execute javascript calls, etc. Basically you can make your own browser with it if you want.
    One thing that’s not friendly to non-GUI developers is that you will have to work inside QApplication loop, instead of being completely free and this can make dramatic impact on code structure.
    Another thing is that UI will be locked until your function return, so it will be better to use threading/multiprocessing module to deal with it.

    Some screens of my little app using it (it’s very crappy, but it will do):
    http://s018.radikal.ru/i518/1209/f6/2a8379a4c732.jpg
    http://s52.radikal.ru/i138/1209/b5/5552bc4894c1.jpg
    http://s55.radikal.ru/i150/1209/e3/e21bcfb96ac5.jpg

    —

    Sorry for this boring wall of text x)

    Please add “New files:” feature in next release!

      1. Silly me, thanks nandaka! Now it’s perfect, i was just confused by setting’s name.

        Some things I felt the need to be addressed:

        wxPython. Nice library, but I wouldn’t put it being wrapper as an advantage, just a feature for specific purpose. In fact one might chose PySide, because it is full-blown UI, but also much more than that.
        Depends on what you want. WxPython is used by xadownloader, app that is very similar to what I’m doing currently.

        Python installation is ordinary and straight-forward thing for us, developers, but for end-users it can be inconvenient to install Python distribution and then creating bat or shortcut or executing py files. Not a big deal, sure, but generally undesirable.

        Qt has a strange licensing history indeed, but PySide still has LGPL license from what I know.

        Simple GUI could be made just for editing config.ini as it’s own small app, then pixiv downloader itself can be kept as is. Just a thought.

    2. I’d suggest using wxPython instead of Qt; it’s the wxWidgets bindings for Python, and the main advantage of wxWidgets is that it’s a wrapper, not a full-blown GUI toolkit. wxWidgets applications running in Windows retain Windows UI elements from GDI etc.

      As for Python 3 conversion, that’s a pretty welcome development and I would honestly just tell people to install Py3k and get over it. Because it never hurts to have Python on your side.

      Just my 2Β’.

      I’m going to clone the source and poke around; maybe I’ll even end up with a working 3k port, ww.

      1. I’ll check about it. Also, if I’m not wrong, Qt is not really free as it own by Nokia…
        Most of my use case is only for downloading (run and forget), so don’t hope too much for a GUI πŸ˜€

        I’m also using Mechanize for its Browser class, but unfortunately it doesn’t support Python 3. Probably need to create a Helper class for replacing it with urllib.openurl.

        Anyway, if you want to poke around, check the PixivModel for the parsing logic. I’m using BeautifulSoup as the parser.

  12. There is one more new error message you may have missed(program told me to inform you);

    “Member error: This user account has been suspended.”

  13. Help~~ whenever i try to download images by a Tags (#3 option in downloader), It only downloads few images but not ever images. There are like over 200 pages belong in one tag, but downloader only downloads only few images. Any solution ?

    1. This is a problem I had from the last version, and it seems to have carried over to the new one. For some odd reason you can not download more than one tag at a time. The tag separator is as folows:

      tagsseparator = ,

      So when I enter a tag search it is usually “tag1, tag2”, which obviously doesn’t work, and unfortunately neither does putting it as “tag1,tag2”. I then tried to change the tag limit option, but that didn’t work either.

      tagslimit = -5

      The default is “1”, so I bumped it up to five, and it didn’t work. I also tried to get rid of the negative sign to make the number a positive, and that didn’t work either. I’m out of ideas on what to do to get this to work right. If you have any advice I would appreciate it, and also, do you have a small tutorial section for what all the options in config.ini does? I’m really trying to be self-sufficient, but I’m reaching the limits of my ability here.

      Also, just as a suggestion, but have you ever heard of NeoDownloader? Your program is great and all, but a front-end would be nice too. This way changing settings would be more user friendly (drop down boxes), plus maybe the ability for multiple connections. I wouldn’t even mind buying it if you put it out as a paid version. Just an idea if you ever want to get around to it.

      1. tagsseparator and tagslimit options is for make filename, not for searching, see readme.txt for more details.
        If you want to search multiple tag, use space as the separator.
        I don’t know how to create GUI in python, and mostly I ran the application as script in background, so I don’t really use the GUI.
        If someone can create GUI, maybe I can help with the backend πŸ™‚

      1. The tags I’m using are δΊŒγƒƒε²©γƒžγƒŸγ‚Ύγ‚¦ 東方. As for the wildcard options, sorry, I’m not using it. The thing is I don’t really know what it’s used for, so I’ve always left it blank. Anyway the config.ini looks like this:

        [Settings]
        proxyaddress =
        useproxy = False
        useragent = Mozilla/5.0 (X11; U; Unix i686; en-US; rv:1.9.0.1) Gecko/2008071615 Fedora/3.0.1-1.fc9

        Firefox/3.0.1
        debughttp = False
        userobots = False
        filenameformat = %artist% (%member_id%)%urlFilename% – %title%
        filenamemangaformat = %artist% (%member_id%)%urlFilename% – %title%
        timeout = 60
        uselist = False
        processfromdb = True
        overwrite = False
        tagsseparator = ,
        daylastupdated = 7
        rootdirectory = C:PixivTouhouMamizou Futatsuiwa
        retry = 3
        retrywait = 5
        createdownloadlists = False
        downloadlistdirectory = .
        irfanviewpath = C:Program FilesIrfanView
        startirfanview = False
        startirfanslide = False
        alwayscheckfilesize = False
        checkupdatedlimit = 0
        downloadavatar = True
        createmangadir = False
        usetagsasdir = False
        useblacklisttags = False
        usesuppresstags = False
        tagslimit = -1

        [Pixiv]
        numberofpage = 0

        The authentication was intentionally left out because…well…personal information. I don’t know if this is enough, but I hope it helps.

        1. Here is the result if you use wildcard = Tags – Partial match (112 page)
          http://www.pixiv.net/search.php?s_mode=s_tag&word=%E4%BA%8C%E3%83%83%E5%B2%A9%E3%83%9E%E3%83%9F%E3%82%BE%E3%82%A6%20%E6%9D%B1%E6%96%B9

          No wildcard = Tags – Exact match (0 result)
          http://www.pixiv.net/tags.php?tag=%E4%BA%8C%E3%83%83%E5%B2%A9%E3%83%9E%E3%83%9F%E3%82%BE%E3%82%A6%20%E6%9D%B1%E6%96%B9

          By Title/Caption (2 page only)
          http://www.pixiv.net/search.php?s_mode=s_tc&word=%E4%BA%8C%E3%83%83%E5%B2%A9%E3%83%9E%E3%83%9F%E3%82%BE%E3%82%A6%20%E6%9D%B1%E6%96%B9

          Try to use wildcard

      2. So basically if it asks for wildcards just hit yes, and enter then? I’ll try that out. If that’s all it is then it’s only a single character entry that has to change to get the script to run right. The only problem is having to make 20+ changes for all the keys to work right. It’s simple..,just annoying.

        Um…not to go about posting things, but would you care if I gave a Mediafire link to a small script for the Logitech G110 keyboard? It’s a keyboard that’s meant for gaming, but I’ve gotten more use out of the macro keys for basic browsing, and definitely out of running your program. Usually if it’s a routine task that is going to be performed ad-nausea, I just shorten that stuff down by binding it to a single macro key, and save myself time/effort.

        So, for your program, the keys being pressed are all the same, except for the start/end pages (one key will download pages 1-4, another 5-9, etc.). So basically it’s all just hit a macro key, minimize the window, click on a different instance of Pixiv Batch Downloader, hit the other key, rinse and repeat.

        Anyway I’m getting sidetracked…since the file is nothing more than a configuration (.lgp) file, it’s useless for anyone that’s not using this series of keyboard, and since I don’t want to solicit merchandise, I was wanting to get permission before posting a link to the config file. I figured since I lack the ability to program then this would be the least I can do in making things run easier for others.

      3. Due to the file being a configuration file I don’t think there really is a way to copy/paste it that would make it usable. As a result here is the file…all 48.72 KB of it.

        http://www.mediafire.com/?6csndacc89v8yex

        Also, just for fun, here is a Pixiv tutorial I made a while back. You can redistribute it if you want.

        http://www.mediafire.com/?26ra8pixb22x5s3

        Also, as a plus, I gave an example of the Logitech keyboard at the end, and the use of macros for it. Yes, yes, I know, it’s product endorsement, but then again, this keyboard makes using your program a lot easier than it normally would be. So, until someone with the programming skill required would get around to making that front end for you, this is the next best thing.

Comments are closed.