Change Log:
- Add Ctrl+C handler for download by member id and download by tags to allow user to skip to next image or abort the whole process.
- Add option to keep signed in, set
keepsignedin = 1
inconfig.ini
on[Authentication]
section. - Add login error message parser.
- Add download time and speed.
- Backup config.ini when saving to config.ini.bak.
- Fix command line option for download by member id and tags to allow single tag search.
Download link for pixiv downloader 20130330, source code in GitHub.
>> I don’t know if img-inf have something to do (temporary?). Maybe I’ll add a check, but worst case if all the images are using img-inf, then no member token can be parsed.
Did you ran some test?
————
As a feature request, I’d like to be able to download only page xx from my bookmark (or from xx to yy). An option in the choice 5 would do I think (start at x end at y, like the dialog for download by member_id).
>> Did you ran some test?
What test :D, I do checking after someone ask about it.
>> I’d like to be able to download only page xx from my bookmark
Need to add more image in my bookmark then, I’ll see about it.
Is there a way to do writeimageinfo for already downloaded images? I only turned it on recently but I’d like the info for past downloads too, without having to re-download everything again.
Thank you!
Nope.
I see. If i give a manual path for the memberid in the list.txt, will it take priority on that parsing maybe?
After a quick test, looks like the priority still lies in the filenameformat. I’ll sleep for now and look into it tommorow. 🙂
Update: this artist have also the problem (only page 1): http://www.pixiv.net/member.php?id=199669
They both do not have avatars, but that’s a hunch.
What is the problem? Can you give more detailed information (screen shot/log file)
It looks like my full explication doesn’t go through wordpress, here’s a pastebin of my reply: http://pastebin.com/cS4uDFdY
This is because of the way they structure the images. E.g.: in 199669, the avatar is empty, so I check the first image in the list (illust_id=34846113) to get the token based on the thumbnail url. Somehow, for this images, the url format, http://i2.pixiv.net/img-inf/img/2013/04/07/03/08/21/34846113_s.jpg, is not the usual one. This cause the token to be parsed to 21
If you compare to the 2nd image, the thumb url is http://i2.pixiv.net/img14/img/blue16/28407891_s.jpg, which parse to blue16.
I don’t know if img-inf have something to do (temporary?). Maybe I’ll add a check, but worst case if all the images are using img-inf, then no member token can be parsed.
Interesting enough, after I manually added my cookie in the config file it works again. I wonder what causes this, the system is running Ubuntu so the chance is that one of the libs got updated causing to break some things. (Bleeding edge, eh.)
I’m assuming it is going to break again once the cookie expires. Which is a shame as I tend to run PixivUtil every 15 minutes or so with a cronjob.
set keepsignedin = 1 in your config.ini and when you login, tick the remember me box and copy the cookie
Alright, fair enough. I suppose that solves the problem for now. Thank you very much for your support, this program has been a big help for almost a year now. I love it.
Since a couple of days ago PixivUtil doesn’t download any images anymore and just returns me this when trying to download a image http://pastebin.com/wLd3hJWF
I’m running the latest source from GitHub on Linux. It used to work perfect until a few days ago.
Can you upload the dump file: Error Medium Page for image 35196562.html? I tried on my pc using option 1 (member_id:3816549) and option 2 (image_id:35196562), but it is working properly.
http://pastebin.com/MtMf6uq0
If I try Option 1 here, it gives me errors as well.
Try to open the page on the browser, it says you are not logged in.
Can you upload the full log file (delete it first, try to replicate the error, and then upload the log file)? See where is the return url (
Logging in, return url: http://www.pixiv.net/mypage.php
)I can’t reply to your last message for some reason, interestingly enough the return URL seems to be http://www.pixiv.net/index.php?return_to=%2Fmypage.php
Here’s the full log file http://pastebin.com/w0z1iKYf
I think I need to update the post-login checking. Anyway, for your case, can you check your login information? Try to delete the cookies and use ssl.
If I delete the cookie the same thing happens, however. It doesn’t write any new cookie information into the config file. SSL was off, turning it on didn’t make any difference. If I turn r18mode on it says it can’t find any more images, even after making a new database. Perhaps it’s because I’m running it on Linux, though it used to work fine for months.
try to change your password to simpler one (e.g.: 1234) and turn on the debughttp, see whether the login information is passed or not.
Worst case, copy you cookie from browser to config.ini.
I made a new Pixiv account and kept the password to 12346, here’s the output from the console (Username replaced with USRNAME for safety measures) http://pastebin.com/XXqYGktw
I’d like to try using the cookie from my browser, however. Could you provide me the name of the cookie I should copy the value from?
PHPSESSID
, I’ll check the log, somehow it redirect you back to index.php. I don’t know if the p_ab_id (A/B testing?) cookie have any effect.There also a difference when the data being sent to login.php, somehow the PHPSESSID is not sent as one of the cookie. Maybe you need to update the library?
Great to see the config.ini backed up now!
However, when I upgraded my files, it did not detect ‘keepSignedIn’, so it backed up my config file. Then it said my login was expired, and created a new cookie value (which was exactly the same value as the manual backup I made prior to upgrading any files), and immediately overwrote the backup with a new backup. You can see it overwritten here: http://i36.tinypic.com/21c72io.jpg
My filename and filenamemanga formats were intact, however! Thank you for that! I can also see a ctrl+c handler being very useful in my future! Major thanks!
Maybe I can append a timestamp on the backed up config?
Perhaps, or maybe just one backup per instance run? Also, I wonder why it thought the cookie was different. It doesn’t seem to think so anymore, however.
The validation is by checking the returned page when requesting http://www.pixiv.net/mypage.php. If it is return different page, the application will assume the cookie is expired.
Thanks very much! This Program is Wonderful. Next version is add a r-18g mode in config.ini if you can.
then this tool is more great 🙂
R-18 should include R-18G, except u want only to get the R-18G? Technically, you can use download by member id and tags (option 10).
good works. thx
bad english , sorry!
Download from image bookmark
this function can add download my private bookmark_tag optional ? not my bookmark all tag
You means, using option 6 and download from your own bookmarked image (member_id = 0), you want to use your own tags rather than then original image tags for the filename? This is not possible as the information is not on the image page itself.
Would it be possible to download images from your own bookmarks. However could they downloaded and sorted into folders with the tag name, instead of the member’s name and ID? This would be incredibly useful for myself and other people who love to organise.
>> sorted into folders with the tag name
You can auto create folder based on tag name (see filenameformat in readme.txt), the problem is: it will create a folder with all the tags inside the image post.
Is it not possible to create a folder only containing the tags you want automatically, except you use the usesuppresstags = true and provide the tags in the suppress_tags.txt file (blacklist).
There is a filenameformat for %searchTags%, but this only for Download by Tags method, as you provide which tag to use as the folder name.
Sorry I have to clarify myself, when I said tags I meant tags that I had created myself to describe the image, not the artist’s tags.
Never tried it before, so I don’t have the sample page to test. Can you save the sample image page and upload it to Mediafire?
Checking the description, I think it will add your user id in the tags added by you, is this correct?
I’ll see what I can do.
Not sure what you wanted me to save, so I saved a few webpages.
I tried using the option of “Download from image bookmark (http://www.pixiv.net/bookmark.php), including
private/hidden bookmarks.”
For the filenameformat (in config.txt) I tried for both the folder name and filename using:
%bookmark% – did nothing (left a blank space where writing should be)
%tags% – Tags saved by artist and not myself were incorporated into the folder/file name. (i’ve included download result of this).
Also, Is it possible to seperate bookmarks by page number? Because I notice that once 1st page is finished it goes on to try download images on the 2nd page (“importing page 2”).
Mediafire link: http://www.mediafire.com/?63ez5aa27d58fka
>> I tried using the option of “Download from image bookmark (http://www.pixiv.net/bookmark.php), including
private/hidden bookmarks.
%bookmark% format currently only work from option 11. Use that one, as this is the updated version. I’ll remove the ability to download another user image bookmark from option 6.
>> Also, Is it possible to seperate bookmarks by page number
I don’t think this is useful as the image might be shifted to another page if user add/remove the bookmarks.
If you mean is limiting the download page range, then it is implemented on option 11.
>> tags that I had created myself to describe the image, not the artist’s tags.
Not possible, as it is doesn’t show up in the image page.
Thanks for creating such a great tool. Is multi-threading a possible feature to add in the future?
no plan for multi-threading.
Hello there, thanks for this tool. I have a specific feature request: Original datestamp on the downloaded file as found on Pixiv. Using a download manager like DownThemAll!, for example, will give the file and the original datestamp (I don’t know if it’s the datestamp from when a file was uploaded to Pixiv, or the original datestamp of the file as found on the uploader’s local copy). This is useful to have.
Actually, there is one other thing I noticed, and it suggests another (hopefully quite simple) feature request. I downloaded a file and noticed it was broken (only partially downloaded), and I had to temporarily rename the sqlite database file in order to convince the program it had never downloaded the file and therefore attempt a fresh download. This worked, but would it be possible to include a menu entry which forces redownload (using download from a specific image tag should be all that is needed for the vast majority of cases; otherwise the database should probably be temporarily renamed or even scrapped anyway)?
>> Original Timestamp
Dunno how to get it, why it is useful?
>> forces redownload
Check
alwaysCheckFileSize
for config.ini in readme.txt, or use the DB Manager (press d on main menu) and select delete image by image id, or you can delete the file and do DB clean up from DB Manager.>> Original Timestamp
Useful because, well, it shows the original date of the file. Useful for historical purposes, i.e. to track an artist’s development, to compare different versions of files (i.e. the Pixiv version against a version posted on a blog) which might have minor differences, and so on.
RE redownloading, I’ll take note of the DB Manager, thanks. It sounds like there are some other useful features of the DB Manager I’ve been neglecting, too 😀
ok, you’ll be angry for this, i just used it a few times and this error appears:
2013-04-02 23:32:49,260 – PixivUtil20130330 – INFO – EXIT
2013-04-02 23:32:49,260 – PixivUtil20130330 – INFO – ###############################################################
2013-04-02 23:32:52,671 – PixivUtil20130330 – INFO – ###############################################################
2013-04-02 23:32:52,671 – PixivUtil20130330 – INFO – Starting…
2013-04-02 23:32:52,677 – PixivUtil20130330 – INFO – Only process member where day last updated >= 7
2013-04-02 23:32:59,181 – PixivUtil20130330 – INFO – Log in using form.
2013-04-02 23:33:09,174 – PixivUtil20130330 – INFO – Logging in, return url: http://www.pixiv.net/mypage.php
2013-04-02 23:33:09,176 – PixivUtil20130330 – INFO – Logged in
2013-04-02 23:33:10,684 – PixivUtil20130330 – INFO – Member id mode.
2013-04-02 23:33:27,221 – PixivUtil20130330 – INFO – Processing Member Id: 93360
2013-04-02 23:33:27,221 – PixivUtil20130330 – INFO – End Page: 2
2013-04-02 23:33:27,378 – PixivUtil20130330 – INFO – Member Url: http://www.pixiv.net/member_illust.php?id=93360&p=1
2013-04-02 23:33:42,253 – PixivUtil20130330 – ERROR – Error at processing Artist Info: (, AttributeError(“‘NoneType’ object has no attribute ‘ul'”,), )
2013-04-02 23:33:42,253 – PixivUtil20130330 – ERROR – Error at processing Artist Info: 93360
Traceback (most recent call last):
File “PixivUtil2.py”, line 420, in processMember
File “PixivModel.pyc”, line 42, in __init__
File “PixivModel.pyc”, line 101, in ParseImages
AttributeError: ‘NoneType’ object has no attribute ‘ul’
2013-04-02 23:33:46,259 – PixivUtil20130330 – INFO – Member Url: http://www.pixiv.net/member_illust.php?id=93360&p=1
Any idea?
upload the dump+log file to mediafire.
no idea whats the dump or log is xD, just uploaded the whole folder xDDD http://www.mediafire.com/?33eey5p3v02c6tx
no dump file 🙁 inside. Anyway, can you go to Pixiv website and set to english language, then edit config.ini and remove the cookie information, then try again? I’ll update the application to generate dump file if encounter any error on the weekend.
Maaaaaaan, you saved my lifeeee, i had like 10 artists waiting for a gallery download xDDDDD, Thank you!
So i have a problem every start. (・ヘ・?)
” 2013-03-31 19:08:58,663 – PixivUtil20130330 – ERROR – Error at processMember(): (, ValueError(“time data ‘2.3.2013 20:13’ does not match format ‘%Y-%m-%d %H:%M'”,), ) Member Id: 369356
Traceback (most recent call last):
File “PixivUtil2.py”, line 492, in processMember
File “PixivUtil2.py”, line 591, in processImage
File “PixivModel.pyc”, line 226, in __init__
File “PixivModel.pyc”, line 313, in ParseWorksData
File “_strptime.pyc”, line 325, in _strptime
ValueError: time data ‘2.3.2013 20:13’ does not match format ‘%Y-%m-%d %H:%M’ “
Change your pixiv language to English or Japanese from pixiv website.
error all the same
delete your cookie entry in config.ini, then retry after you set the language to english from pixiv website. If still have error, please upload the page to mediafire.
Yup, this fixed the logging in problem I had before. Thanks.
Will keep you updated if I hit any more snags.
Whoops, sorry. I’m an idiot with these things, and I need your help once more.
I’m trying to download images from my own bookmarks (images I bookmarked myself), and I don’t think I have any private images. I got this error:
http://prntscr.com/yhf0g
Am I using the wrong command here?
most likely they change the styling for bookmarked page, I look about it.
Weird, it is working on my pc (both private or public, or even another member_id bookmarks).
Can you upload the bookmark page from your browser (html only, http://www.pixiv.net/bookmark.php) and upload it to mediafire?
Sorry I took so long.
Oh, and I hope you could deal with the…content within the page.
http://www.mediafire.com/view/?xace4snkgx6mh65
Weird, I tried to parse the page and it is working.
It is returning
[20609344, 27923863, 17550202, 33490758, 25221094, 23215877, 28324516, 1336118, 32586414, 31524392, 30255682, 29938244, 27321420, 26303661, 25162485, 22076151, 11648522, 32230541, 14155233, 32221699]
, which is the list of image id in your bookmark.Are you using exe version or source code version? If you are using source code version, update your library for mechanize and BeautifulSoup. No changes on the parse method from v20130330 version.
I’m using the exe.
Alright, so what I did was try to simply download ONE image, using one of the image IDs you provided.
http://prntscr.com/yyt1n
And this happened.
Hmm.
I tried disabling all the userscripts and junk, and it still doesn’t work…
This is depressing…
delete your config.ini and try again?
When I tried for my bookmarks, same error as my first post.
When I tried for single image, (used 31524392), I got same error as second screenshot.
However, it’s worth noting that this .html file appeared in the same directory, after the single image error.
http://www.mediafire.com/view/?y233cyyqlt9ybfw
Also, does userscripts even affect anything? I doubt it does.
If you open the page, it says you need to log in. Most likely, the log in is expired or failed.
upload pixivutil.log to mediafire, or try to find this line:
INFO - Logging in, return url:
. It should not return login.php.Sorry I’m late. I had class.
Found the line.
2013-04-03 12:48:10,563 – PixivUtil20130330 – INFO – Logging in, return url: http://www.pixiv.net/index.php?return_to=%2Fmypage.php
Doesn’t seem to return to login.php
That’s weird, if you open the html file, it says you need to log-in to Pixiv. Is R-18 is enabled in your account settings in Pixiv Website?
Definitely. I always have R-18 enabled.
http://prntscr.com/z0yq3