Change Log:
- Update parser to the latest pixiv layout.
- Update last page detection (20130222b).
- Update sanitizeFilename: remove new line marker ‘rn’ (20130222c).
Download link for pixiv downloader 20130222c, source code in GitHub.
If you are using the source code version and encounter ValueError: invalid literal for int() with base 10: '<something>', please modify _html.py from mechanize library with this pastebin.
Three questions:
How does one modify _html.py? Is it the same thing as _html.pyc?
I appended %R-18% to the filenameformat. It looks like this: %urlFilename% – %title% %R-18%
However, it leaves a space for non-R-18 images.
How do I retain a space for R-18 and leave it out for non-R-18?
Lastly, can I exclude the thumbnails for manga downloads?
Thanks! :3
No, .pyc is the compiled .py script.
Search for the file. If you are using Windows, usually it is on C:Python27Libmechanize.
Search for
def unescape_charref(data, encoding):and replace the code from the pastebin.Should be fixed in the latest version.
Thanks.
Thumbnail issue is fixed, but the filename one still persists.
The filenames return as:
123 – abc R-18.jpg (for R-18, as it should be)
456 – def .jpg (for non-R-18, giving an unwanted space)
Alternatively I put an underscore in place of the space, but ideally I’d like to eliminate any other character (including spaces) that appears after %title% _if_ the image is non-R-18.
Ok, I found out the issue, I’ll fix it in the next release.
Not bug report or such, but something I’d like to know if it’s possible, there are some options that are still unclear to me.
Let’s say, I’m without connection for a while (may happen to me in the future) and when I’m back online, I want to download all the pics my bookmarked artists published in the meantime; option 8 would be useless because the new pics page keeps “only” the latest 2000 (yes, I’ve bookmarked a lot of artists :p)
Using the “e” option I can generate a list of all the member_ids of my bookmarks but the main question is, can I set the program to not download pictures older than X days before the system date?
If that is not possible, I’d like to suggest it as a future feature. I speak as a “profane” of programming but, noticing how the downloader also reads the works’ dates (in fact, you can set it so that the filenames also include the dates), the implementation _may_ be possible.
Technically, it can be done. You better put the request in GitHub issue tracker, so I don’t forget it 😀
Done, I hope I did it correctly 🙂
I can’t seem to get he manga mode to work properly. It saves each page in a different folder, instead of having the entire manga in one folder.
How do I fix this?
Can you upload your config.ini to pastebin and give the sample manga id?
I tried it with various configurations so I’ll just post my most recent one. And for the sample manga id, I didn’t download a specific manga but rather batch downloaded several tags.
http://pastebin.com/c2JhVaZ8
Weird. Tried in my pc, each manga will create a directory containing the pages as configured.
D:TempPixiv鳴滝しん - 誰か助けて～！ 34072502_big
- _p0.jpg
- _p1.jpg
D:TempPixivトイレ籠 - みっくみくな反応171 33928250_big
- _p0.jpg
- _p1.jpg
I tried the configuration I posted here again (after trying several other things) and for some mysterious reason it suddenly works fine even though the same configuration didn’t work yesterday.
Strange, but if everything’s working then it’s fine.
Thanks for the great app. I’m pretty much using it exclusively now for my art needs.
It’s slow but I rather save art from pixiv than danbooru or the likes.
This is a godsend. I was using an earlier version a while ago, which I noticed stopped working recently. Now I’ve moved onto this one everything works fine. Thanks!
Oh and I’ve just encountered this error while downloading:
Image #579
Processing Image Id: 32999235
Traceback (most recent call last):
File “PixivUtil2.py”, line 568, in processImage
File “PixivModel.pyc”, line 213, in __init__
File “PixivModel.pyc”, line 44, in __init__
File “PixivModel.pyc”, line 61, in ParseInfo
AttributeError: ‘NoneType’ object has no attribute ‘find’
Error at processImage(): (, AttributeError(“‘N
oneType’ object has no attribute ‘find'”,), )
Dumping html to: Error Medium Page for image 32999235.html
Error at processNewIllustFromBookmark(): (, At
tributeError(“‘NoneType’ object has no attribute ‘find'”,), )
Traceback (most recent call last):
File “PixivUtil2.py”, line 1482, in main
File “PixivUtil2.py”, line 1280, in menuDownloadNewIllustFromBookmark
File “PixivUtil2.py”, line 974, in processNewIllustFromBookmark
File “PixivUtil2.py”, line 568, in processImage
File “PixivModel.pyc”, line 213, in __init__
File “PixivModel.pyc”, line 44, in __init__
File “PixivModel.pyc”, line 61, in ParseInfo
AttributeError: ‘NoneType’ object has no attribute ‘find’
press enter to exit.
can you retry again? I’ve tried to download the individual image (32999235) and it working. Looks like you are logged out when downloading the image (check the dump file => Error Medium Page for image 32999235.html)
Yep seems like I’ve logged out while downloading. Was able to finish downloading when retried.
Thanks for the help.
Hey thanks for the update. I’ve got two questions:
– It this problem caused by me or from the server? I’ve got this error for couple of images while downloading new new images from bookmark
Image #267
Processing Image Id: 33502819
Title: イケこーりん
Tags : だが代金は払わない, どんぐり, なにこれイケメン, イケメン, 東方, 森近霖之
助
Mode : big
Image URL : http://i2.pixiv.net/img18/img/kuronekone_ro/33502819.jpg
Filename : C:UsersEricDesktoppixivutil20130222c一日一枚（4ヶ月目）ネーロ (
335493)33502819.jpg
Using Referer: http://www.pixiv.net/member_illust.php?mode=big&illust_id=3350281
9
[downloadImage()] HTTP Error 504: Gateway Time-out (http://i2.pixiv.net/img18/im
g/kuronekone_ro/33502819.jpg)
1 2 3 4
Using Referer: http://www.pixiv.net/member_illust.php?mode=big&illust_id=3350281
9
[downloadImage()] HTTP Error 504: Gateway Time-out (http://i2.pixiv.net/img18/im
g/kuronekone_ro/33502819.jpg)
1 2 3 4
Using Referer: http://www.pixiv.net/member_illust.php?mode=big&illust_id=3350281
9
[downloadImage()] HTTP Error 504: Gateway Time-out (http://i2.pixiv.net/img18/im
g/kuronekone_ro/33502819.jpg)
1 2 3 4
Using Referer: http://www.pixiv.net/member_illust.php?mode=big&illust_id=3350281
9
[downloadImage()] HTTP Error 504: Gateway Time-out (http://i2.pixiv.net/img18/im
g/kuronekone_ro/33502819.jpg)
Giving up url: http://i2.pixiv.net/img18/img/kuronekone_ro/33502819.jpg
– Is there any ways to speed up the download? as it takes a fair amount of time to download in pages (i.e when downloading
For example, it takes around approximately 8~9 minutes to download 50 images (some includes manga) – i.e. 5~12 second for each picture with the size between 200KB~1MB.
I don’t know if this is normal or not but if possible is there any ways to improve the speed?
Thanks,
504: usually network error, either from your side or their side.
No way to speed up the download speed, except if you want to code it yourself.
[*re-send*
Sorry, i this gets posted 2-3x; my reply wasn’t showing up above so i decided to make a new post since i wasn’t sure if my comment got posted correctly. I am the same kiumi in the post, please just ctrl+f to find ..OTL]
My version is newly download and extracted at my desktop so i think nothing should be missing? =3
i have also not edited anything in config.ini yet
http://i444.photobucket.com/albums/qq170/arunade/1_zps03ba5ddf.png
http://i444.photobucket.com/albums/qq170/arunade/2_zpsd0ac447c.png
Thanks again for helping!
Are you running from batch file? Can you upload your batch file to pastebin?
If not, then just enter the start page (e.g.:1) and later the end page (e.g.: 0 for all pages)
Hey^^
I seem to have problem getting the program to run.
2013-02-28 14:09:10,342 – PixivUtil20130222c – INFO – ###############################################################
2013-02-28 14:09:10,354 – PixivUtil20130222c – INFO – Starting…
2013-02-28 14:09:10,357 – PixivUtil20130222c – ERROR – Error at loadConfig()
Traceback (most recent call last):
File “PixivConfig.pyc”, line 71, in loadConfig
File “PixivHelper.pyc”, line 223, in OpenTextFile
File “codecs.pyc”, line 881, in open
IOError: [Errno 2] No such file or directory: u’C:\Users\User\Desktop\pixivutil20130222c\config.ini’
2013-02-28 14:09:10,447 – PixivUtil20130222c – INFO – Only process member where day last updated >= 7
2013-02-28 14:09:15,661 – PixivUtil20130222c – INFO – Log in using form.
2013-02-28 14:09:36,693 – PixivUtil20130222c – INFO – Logged in
2013-02-28 14:09:52,885 – PixivUtil20130222c – INFO – Member id mode.
2013-02-28 14:17:10,473 – PixivUtil20130222c – INFO –
I want to download using member_id , but whenever i get to the point where i enter the # for the id it doesn’t do anything.
Any suggestions or am i doing something wrong? ;3;)//
can you give me the screen shot? the IOError is because there is no config.ini in that folder, and this should have any effect.
[ gosh XD i’m really having a bit of a difficulty replying to posts here. Sorry for any inconvenience ]
in answer to your last question: i don’t think i’m running from batch file.
Entering the start page & end page worked for my problem though! Thank you so much!
Hi,
I have a ‘little’ bug since the latest update (I don’t know why), i have a error like #13 (aguoloaguolo), but not exactly the same. (http://pastebin.com/X5EbSBrb)
Setting of Pivix is on English (tried to switch in Japanese on Pixiv / regional language), but same problem as above (paste)
Info:
OS : Windows 7 SP1 (Home Prenium) (*Fresh Install and fully update*)
Pixiv : English or Japanese
Regional Language : English (Bug) or Japanese (99% of the time for play VNs) but bug too.
Looks like the returned page still in French (
u'end_span a xe9txe9 approuvxe9e.n{{#if more}}'==>end_span a été approuvée..{{#if more}}
Double check the language setting in pixiv.net and try to remove the cookie from config.ini and relogin?
works great for me so far, thanks a lot for the update ^_^
Hi, I’ve been getting this error every time I’ve been using 8: http://pastebin.com/6ac8tTg3
I always remove the %title% bit from the filenameformat and manga sections so I’m not sure if that has something to do with it now.
That said, thank you so much for constantly updating this handy program.
Somehow for this member name have newline “/r/n” in it. I’ll update it later on.
Thanks so much!
Not sure what is the cause, I guess this is the explanation: http://stackoverflow.com/questions/1522125/how-to-fix-such-clientform-bug
EDIT: should be reply for http://nandaka.wordpress.com/2013/02/22/pixiv-downloader-20130222/#comment-3167
Hello,
First, thank you for this great piece of program i’ve been already using for quite some time. I trully appreciate the regular updates.
for the 1st time in 2 years, i’m meeting this error:
Traceback (most recent call last):
File “PixivUtil2.py”, line 414, in processMember
File “BeautifulSoup.pyc”, line 1519, in __init__
File “BeautifulSoup.pyc”, line 1144, in __init__
File “BeautifulSoup.pyc”, line 1186, in _feed
File “sgmllib.pyc”, line 104, in feed
File “sgmllib.pyc”, line 143, in goahead
File “sgmllib.pyc”, line 320, in parse_endtag
File “sgmllib.pyc”, line 358, in finish_endtag
UnicodeEncodeError: ‘ascii’ codec can’t encode character u’xe9′ in position 11:
ordinal not in range(128)
Error at processing Artist Info: (, Unicod
eEncodeError(‘ascii’, u’end_span a xe9txe9 approuvxe9e.n{{#if more}}’, 11, 1
2, ‘ordinal not in range(128)’), )
1 2 3 4
Any idea ? I’ve tried changing regional setting or using something like applocale, but same results.
can you give me the member id? That error is came from the library.
2363
But i met the same error even updating my list with other IDs
It is working in my pc using the 20130222c.
Are you running from source code or exe? If from script, try to update the library. If not, try to set up you computer regional setting to japanese.
In pixiv website, what is the language you are using? Only English and Japanese are supported.
That was the solution. Thank you Nandaka.
My language setting inside my pixiv profile was set on French. Had to set it to english and the problem disappeared.
As for the regional settings on Windows (Windows 7), I’m still on french and it has never been a problem.
I am getting issues when trying to update from member ID 3104565 (R-18 warning) who has worked previously, and also this specific picture 33725282 by a different artist. Occurs with current Git version, I can post the log if necessary.
post the log to pastebin
Sure OK.
http://pastebin.com/MkJ8yQrg
Are you using the source code version? mechanize have a bug on _html.py file, I’ve modify the file in http://pastebin.com/5bT5HFkb
I’m using the Github version of PixivUtil2 on Debian Testing. Applying that change did fix it, but I’m curious why that’s necessary. Mechanize hasn’t been updated for at least a year and has never caused issues with PixivUtil2 before that.
Good to have it working again 🙂
I notice that now the download progress for each pic is almost never visibie during the download, only when finished, it’s not vital but would be nice if it could be fixed. Also, it seems a bit slower to me because it must elaborate longer (due to the necessary changes to adapt to the new layout), or is it just a false impression?
Hm. Well, this happened when trying to download a specific user. The update from bookmarks seems to work just fine, though:
http://pastebin.com/4xk3bMtz
Working from the latest version.
I assume you are running from the source code? There is a bug on the mechanize library on the _html.py file. You need to modify the file to handle this.
while using 8. Download new illust from bookmark
from page 1 to page 10, it will only process page 1,then said Limit or last page reached.
Seems it only process 1 page…
Please fix. Thank you!
can you upload the log file?
2013-02-23 01:15:46,845 – PixivUtil20130222 – INFO – ###############################################################
2013-02-23 01:15:46,845 – PixivUtil20130222 – INFO – Starting…
2013-02-23 01:15:46,855 – PixivUtil20130222 – INFO – Using proxy: 127.0.0.1:8087
2013-02-23 01:15:46,861 – PixivUtil20130222 – INFO – Only process member where day last updated >= 7
2013-02-23 01:15:46,862 – PixivUtil20130222 – INFO – Using Username: rogerlhc
2013-02-23 01:15:46,867 – PixivUtil20130222 – INFO – logging in with saved cookie
2013-02-23 01:15:46,869 – PixivUtil20130222 – INFO – Trying to log with saved cookie
2013-02-23 01:15:49,401 – PixivUtil20130222 – INFO – Logged in using cookie
2013-02-23 01:15:51,676 – PixivUtil20130222 – INFO – New Illust from Bookmark mode.
=====
PixivDownloader2 version 20130222
https://nandaka.wordpress.com/tag/pixiv-downloader/
Reading F:pixivwatch downloaderconfig.ini …
done.
Using proxy: *****
Creating database… done.
Only process member where day last updated >= 7
Using Username:******
logging in with saved cookie
Trying to log with saved cookie
done.
PixivDownloader2 version 20130222
https://nandaka.wordpress.com/tag/pixiv-downloader/
1. Download by member_id
2. Download by image_id
3. Download by tags
4. Download from list
5. Download from online user bookmark
6. Download from online image bookmark
7. Download from tags list
8. Download new illust from bookmark
9. Download by Title/Caption
10. Download by Tag and Member Id
11. Download Member Bookmark
————————
d. Manage database
e. Export online bookmark
x. Exit
Input: 8
Start Page (default=1):
End Page (default=0, 0 for no limit): 20
Processing New Illust from bookmark
Page #1
Image #1
Processing Image Id: 33771273
Already downloaded: 33771273
Image #2
Processing Image Id: 33770482
Already downloaded: 33770482
Image #3
Processing Image Id: 33770026
Already downloaded: 33770026
Image #4
Processing Image Id: 33769368
Already downloaded: 33769368
Image #5
Processing Image Id: 33766281
Already downloaded: 33766281
Image #6
Processing Image Id: 33765565
Already downloaded: 33765565
Image #7
Processing Image Id: 33764693
Already downloaded: 33764693
Image #8
Processing Image Id: 33764129
Already downloaded: 33764129
Image #9
Processing Image Id: 33761936
Already downloaded: 33761936
Image #10
Processing Image Id: 33761591
Already downloaded: 33761591
Image #11
Processing Image Id: 33761345
Already downloaded: 33761345
Image #12
Processing Image Id: 33756458
Already downloaded: 33756458
Image #13
Processing Image Id: 33755349
Already downloaded: 33755349
Image #14
Processing Image Id: 33754619
Already downloaded: 33754619
Image #15
Processing Image Id: 33753584
Already downloaded: 33753584
Image #16
Processing Image Id: 33753143
Already downloaded: 33753143
Image #17
Processing Image Id: 33751108
Already downloaded: 33751108
Image #18
Processing Image Id: 33749731
Already downloaded: 33749731
Image #19
Processing Image Id: 33749298
Already downloaded: 33749298
Image #20
Processing Image Id: 33747989
Already downloaded: 33747989
Limit or last page reached.
Done.
PixivDownloader2 version 20130222
https://nandaka.wordpress.com/tag/pixiv-downloader/
1. Download by member_id
2. Download by image_id
3. Download by tags
4. Download from list
5. Download from online user bookmark
6. Download from online image bookmark
7. Download from tags list
8. Download new illust from bookmark
9. Download by Title/Caption
10. Download by Tag and Member Id
11. Download Member Bookmark
————————
d. Manage database
e. Export online bookmark
x. Exit
Input:
try the latest update (20130222b)