Change Log for 20141004c:
- Update parser.
- Update download logic for manga page (20141004b)
- Update parser for the new 2 page manga mode (20141004c)
Download link for pixiv downloader 20141004c, source code in GitHub.
Tested working properly on 2014-10-03 for download by member id (normal, manga, and ugoira), unless Pixiv change their css again…
You guys better donate so I can buy more figures, got 2 on preorder for this month 😛
Donation Link on the side bar ==> 😀
UPDATED!!!!!
So the program will be downloading fine for a long while, then for a reason that I’m not aware of something breaks.
Like this: http://puu.sh/c0mbS.png
It seems it goes through the rest of the artists on the list, but it fails for all of them.
Also the error pages don’t seem to work. When I click them I get this: http://puu.sh/c0iWv.png
Clicking “here” just ends up on that same page.
Looks like you got network issue for this case, for whatever reason, the page is redirected to 1.1.1.1.
By right, the dump page should contains pixiv data.
Check if your firewall/av?
Hi!
There are some rare problem on certain pages. I download by tags and have no problem except on those three pics : 46336451, 46297438 and 46360787
Here are the messages :
Image #81
Image Id: 46336451
Bookmark Count: 7
Processing Image Id: 46336451
Espace insuffisant pour traiter cette commande.
Title: ????
Tags : R-18G, ??????? ????? ????? ??? ??? ???? ???
Date : 2014-10-04 03:01:00
Mode : manga
Traceback (most recent call last):
File “PixivUtil2.py”, line 684, in process_image
File “PixivModel.pyc”, line 371, in ParseImages
File “PixivModel.pyc”, line 406, in CheckMangaType
File “PixivModel.pyc”, line 428, in ParseMangaImagesNew
AttributeError: ‘NoneType’ object has no attribute ‘findAll’
Error at process_image(): (, AttributeError(“‘
NoneType’ object has no attribute ‘findAll'”,),
)
Cannot medium dump page for image_id: 46336451
Error at process_tags(): (, UnboundLocalErr
or(“local variable ‘parse_medium_page’ referenced before assignment”,), )
Dumping html to: Error page for search tags %E8%AA%BF%E6%95%99+OR+%E8%86%A3%E5%8
6%85%E6%94%BE%E5%B0%BF+OR+%E8%82%89%E4%BE%BF%E5%99%A8+OR+%E8%82%89%E5%A5%B4%E9%9
A%B7+OR+%E8%99%9A%E3%82%8D%E7%9B%AE+OR+%E7%B2%BE%E6%B6%B2%E4%BE%BF%E6%89%80+OR+%
E9%A6%96%E8%BC%AA+OR+%E8%86%A3%E9%8F%A1+OR+%E4%B8%89%E7%A9%B4+OR+%E4%BA%8C%E7%A9
%B4+OR+%E8%BC%AA%E5%A7%A6+OR+%E6%BA%A2%E3%82%8C%E7%B2%BE%E6%B6%B2+OR+%E6%B1%9A%E
3%81%84%E3%81%8A%E3%81%A3%E3%81%95%E3%82%93+OR+%E3%83%93%E3%83%83%E3%83%81+OR+%E
5%B7%A8%E6%A0%B9+OR+%E7%97%B4%E6%BC%A2+OR+%E9%99%B5%E8%BE%B1+OR+%E3%81%9A%E3%82%
89%E3%81%97%E6%8C%BF%E5%85%A5+OR+%E3%83%AD%E3%83%AA%E3%83%93%E3%83%83%E3%83%81+O
R+%E7%AC%91%E9%A1%94%E5%BC%B7%E5%88%B6+OR+%E8%BA%AB%E4%BD%93%E3%81%AB%E8%90%BD%E
6%9B%B8%E3%81%8D+OR+%E6%AD%A3%E3%81%AE%E5%AD%97+OR+%E3%81%84%E3%81%98%E3%82%81+O
R+%E7%AC%91%E9%A1%94%E5%BC%B7%E5%88%B6+OR+%E5%A3%B2%E6%98%A5+OR+%E7%97%B4%E5%A5%
B3+OR+%E6%8F%B4%E5%8A%A9%E4%BA%A4%E9%9A%9B+OR+%E6%8F%B4%E4%BA%A4+OR+%E5%A3%B2%E6
%98%A5.html
Error at process_tags_list(): (, UnboundLoc
alError(“local variable ‘parse_medium_page’ referenced before assignment”,), )
Traceback (most recent call last):
File “PixivUtil2.py”, line 1813, in main
File “PixivUtil2.py”, line 1621, in main_loop
File “PixivUtil2.py”, line 1496, in menu_download_from_tags_list
File “PixivUtil2.py”, line 952, in process_tags_list
File “PixivUtil2.py”, line 897, in process_tags
File “PixivUtil2.py”, line 785, in process_image
UnboundLocalError: local variable ‘parse_medium_page’ referenced before assignme
nt
press enter to exit.
Image #236
Image Id: 46297438
Bookmark Count: 6
Processing Image Id: 46297438
Espace insuffisant pour traiter cette commande.
Title: ???????????????
Tags : R-18G, ?????? ??? SM, ??
Date : 2014-10-02 00:07:00
Mode : manga
Traceback (most recent call last):
File “PixivUtil2.py”, line 684, in process_image
File “PixivModel.pyc”, line 371, in ParseImages
File “PixivModel.pyc”, line 406, in CheckMangaType
File “PixivModel.pyc”, line 428, in ParseMangaImagesNew
AttributeError: ‘NoneType’ object has no attribute ‘findAll’
Error at process_image(): (, AttributeError(“‘
NoneType’ object has no attribute ‘findAll'”,),
)
Cannot medium dump page for image_id: 46297438
Error at process_tags(): (, UnboundLocalErr
or(“local variable ‘parse_medium_page’ referenced before assignment”,), )
Dumping html to: Error page for search tags %E3%83%9C%E3%83%B3%E3%83%87%E3%83%BC
%E3%82%B8+OR+%E5%8F%A3%E6%9E%B7+OR+%E3%82%AF%E3%83%AA%E8%B2%AC%E3%82%81+OR+%E3%8
3%90%E3%82%A4%E3%83%96+OR+%E7%B7%8A%E7%B8%9B+OR+%E7%B2%BE%E6%B6%B2+OR+%E3%83%90%
E3%82%A4%E3%83%96IN%E3%83%91%E3%83%B3%E3%83%84+OR+%E3%83%AD%E3%83%BC%E3%82%BF%E3
%83%BC+OR+%E8%87%AD%E3%81%84+OR+%E9%80%86%E3%83%AA%E3%83%A7%E3%83%8A+OR+%E3%83%9
2%E3%83%88%E3%82%A4%E3%83%8C+OR+%E3%83%A9%E3%83%86%E3%83%83%E3%82%AF%E3%82%B9+OR
+%E3%83%A9%E3%83%90%E3%83%BC+OR+%E3%83%9E%E3%82%B9%E3%82%AF%E3%82%AE%E3%83%A3%E3
%82%B0+OR+%E3%83%9D%E3%83%8B%E3%83%BC%E3%82%AC%E3%83%BC%E3%83%AB+OR+%E6%8B%B7%E5
%95%8F+OR+%E5%85%A8%E9%A0%AD%E3%83%9E%E3%82%B9%E3%82%AF+OR+%E6%8B%98%E6%9D%9F+OR
+%E5%8C%82%E3%81%84.html
Error at process_tags_list(): (, UnboundLoc
alError(“local variable ‘parse_medium_page’ referenced before assignment”,), )
Traceback (most recent call last):
File “PixivUtil2.py”, line 1813, in main
File “PixivUtil2.py”, line 1621, in main_loop
File “PixivUtil2.py”, line 1496, in menu_download_from_tags_list
File “PixivUtil2.py”, line 952, in process_tags_list
File “PixivUtil2.py”, line 897, in process_tags
File “PixivUtil2.py”, line 785, in process_image
UnboundLocalError: local variable ‘parse_medium_page’ referenced before assignme
nt
press enter to exit.
Image #27
Image Id: 46360787
Bookmark Count: 8
Processing Image Id: 46360787
Espace insuffisant pour traiter cette commande.
Title: ?????????
Tags : R-18, ?????? ????? ?
Date : 2014-10-05 09:12:00
Mode : manga
Traceback (most recent call last):
File “PixivUtil2.py”, line 684, in process_image
File “PixivModel.pyc”, line 371, in ParseImages
File “PixivModel.pyc”, line 406, in CheckMangaType
File “PixivModel.pyc”, line 428, in ParseMangaImagesNew
AttributeError: ‘NoneType’ object has no attribute ‘findAll’
Error at process_image(): (, AttributeError(“‘
NoneType’ object has no attribute ‘findAll'”,),
)
Cannot medium dump page for image_id: 46360787
Error at process_tags(): (, UnboundLocalErr
or(“local variable ‘parse_medium_page’ referenced before assignment”,), )
Dumping html to: Error page for search tags %E5%8C%85%E8%8C%8E+OR+%E3%81%B5%E3%8
1%9F%E3%81%AA%E3%82%8A+OR+%E7%94%B7%E3%81%AE%E5%A8%98+OR+%E5%BD%A2%E3%81%8F%E3%8
1%A3%E3%81%8D%E3%82%8A%E3%83%9A%E3%83%8B%E3%82%B9+OR+%E3%83%AC%E3%82%BA+OR+%E3%8
3%95%E3%82%BF%E3%83%8A%E3%83%AA.html
Error at process_tags_list(): (, UnboundLoc
alError(“local variable ‘parse_medium_page’ referenced before assignment”,), )
Traceback (most recent call last):
File “PixivUtil2.py”, line 1813, in main
File “PixivUtil2.py”, line 1621, in main_loop
File “PixivUtil2.py”, line 1496, in menu_download_from_tags_list
File “PixivUtil2.py”, line 952, in process_tags_list
File “PixivUtil2.py”, line 897, in process_tags
File “PixivUtil2.py”, line 785, in process_image
UnboundLocalError: local variable ‘parse_medium_page’ referenced before assignme
nt
press enter to exit.
Thanks in advance 😉
Yeah, still getting problems with the manga reader. When it encounters these types of manga, it tries a few times, then gives up on everything afterwards.
Processing Image Id: 32431170
Title: 幼馴染(♀)に何があった?(ビッチ化)
Tags : R-18, 漫画, 3Dカスタム少女, ビッチ, ピアス, タトゥー
Date : 2012-12-28 17:19:00
Mode : manga
Traceback (most recent call last):
File “PixivUtil2.py”, line 684, in process_image
File “PixivModel.pyc”, line 371, in ParseImages
File “PixivModel.pyc”, line 406, in CheckMangaType
File “PixivModel.pyc”, line 428, in ParseMangaImagesNew
AttributeError: ‘NoneType’ object has no attribute ‘findAll’
Error at process_image(): (, AttributeError(“‘
NoneType’ object has no attribute ‘findAll'”,),
)
Cannot medium dump page for image_id: 32431170
Stuff happened, trying again after 2 second ( 1 )
Traceback (most recent call last):
File “PixivUtil2.py”, line 506, in process_member
File “PixivUtil2.py”, line 785, in process_image
UnboundLocalError: local variable ‘parse_medium_page’ referenced before assignme
nt
….[immediately after attempt #5]
Giving up image_id: 32431170
PixivDownloader2 version 20141004c
https://nandaka.wordpress.com/tag/pixiv-downloader/
1. Download by member_id
It’s my first time using this. I get this error after “Login using form”
Error at pixiv_login(): (, <
httperror_seek_wrapper (mechanize._http.RobotExclusionError instance) at 0x28437
30 whose wrapped object = <closeable_response at 0x2876620 whose fp = >>, )
failed
Traceback (most recent call last):
File “PixivUtil2.py”, line 1805, in main
File “PixivUtil2.py”, line 260, in pixiv_login
File “mechanize_mechanize.pyc”, line 203, in open
File “mechanize_mechanize.pyc”, line 255, in _mech_open
httperror_seek_wrapper: HTTP Error 403: request disallowed by robots.txt
press enter to exit.
I’ve already put “userobots = False” in config.ini
Thats weird? Are you running in Windows? Check if the application pick the correct config. If you are using shortcut, make sure the working directory is the same with the application location.
Yes, I’m running on Windows, 7 to be precise. I just run PixivUtil2.exe from the extracted folder. I also tried Run as Admin.
I just noticed – this change might have happened some time ago – that the new default INI file now defaults to “overwrite = false” which I assume captures the same variable as the old “Overwrite Old File” variable. Defaulting to “false” is a much appreciated change.
I will keep a close eye on new downloads – I believe that about the time the Pixiv backend changed some single-image illustrations might have been uploaded with “_0.img” at the end. I think this is something Pixiv Downloader already handles well, however.
Wait, I was confused. I was thinking of “backupoldfile” which is unfortunately set to False by default (I’d rather prune by hand than have the software overwrite unique old versions of files). I also note that using robots is still on by default, but I’ve never had success using the program unless this is set to False. Are there users for which using robots works?
I purposely set Use Robots to True, so they read the readme.txt 😛
I can understand the reasoning that whatever default settings are selected is your business, and up to the user to set them, but this doesn’t answer my question. whinette confirms it works but why it does is mysterious (and the readme doesn’t shed light on why).
Robots.txt? This is set from the pixiv webserver to avoid robots (e.g. webcrawler, site ripper) to index/download the whole site. You can look the purpose of robots.txt on google.
Yeazh I use robots.txt and the program works nicely (until pixiv break the layout, haha).
Donated~ 🙂
Tried downloading new illustrations from bookmark and kept getting this error every single time:
“UnboundLocalError: local variable ‘parse_medium_page’ referenced before assignment”
http://pastebin.com/v7FfuZYc
Not sure what I should do about it. :
There is a problem on (old) manga mode:
It downloads only pair pages. I’ll put an issue on github.
Log :
Image ids: 30507937
Processing Image Id: 30507937
Title: 紅楼夢8新刊「共食い禁止令!」
fuck that crap, screenshot:
https://www.dropbox.com/s/0e83l5i3wtba2p3/Capture%20d%27%C3%A9cran%202014-10-03%2019.13.22.png?dl=0
This was already fixed in the git repo by the time nandaka made the comment at #3. Update your install.
Thanks, willdo !
My juron was about my inability to paste @ windows interpreting control character in my clipboard; not the bug by the way. 😉
You might have less frustration with a proper terminal emulator for cygwin (such as konsole), a virtual machine running linux, a raspberry pi accessed through putty, or barring all that, powershell. 😉
Hello,
Thanks for updating the Pixiv downloader. It must be frustrating keeping up with all the changes Pixiv have made in the last week.
It now seems to work fine for everything apart from older manga posts that were made before Pixiv changed. For these, the program will only download even numbered pages (see below). Many thanks.
Input: 2
Image ids: 26743783
Processing Image Id: 26743783
Title: ?????????????
Tags : ????? ?????? ??????? ???????? ????????? ?????????????? ??????? ???? ?????
??????????? ??????
Date : 2012-04-23 00:57:00
Mode : manga
Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
llust_id=26743783&page=0
Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
llust_id=26743783&page=1
Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
llust_id=26743783&page=2
Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
llust_id=26743783&page=3
Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
llust_id=26743783&page=4
Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
llust_id=26743783&page=5
Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
llust_id=26743783&page=6
Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
llust_id=26743783&page=7
Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
llust_id=26743783&page=8
Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
llust_id=26743783&page=9
Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
llust_id=26743783&page=10
Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
llust_id=26743783&page=11
Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
llust_id=26743783&page=12
Fetching big image page: http://www.pixiv.net/member_illust.php?mode=manga_big&i
llust_id=26743783&page=13
Page Count : 14
Image URL : http://i1.pixiv.net/img65/img/kamachi52/26743783_big_p0.png
Filename : ??????????????????????? (2544637)26743783_big_p0.png
Using Referer: http://www.pixiv.net/member_illust.php?mode=medium&illust_id=2674
3783
Start downloading… 275980 of 275980 Bytes Completed in 2.06s (130.83 Ki
B/s)
done.
Image URL : http://i1.pixiv.net/img65/img/kamachi52/26743783_big_p2.png
Filename : ??????????????????????? (2544637)26743783_big_p2.png
Using Referer: http://www.pixiv.net/member_illust.php?mode=medium&illust_id=2674
3783
Start downloading… 317646 of 317646 Bytes Completed in 2.25s (137.87 Ki
B/s)
done.
Image URL : http://i1.pixiv.net/img65/img/kamachi52/26743783_big_p4.png
Filename : ??????????????????????? (2544637)26743783_big_p4.png
Using Referer: http://www.pixiv.net/member_illust.php?mode=medium&illust_id=2674
3783
Start downloading… 253672 of 253672 Bytes Completed in 2.001s (123.80 K
iB/s)
done.
Image URL : http://i1.pixiv.net/img65/img/kamachi52/26743783_big_p6.png
Filename : ??????????????????????? (2544637)26743783_big_p6.png
Using Referer: http://www.pixiv.net/member_illust.php?mode=medium&illust_id=2674
3783
Start downloading… 142728 of 142728 Bytes Completed in 1.5s (92.92 KiB/
s)
done.
Image URL : http://i1.pixiv.net/img65/img/kamachi52/26743783_big_p8.png
Filename : ??????????????????????? (2544637)26743783_big_p8.png
Using Referer: http://www.pixiv.net/member_illust.php?mode=medium&illust_id=2674
3783
Start downloading… 211058 of 211058 Bytes Completed in 1.82s (113.25 Ki
B/s)
done.
Image URL : http://i1.pixiv.net/img65/img/kamachi52/26743783_big_p10.png
Filename : ??????????????????????? (2544637)26743783_big_p10.png
Using Referer: http://www.pixiv.net/member_illust.php?mode=medium&illust_id=2674
3783
Start downloading… 210756 of 210756 Bytes Completed in 1.76s (116.94 Ki
B/s)
done.
Image URL : http://i1.pixiv.net/img65/img/kamachi52/26743783_big_p12.png
Filename : ??????????????????????? (2544637)26743783_big_p12.png
Using Referer: http://www.pixiv.net/member_illust.php?mode=medium&illust_id=2674
3783
Start downloading… 151201 of 151201 Bytes Completed in 1.5s (98.44 KiB/
s)
done.
Actually, I find that there’s a new type of manga view (right to left) that trips up the program, like image ids 46322053, 46320252 and 46320003.
this one new.
Upvoted on paypal. 🙂
You do God’s work.
That’s true, nice find!