URI problems, need some love

Discussion:

Kevin Kornemann

2013-02-25 23:55:16 UTC

Got a form, hitting a button with Mechanize. Thus far have been going
through login steps quite nicely so I know my Mechanize basics are up to
snuff.

Then I get:

URI::InvalidURIError: bad URI(is not URI?):
CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz1:-1
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:176:in
`split'
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:211:in
`parse'
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:747:in
`parse'
from /Users/kevin/.rvm/gems/ruby-1.9.3-***@texas/gems/mechanize-2.5.1/lib/mechanize/http/agent.rb:621:in
`rescue in resolve'
from /Users/kevin/.rvm/gems/ruby-1.9.3-***@texas/gems/mechanize-2.5.1/lib/mechanize/http/agent.rb:618:in
`resolve'
from /Users/kevin/.rvm/gems/ruby-1.9.3-***@texas/gems/mechanize-2.5.1/lib/mechanize/http/agent.rb:214:in
`fetch'
from /Users/kevin/.rvm/gems/ruby-1.9.3-***@texas/gems/mechanize-2.5.1/lib/mechanize.rb:1229:in
`post_form'
from /Users/kevin/.rvm/gems/ruby-1.9.3-***@texas/gems/mechanize-2.5.1/lib/mechanize.rb:515:in
`submit'
from /Users/kevin/.rvm/gems/ruby-1.9.3-***@texas/gems/mechanize-2.5.1/lib/mechanize/form.rb:178:in
`submit'
from (irb):69
from /Users/kevin/.rvm/gems/ruby-1.9.3-***@texas/gems/railties-3.2.12/lib/rails/commands/console.rb:47:in
`start'
from /Users/kevin/.rvm/gems/ruby-1.9.3-***@texas/gems/railties-3.2.12/lib/rails/commands/console.rb:8:in
`start'
from /Users/kevin/.rvm/gems/ruby-1.9.3-***@texas/gems/railties-3.2.12/lib/rails/commands.rb:41:in
`<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'

From the googling I've done, seems to be related to an invalid character in

the re-direct, but really I have no idea...

Any way of ignoring these weird characters and continuing? Anyway of fixing
this?

Best,
Kevin

Godfrey Chan

2013-02-26 02:06:26 UTC

Permalink

Hi Kevin,

It looks like the "action" attribute of your form contains something along the lines of "CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz". The semicolon in here is not strictly a valid character in an URL. Try replacing it with %3B.

form = page.at( ...select the form with a selector... )

action = form.attr('action')

action = action.gsub(';','%3B').gsub('=','%3D')

form.attr('action', action)

# then submit the form as usual (You can't do form.submit because this form object we have here is a nokogiri node. You need to go through mechanize to submit the form.)

GodfreyÂ
â
Sent from Mailbox for iPhone

Post by Kevin Kornemann
Got a form, hitting a button with Mechanize. Thus far have been going
through login steps quite nicely so I know my Mechanize basics are up to
snuff.
CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz1:-1
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:176:in
`split'
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:211:in
`parse'
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:747:in
`parse'
`rescue in resolve'
`resolve'
`fetch'
`post_form'
`submit'
`submit'
from (irb):69
`start'
`start'
`<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'

From the googling I've done, seems to be related to an invalid character in

the re-direct, but really I have no idea...
Any way of ignoring these weird characters and continuing? Anyway of fixing
this?
Best,
Kevin

Kevin Kornemann

2013-02-26 03:29:28 UTC

Permalink

Hi Godfrey,

Thanks for your help. Really working hard to figure this out. I ran into a
road block that is my inexperience with nokogiri and mech.

When I try to grab the action (action = form.attr('action')), I get
undefined method... Here is what I'm doing exactly:

agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
page = agent.get('https://somesite.jsp')

form = agent.page.forms.first
action = form.attr('action')

and undefined method...
I tried seeing if I could grab the action various ways to no avail.

My other question is, assuming I do this successfuly, I would normally use
"form.submit" but you're saying this won't work? What should I try?

Thanks again for your help!!
Kevin

Post by Godfrey Chan
Hi Kevin,
It looks like the "action" attribute of your form contains something along
the lines of "CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz". The
semicolon in here is not strictly a valid character in an URL. Try
replacing it with %3B.
form = page.at( ...select the form with a selector... )
action = form.attr('action')
action = action.gsub(';','%3B').gsub('=','%3D')
form.attr('action', action)
# then submit the form as usual (You can't do form.submit because this
form object we have here is a nokogiri node. You need to go through
mechanize to submit the form.)
Godfrey

Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone

_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users

Kevin Kornemann

2013-02-26 03:35:44 UTC

Permalink

actually, this grabbed it:

action = form.action

and your gsub worked a charm.

Just need to get this to submit... Because you're right form.submit didn't
work... I tried what you gave me:

action = form.attr('action')

But undefined meth again on attr...

Post by Kevin Kornemann
Hi Godfrey,
Thanks for your help. Really working hard to figure this out. I ran into a
road block that is my inexperience with nokogiri and mech.
When I try to grab the action (action = form.attr('action')), I get
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
page = agent.get('https://somesite.jsp')
form = agent.page.forms.first
action = form.attr('action')
and undefined method...
I tried seeing if I could grab the action various ways to no avail.
My other question is, assuming I do this successfuly, I would normally use
"form.submit" but you're saying this won't work? What should I try?
Thanks again for your help!!
Kevin

Post by Godfrey Chan
Hi Kevin,
It looks like the "action" attribute of your form contains something
along the lines of "CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz".
The semicolon in here is not strictly a valid character in an URL. Try
replacing it with %3B.
form = page.at( ...select the form with a selector... )
action = form.attr('action')
action = action.gsub(';','%3B').gsub('=','%3D')
form.attr('action', action)
# then submit the form as usual (You can't do form.submit because this
form object we have here is a nokogiri node. You need to go through
mechanize to submit the form.)
Godfrey

Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone

_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users

Kevin Kornemann

2013-02-26 03:38:02 UTC

Permalink

sorry meant I tried:

form.attr('action', action)

to no avail...?

Post by Kevin Kornemann
action = form.action
and your gsub worked a charm.
Just need to get this to submit... Because you're right form.submit didn't
action = form.attr('action')
But undefined meth again on attr...

Post by Kevin Kornemann
Hi Godfrey,
Thanks for your help. Really working hard to figure this out. I ran into
a road block that is my inexperience with nokogiri and mech.
When I try to grab the action (action = form.attr('action')), I get
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
page = agent.get('https://somesite.jsp')
form = agent.page.forms.first
action = form.attr('action')
and undefined method...
I tried seeing if I could grab the action various ways to no avail.
My other question is, assuming I do this successfuly, I would normally
use "form.submit" but you're saying this won't work? What should I try?
Thanks again for your help!!
Kevin

Post by Godfrey Chan
Hi Kevin,
It looks like the "action" attribute of your form contains something
along the lines of "CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz".
The semicolon in here is not strictly a valid character in an URL. Try
replacing it with %3B.
form = page.at( ...select the form with a selector... )
action = form.attr('action')
action = action.gsub(';','%3B').gsub('=','%3D')
form.attr('action', action)
# then submit the form as usual (You can't do form.submit because this
form object we have here is a nokogiri node. You need to go through
mechanize to submit the form.)
Godfrey

Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone

_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users

Godfrey Chan

2013-02-26 03:50:45 UTC

Permalink

Hi Kevin,

IÂ don't currently have my computer with me to check the codeÂ - so sorryÂ I can't be more helpful. I might have made some of these methods up, so apply a good dose of common sense if things didn't work as expected :)

I believe this should do the trick:

form = agent.current_page.forms.first

form.action = form.action.gsub(...).gsub(...)

form.submit

Forget what I said earlier about the nokogiri stuff and form.submit won't work etc. I wasn't aware that you can change the form action from mechanize.

If form.submit somehowÂ didn't work, please post a trace.

Godfrey

â
Sent from Mailbox for iPhone

Post by Godfrey Chan
form.attr('action', action)
to no avail...?

Post by Kevin Kornemann
Hi Godfrey,
Thanks for your help. Really working hard to figure this out. I ran into
a road block that is my inexperience with nokogiri and mech.
When I try to grab the action (action = form.attr('action')), I get
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
page = agent.get('https://somesite.jsp')
form = agent.page.forms.first
action = form.attr('action')
and undefined method...
I tried seeing if I could grab the action various ways to no avail.
My other question is, assuming I do this successfuly, I would normally
use "form.submit" but you're saying this won't work? What should I try?
Thanks again for your help!!
Kevin

Post by Godfrey Chan
Hi Kevin,
It looks like the "action" attribute of your form contains something
along the lines of "CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz".
The semicolon in here is not strictly a valid character in an URL. Try
replacing it with %3B.
form = page.at( ...select the form with a selector... )
action = form.attr('action')
action = action.gsub(';','%3B').gsub('=','%3D')
form.attr('action', action)
# then submit the form as usual (You can't do form.submit because this
form object we have here is a nokogiri node. You need to go through
mechanize to submit the form.)
Godfrey
â
Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone

_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users

Kevin Kornemann

2013-02-26 04:06:44 UTC

Permalink

Hi Godfrey,

1) Thanks again for your help, I really appreciate it.
2) yeah, so I can grab the action, gsub it nicely (I see the return). But
when I 'form.submit' it errors out on the original URI, as if the new
version wasn't changed at all...

I'll keep trying and let you know how I do. ENough for tonight.
thanks,
Kevin

Post by Godfrey Chan
Hi Kevin,
I don't currently have my computer with me to check the code - so sorry I
can't be more helpful. I might have made some of these methods up, so apply
a good dose of common sense if things didn't work as expected :)
form = agent.current_page.forms.first
form.action = form.action.gsub(...).gsub(...)
form.submit
Forget what I said earlier about the nokogiri stuff and form.submit won't
work etc. I wasn't aware that you can change the form action from mechanize.
If form.submit somehow didn't work, please post a trace.
Godfrey

Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone

Post by Godfrey Chan
form.attr('action', action)
to no avail...?

Post by Kevin Kornemann
action = form.action
and your gsub worked a charm.
Just need to get this to submit... Because you're right form.submit
action = form.attr('action')
But undefined meth again on attr...
On Mon, Feb 25, 2013 at 10:29 PM, Kevin Kornemann <

Post by Kevin Kornemann
Hi Godfrey,
Thanks for your help. Really working hard to figure this out. I ran
into a road block that is my inexperience with nokogiri and mech.
When I try to grab the action (action = form.attr('action')), I get
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
page = agent.get('https://somesite.jsp')
form = agent.page.forms.first
action = form.attr('action')
and undefined method...
I tried seeing if I could grab the action various ways to no avail.
My other question is, assuming I do this successfuly, I would normally
use "form.submit" but you're saying this won't work? What should I try?
Thanks again for your help!!
Kevin

Post by Godfrey Chan
Hi Kevin,
It looks like the "action" attribute of your form contains something
along the lines of "CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz".
The semicolon in here is not strictly a valid character in an URL. Try
replacing it with %3B.
form = page.at( ...select the form with a selector... )
action = form.attr('action')
action = action.gsub(';','%3B').gsub('=','%3D')
form.attr('action', action)
# then submit the form as usual (You can't do form.submit because this
form object we have here is a nokogiri node. You need to go through
mechanize to submit the form.)
Godfrey

Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone
On Mon, Feb 25, 2013 at 4:01 PM, Kevin Kornemann <

_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users

Godfrey Chan

2013-02-26 06:58:10 UTC

Permalink

Hi Kevin,

I finally have access to my computer now, and it turns out the semi colon
in the action is not the corrupt - the colon ("...:-1") was the issue.
(Didn't notice that before, I thought it was a line number or some sort of
error code. That's a *really* weird URI.)

Post by Kevin Kornemann

require 'mechanize'
agent = Mechanize.new
page = agent.get 'http://formtest.dev/form_test.html'
form = page.forms.first
form.submit

URI::InvalidURIError: bad URI(is not URI?):
CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz1:-1

Post by Kevin Kornemann

form.action = CGI.escape(form.action)
form.submit # this should work now

If your server fail to URL decode that for some odd reason and refuse to
process that, you can try:

form.action = './' + form.action

Hope it helps.

Godfrey

Post by Kevin Kornemann
Hi Godfrey,
1) Thanks again for your help, I really appreciate it.
2) yeah, so I can grab the action, gsub it nicely (I see the return). But
when I 'form.submit' it errors out on the original URI, as if the new
version wasn't changed at all...
I'll keep trying and let you know how I do. ENough for tonight.
thanks,
Kevin

Hi Kevin,
I don't currently have my computer with me to check the code - so sorry I
can't be more helpful. I might have made some of these methods up, so apply
a good dose of common sense if things didn't work as expected :)
form = agent.current_page.forms.first
form.action = form.action.gsub(...).gsub(...)
form.submit
Forget what I said earlier about the nokogiri stuff and form.submit won't
work etc. I wasn't aware that you can change the form action from mechanize.
If form.submit somehow didn't work, please post a trace.
Godfrey
â
Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone

Post by Godfrey Chan
form.attr('action', action)
to no avail...?
On Mon, Feb 25, 2013 at 10:35 PM, Kevin Kornemann <

Post by Kevin Kornemann
Hi Godfrey,
Thanks for your help. Really working hard to figure this out. I ran
into a road block that is my inexperience with nokogiri and mech.
When I try to grab the action (action = form.attr('action')), I get
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
page = agent.get('https://somesite.jsp')
form = agent.page.forms.first
action = form.attr('action')
and undefined method...
I tried seeing if I could grab the action various ways to no avail.
My other question is, assuming I do this successfuly, I would normally
use "form.submit" but you're saying this won't work? What should I try?
Thanks again for your help!!
Kevin

Post by Godfrey Chan
Hi Kevin,
It looks like the "action" attribute of your form contains something
along the lines of "CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz".
The semicolon in here is not strictly a valid character in an URL. Try
replacing it with %3B.
form = page.at( ...select the form with a selector... )
action = form.attr('action')
action = action.gsub(';','%3B').gsub('=','%3D')
form.attr('action', action)
# then submit the form as usual (You can't do form.submit because
this form object we have here is a nokogiri node. You need to go through
mechanize to submit the form.)
Godfrey
â
Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone
On Mon, Feb 25, 2013 at 4:01 PM, Kevin Kornemann <

Post by Kevin Kornemann
Got a form, hitting a button with Mechanize. Thus far have been
going through login steps quite nicely so I know my Mechanize basics are up
to snuff.
CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz1:-1
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:176:in
`split'
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:211:in
`parse'
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:747:in
`parse'
`rescue in resolve'
`resolve'
`fetch'
`post_form'
`submit'
`submit'
from (irb):69
`start'
`start'
`<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'
From the googling I've done, seems to be related to an invalid
character in the re-direct, but really I have no idea...
Any way of ignoring these weird characters and continuing? Anyway of
fixing this?
Best,
Kevin

_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users

Kevin Kornemann

2013-02-26 14:02:09 UTC

Permalink

So, that got past the URI error, but the system just bounced me to a
"session expired" page... Actually last night I did you gsub trick and
submitted the new string with a manual agent.post but the same thing
happened, it just bounced me to a session expired page.

This is the last page (and form) of many pages/forms before it that all
have similar action URI strings (with the same weird chars) that don't
raise an mech exceptions. Which leads me to believe this is something about
the agent not looking enough like a browser that this site doesn't like,
and this last page is looking at headers or something and bouncing requests
from non browsers... But I'm no expert this is just my hunch at this point.

Is there a more robust method of setting up mech to look more like a
browser??? besides the usual agent.user_agent_alias = 'Windows IE 6'

Headers?

Thanks again,
Kevin

Post by Godfrey Chan
Hi Kevin,
I finally have access to my computer now, and it turns out the semi colon
in the action is not the corrupt - the colon ("...:-1") was the issue.
(Didn't notice that before, I thought it was a line number or some sort of
error code. That's a *really* weird URI.)

Post by Kevin Kornemann

require 'mechanize'
agent = Mechanize.new
page = agent.get 'http://formtest.dev/form_test.html'
form = page.forms.first
form.submit

CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz1:-1

Post by Kevin Kornemann

form.action = CGI.escape(form.action)
form.submit # this should work now

If your server fail to URL decode that for some odd reason and refuse to
form.action = './' + form.action
Hope it helps.
Godfrey

Hi Kevin,
I don't currently have my computer with me to check the code - so
sorry I can't be more helpful. I might have made some of these methods up,
so apply a good dose of common sense if things didn't work as expected :)
form = agent.current_page.forms.first
form.action = form.action.gsub(...).gsub(...)
form.submit
Forget what I said earlier about the nokogiri stuff and form.submit
won't work etc. I wasn't aware that you can change the form action from
mechanize.
If form.submit somehow didn't work, please post a trace.
Godfrey

Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone

Post by Godfrey Chan
form.attr('action', action)
to no avail...?
On Mon, Feb 25, 2013 at 10:35 PM, Kevin Kornemann <

Post by Kevin Kornemann
Hi Godfrey,
Thanks for your help. Really working hard to figure this out. I ran
into a road block that is my inexperience with nokogiri and mech.
When I try to grab the action (action = form.attr('action')), I get
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
page = agent.get('https://somesite.jsp')
form = agent.page.forms.first
action = form.attr('action')
and undefined method...
I tried seeing if I could grab the action various ways to no avail.
My other question is, assuming I do this successfuly, I would
normally use "form.submit" but you're saying this won't work? What should I
try?
Thanks again for your help!!
Kevin

Post by Godfrey Chan
Hi Kevin,
It looks like the "action" attribute of your form contains something
along the lines of "CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz".
The semicolon in here is not strictly a valid character in an URL. Try
replacing it with %3B.
form = page.at( ...select the form with a selector... )
action = form.attr('action')
action = action.gsub(';','%3B').gsub('=','%3D')
form.attr('action', action)
# then submit the form as usual (You can't do form.submit because
this form object we have here is a nokogiri node. You need to go through
mechanize to submit the form.)
Godfrey

Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone
On Mon, Feb 25, 2013 at 4:01 PM, Kevin Kornemann <

Post by Kevin Kornemann
Got a form, hitting a button with Mechanize. Thus far have been
going through login steps quite nicely so I know my Mechanize basics are up
to snuff.
CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz1:-1
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:176:in
`split'
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:211:in
`parse'
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:747:in
`parse'
`rescue in resolve'
`resolve'
`fetch'
`post_form'
`submit'
`submit'
from (irb):69
`start'
`start'
`<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'
From the googling I've done, seems to be related to an invalid
character in the re-direct, but really I have no idea...
Any way of ignoring these weird characters and continuing? Anyway
of fixing this?
Best,
Kevin

_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users

Matt White

2013-02-26 15:06:57 UTC

Permalink

You might try something other than IE6. See http://rubydoc.info/gems/mechanize/Mechanize, scroll down just a bit to see all the options.

So, that got past the URI error, but the system just bounced me to a "session expired" page... Actually last night I did you gsub trick and submitted the new string with a manual agent.post but the same thing happened, it just bounced me to a session expired page.
This is the last page (and form) of many pages/forms before it that all have similar action URI strings (with the same weird chars) that don't raise an mech exceptions. Which leads me to believe this is something about the agent not looking enough like a browser that this site doesn't like, and this last page is looking at headers or something and bouncing requests from non browsers... But I'm no expert this is just my hunch at this point.
Is there a more robust method of setting up mech to look more like a browser??? besides the usual agent.user_agent_alias = 'Windows IE 6'
Headers?
Thanks again,
Kevin

Post by Godfrey Chan
Hi Kevin,
I finally have access to my computer now, and it turns out the semi colon in the action is not the corrupt - the colon ("...:-1") was the issue. (Didn't notice that before, I thought it was a line number or some sort of error code. That's a *really* weird URI.)

Post by Kevin Kornemann

require 'mechanize'
agent = Mechanize.new
page = agent.get 'http://formtest.dev/form_test.html'
form = page.forms.first
form.submit

URI::InvalidURIError: bad URI(is not URI?): CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz1:-1

Post by Kevin Kornemann

form.action = CGI.escape(form.action)
form.submit # this should work now

form.action = './' + form.action
Hope it helps.
Godfrey

Post by Kevin Kornemann
Hi Godfrey,
1) Thanks again for your help, I really appreciate it.
2) yeah, so I can grab the action, gsub it nicely (I see the return). But when I 'form.submit' it errors out on the original URI, as if the new version wasn't changed at all...
I'll keep trying and let you know how I do. ENough for tonight.
thanks,
Kevin

Hi Kevin,
I don't currently have my computer with me to check the code - so sorry I can't be more helpful. I might have made some of these methods up, so apply a good dose of common sense if things didn't work as expected :)
form = agent.current_page.forms.first
form.action = form.action.gsub(...).gsub(...)
form.submit
Forget what I said earlier about the nokogiri stuff and form.submit won't work etc. I wasn't aware that you can change the form action from mechanize.
If form.submit somehow didn't work, please post a trace.
Godfrey
â
Sent from Mailbox for iPhone

Post by Godfrey Chan
form.attr('action', action)
to no avail...?

Post by Kevin Kornemann
action = form.action
and your gsub worked a charm.
action = form.attr('action')
But undefined meth again on attr...

Post by Kevin Kornemann
Hi Godfrey,
Thanks for your help. Really working hard to figure this out. I ran into a road block that is my inexperience with nokogiri and mech.
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
page = agent.get('https://somesite.jsp')
form = agent.page.forms.first
action = form.attr('action')
and undefined method...
I tried seeing if I could grab the action various ways to no avail.
My other question is, assuming I do this successfuly, I would normally use "form.submit" but you're saying this won't work? What should I try?
Thanks again for your help!!
Kevin

Post by Godfrey Chan
Hi Kevin,
It looks like the "action" attribute of your form contains something along the lines of "CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz". The semicolon in here is not strictly a valid character in an URL. Try replacing it with %3B.
form = page.at( ...select the form with a selector... )
action = form.attr('action')
action = action.gsub(';','%3B').gsub('=','%3D')
form.attr('action', action)
# then submit the form as usual (You can't do form.submit because this form object we have here is a nokogiri node. You need to go through mechanize to submit the form.)
Godfrey
â
Sent from Mailbox for iPhone

Got a form, hitting a button with Mechanize. Thus far have been going through login steps quite nicely so I know my Mechanize basics are up to snuff.
URI::InvalidURIError: bad URI(is not URI?): CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz1:-1
from /Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:176:in `split'
from /Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:211:in `parse'
from /Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:747:in `parse'
from (irb):69
from script/rails:6:in `require'
from script/rails:6:in `<main>'
From the googling I've done, seems to be related to an invalid character in the re-direct, but really I have no idea...
Any way of ignoring these weird characters and continuing? Anyway of fixing this?
Best,
Kevin

_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users

Kevin Kornemann

2013-02-26 15:49:07 UTC

Permalink

Yeah I started with, "Mac Safari"

It's an old site, so I thought try the oldest crappiest browser I could
use. But I suppose trying them all will only be slightly futile.

Now I'm wondering if it's a lost cookie thing. I understand Mech tracks
these automatically but I see from googling people still also manually
store them in a "cookie_jar"

pain...

Post by Matt White
You might try something other than IE6. See
http://rubydoc.info/gems/mechanize/Mechanize, scroll down just a bit to
see all the options.
So, that got past the URI error, but the system just bounced me to a
"session expired" page... Actually last night I did you gsub trick and
submitted the new string with a manual agent.post but the same thing
happened, it just bounced me to a session expired page.
This is the last page (and form) of many pages/forms before it that all
have similar action URI strings (with the same weird chars) that don't
raise an mech exceptions. Which leads me to believe this is something about
the agent not looking enough like a browser that this site doesn't like,
and this last page is looking at headers or something and bouncing requests
from non browsers... But I'm no expert this is just my hunch at this point.
Is there a more robust method of setting up mech to look more like a
browser??? besides the usual agent.user_agent_alias = 'Windows IE 6'
Headers?
Thanks again,
Kevin

Post by Kevin Kornemann

require 'mechanize'
agent = Mechanize.new
page = agent.get 'http://formtest.dev/form_test.html'
form = page.forms.first
form.submit

CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz1:-1

Post by Kevin Kornemann

form.action = CGI.escape(form.action)
form.submit # this should work now

If your server fail to URL decode that for some odd reason and refuse to
form.action = './' + form.action
Hope it helps.
Godfrey

Post by Kevin Kornemann
Hi Godfrey,
1) Thanks again for your help, I really appreciate it.
2) yeah, so I can grab the action, gsub it nicely (I see the return).
But when I 'form.submit' it errors out on the original URI, as if the new
version wasn't changed at all...
I'll keep trying and let you know how I do. ENough for tonight.
thanks,
Kevin

Hi Kevin,
I don't currently have my computer with me to check the code - so
sorry I can't be more helpful. I might have made some of these methods up,
so apply a good dose of common sense if things didn't work as expected :)
form = agent.current_page.forms.first
form.action = form.action.gsub(...).gsub(...)
form.submit
Forget what I said earlier about the nokogiri stuff and form.submit
won't work etc. I wasn't aware that you can change the form action from
mechanize.
If form.submit somehow didn't work, please post a trace.
Godfrey

Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone
On Mon, Feb 25, 2013 at 7:38 PM, Kevin Kornemann <

Post by Godfrey Chan
form.attr('action', action)
to no avail...?
On Mon, Feb 25, 2013 at 10:35 PM, Kevin Kornemann <

Post by Kevin Kornemann
Hi Godfrey,
Thanks for your help. Really working hard to figure this out. I ran
into a road block that is my inexperience with nokogiri and mech.
When I try to grab the action (action = form.attr('action')), I get
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
page = agent.get('https://somesite.jsp')
form = agent.page.forms.first
action = form.attr('action')
and undefined method...
I tried seeing if I could grab the action various ways to no avail.
My other question is, assuming I do this successfuly, I would
normally use "form.submit" but you're saying this won't work? What should I
try?
Thanks again for your help!!
Kevin

Post by Godfrey Chan
Hi Kevin,
It looks like the "action" attribute of your form contains
something along the lines of "CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz".
The semicolon in here is not strictly a valid character in an URL. Try
replacing it with %3B.
form = page.at( ...select the form with a selector... )
action = form.attr('action')
action = action.gsub(';','%3B').gsub('=','%3D')
form.attr('action', action)
# then submit the form as usual (You can't do form.submit because
this form object we have here is a nokogiri node. You need to go through
mechanize to submit the form.)
Godfrey

Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone
On Mon, Feb 25, 2013 at 4:01 PM, Kevin Kornemann <

Post by Kevin Kornemann
Got a form, hitting a button with Mechanize. Thus far have been
going through login steps quite nicely so I know my Mechanize basics are up
to snuff.
CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz1:-1
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:176:in
`split'
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:211:in
`parse'
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:747:in
`parse'
`rescue in resolve'
`resolve'
`fetch'
`post_form'
`submit'
`submit'
from (irb):69
`start'
`start'
`<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'
From the googling I've done, seems to be related to an invalid
character in the re-direct, but really I have no idea...
Any way of ignoring these weird characters and continuing? Anyway
of fixing this?
Best,
Kevin

_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users

_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users
_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users

Godfrey Chan

2013-02-27 02:43:05 UTC

Permalink

I doubt that it's a UA string or cookie issue - I'd very carefully compare
the URIs from the previous pages and the offending form target URI.

Also, have you tried "form.action = './' + form.action" approach instead of
the gsub/CGI.escape approach? It could be that your server doesn't decode
the URL properly and hence missed the session ID.

Post by Kevin Kornemann
Yeah I started with, "Mac Safari"
It's an old site, so I thought try the oldest crappiest browser I could
use. But I suppose trying them all will only be slightly futile.
Now I'm wondering if it's a lost cookie thing. I understand Mech tracks
these automatically but I see from googling people still also manually
store them in a "cookie_jar"
pain...

Post by Godfrey Chan
Hi Kevin,
I finally have access to my computer now, and it turns out the semi
colon in the action is not the corrupt - the colon ("...:-1") was the
issue. (Didn't notice that before, I thought it was a line number or some
sort of error code. That's a *really* weird URI.)

Post by Kevin Kornemann

require 'mechanize'
agent = Mechanize.new
page = agent.get 'http://formtest.dev/form_test.html'
form = page.forms.first
form.submit

CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz1:-1

Post by Kevin Kornemann

form.action = CGI.escape(form.action)
form.submit # this should work now

If your server fail to URL decode that for some odd reason and refuse to
form.action = './' + form.action
Hope it helps.
Godfrey
On Mon, Feb 25, 2013 at 8:06 PM, Kevin Kornemann <

Post by Kevin Kornemann
Hi Godfrey,
1) Thanks again for your help, I really appreciate it.
2) yeah, so I can grab the action, gsub it nicely (I see the return).
But when I 'form.submit' it errors out on the original URI, as if the new
version wasn't changed at all...
I'll keep trying and let you know how I do. ENough for tonight.
thanks,
Kevin

Hi Kevin,
I don't currently have my computer with me to check the code - so
sorry I can't be more helpful. I might have made some of these methods up,
so apply a good dose of common sense if things didn't work as expected :)
form = agent.current_page.forms.first
form.action = form.action.gsub(...).gsub(...)
form.submit
Forget what I said earlier about the nokogiri stuff and form.submit
won't work etc. I wasn't aware that you can change the form action from
mechanize.
If form.submit somehow didn't work, please post a trace.
Godfrey
â
Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone
On Mon, Feb 25, 2013 at 7:38 PM, Kevin Kornemann <

Post by Godfrey Chan
form.attr('action', action)
to no avail...?
On Mon, Feb 25, 2013 at 10:35 PM, Kevin Kornemann <

Post by Kevin Kornemann
Hi Godfrey,
Thanks for your help. Really working hard to figure this out. I ran
into a road block that is my inexperience with nokogiri and mech.
When I try to grab the action (action = form.attr('action')), I
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
page = agent.get('https://somesite.jsp')
form = agent.page.forms.first
action = form.attr('action')
and undefined method...
I tried seeing if I could grab the action various ways to no avail.
My other question is, assuming I do this successfuly, I would
normally use "form.submit" but you're saying this won't work? What should I
try?
Thanks again for your help!!
Kevin

Post by Godfrey Chan
Hi Kevin,
It looks like the "action" attribute of your form contains
something along the lines of "CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz".
The semicolon in here is not strictly a valid character in an URL. Try
replacing it with %3B.
form = page.at( ...select the form with a selector... )
action = form.attr('action')
action = action.gsub(';','%3B').gsub('=','%3D')
form.attr('action', action)
# then submit the form as usual (You can't do form.submit because
this form object we have here is a nokogiri node. You need to go through
mechanize to submit the form.)
Godfrey
â
Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone
On Mon, Feb 25, 2013 at 4:01 PM, Kevin Kornemann <

Post by Kevin Kornemann
Got a form, hitting a button with Mechanize. Thus far have been
going through login steps quite nicely so I know my Mechanize basics are up
to snuff.
CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz1:-1
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:176:in
`split'
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:211:in
`parse'
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:747:in
`parse'
`rescue in resolve'
`resolve'
`fetch'
`post_form'
`submit'
`submit'
from (irb):69
`start'
`start'
`<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'
From the googling I've done, seems to be related to an invalid
character in the re-direct, but really I have no idea...
Any way of ignoring these weird characters and continuing? Anyway
of fixing this?
Best,
Kevin

_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users

Kevin Kornemann

2013-02-27 04:06:20 UTC

Permalink

that did it.

The identifier strings (no idea if that is what they are called) were all
preceded by a path of some sort, and this one wasn't. So simply adding the
./ before the id did the trick. I'm in! Just like Matthew Broderick in War
Games...

Can't express how my gratitude enough for your help. I googled and googled
and googled, nothing suggested this, which now that I understand what was
happening, well perhaps it's too simple a topic to deserve a stack overflow
post. Anyway, hopefully this helps a fellow newb in the future.

Thanks Godfrey.
Kevin

Post by Godfrey Chan
I doubt that it's a UA string or cookie issue - I'd very carefully compare
the URIs from the previous pages and the offending form target URI.
Also, have you tried "form.action = './' + form.action" approach instead
of the gsub/CGI.escape approach? It could be that your server doesn't
decode the URL properly and hence missed the session ID.

Post by Godfrey Chan
Hi Kevin,
I finally have access to my computer now, and it turns out the semi
colon in the action is not the corrupt - the colon ("...:-1") was the
issue. (Didn't notice that before, I thought it was a line number or some
sort of error code. That's a *really* weird URI.)

Post by Kevin Kornemann

require 'mechanize'
agent = Mechanize.new
page = agent.get 'http://formtest.dev/form_test.html'
form = page.forms.first
form.submit

CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz1:-1

Post by Kevin Kornemann

form.action = CGI.escape(form.action)
form.submit # this should work now

If your server fail to URL decode that for some odd reason and refuse
form.action = './' + form.action
Hope it helps.
Godfrey
On Mon, Feb 25, 2013 at 8:06 PM, Kevin Kornemann <

Post by Kevin Kornemann
Hi Godfrey,
1) Thanks again for your help, I really appreciate it.
2) yeah, so I can grab the action, gsub it nicely (I see the return).
But when I 'form.submit' it errors out on the original URI, as if the new
version wasn't changed at all...
I'll keep trying and let you know how I do. ENough for tonight.
thanks,
Kevin

Hi Kevin,
I don't currently have my computer with me to check the code - so
sorry I can't be more helpful. I might have made some of these methods up,
so apply a good dose of common sense if things didn't work as expected :)
form = agent.current_page.forms.first
form.action = form.action.gsub(...).gsub(...)
form.submit
Forget what I said earlier about the nokogiri stuff and form.submit
won't work etc. I wasn't aware that you can change the form action from
mechanize.
If form.submit somehow didn't work, please post a trace.
Godfrey

Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone
On Mon, Feb 25, 2013 at 7:38 PM, Kevin Kornemann <

Post by Godfrey Chan
form.attr('action', action)
to no avail...?
On Mon, Feb 25, 2013 at 10:35 PM, Kevin Kornemann <

Post by Kevin Kornemann
Hi Godfrey,
Thanks for your help. Really working hard to figure this out. I
ran into a road block that is my inexperience with nokogiri and mech.
When I try to grab the action (action = form.attr('action')), I
agent = Mechanize.new
agent.user_agent_alias = 'Mac Safari'
page = agent.get('https://somesite.jsp')
form = agent.page.forms.first
action = form.attr('action')
and undefined method...
I tried seeing if I could grab the action various ways to no avail.
My other question is, assuming I do this successfuly, I would
normally use "form.submit" but you're saying this won't work? What should I
try?
Thanks again for your help!!
Kevin
On Mon, Feb 25, 2013 at 9:06 PM, Godfrey Chan <

Post by Godfrey Chan
Hi Kevin,
It looks like the "action" attribute of your form contains
something along the lines of "CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz".
The semicolon in here is not strictly a valid character in an URL. Try
replacing it with %3B.
form = page.at( ...select the form with a selector... )
action = form.attr('action')
action = action.gsub(';','%3B').gsub('=','%3D')
form.attr('action', action)
# then submit the form as usual (You can't do form.submit because
this form object we have here is a nokogiri node. You need to go through
mechanize to submit the form.)
Godfrey

Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone
On Mon, Feb 25, 2013 at 4:01 PM, Kevin Kornemann <

Post by Kevin Kornemann
Got a form, hitting a button with Mechanize. Thus far have been
going through login steps quite nicely so I know my Mechanize basics are up
to snuff.
CBMenu.jsp;jsessionid=0000FWXUFvwFz8oQcES43L6CCz1:-1
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:176:in
`split'
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:211:in
`parse'
from
/Users/kevin/.rvm/rubies/ruby-1.9.3-p286/lib/ruby/1.9.1/uri/common.rb:747:in
`parse'
`rescue in resolve'
`resolve'
`fetch'
`post_form'
`submit'
`submit'
from (irb):69
`start'
`start'
`<top (required)>'
from script/rails:6:in `require'
from script/rails:6:in `<main>'
From the googling I've done, seems to be related to an invalid
character in the re-direct, but really I have no idea...
Any way of ignoring these weird characters and continuing?
Anyway of fixing this?
Best,
Kevin

_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users