different page from ruby's mechanize than browser

[Josh Stratton]

My friend wrote a script, which uses ruby's mechanize to get a cookie off a
page after login.

* start mechanize
* go through login page form
* get cookie value from page delivered by form

The script seems to work for him even on my machine, but doesn't for me.
I've tried doing this interactively and I noticed I get a different page
title returned from Mechanize than the one for my browser. When I log in,
the page title is "SSL VPN - Home", but when I look at the title attribute
on my returned submission object I get "Please wait..."

irb(main):084:0> intermediate.title
=> "Please wait..."
irb(main):085:0> intermediate.iframes
=> [#<Mechanize::Page::Frame nil "/dana-na/html/blank.html">
Is Mechanize possibly not waiting long enough to get the destination page?
The "Please wait..." seems to hint at some intermediate page that gets
refreshed or redirected that the Mechanize session isn't getting to. Either
way I don't get the same title value.

Am I at some intermediate page my browser redirects from that mechanize
gets stuck at?

=> #<Mechanize::Page
{url
#<URI::HTTPS:0x00000002b49338 URL:
https://vpn1.example.com/dana/home/starter0.cgi?check=yes>}
{meta_refresh}
{title "Please wait..."}
{iframes #<Mechanize::Page::Frame nil "/dana-na/html/blank.html">}
{frames}
{links}
{forms
#<Mechanize::Form
{name "frmGrab"}
{method "POST"}
{action "/dana/home/starter0.cgi"}
{fields
[hidden:0x15b9860 type: hidden name: xsauth value:
a0395604a9f4f531504a7f6b6dc86051]
[hidden:0x15b94a0 type: hidden name: tz_offset value: ]
[hidden:0x15b92e8 type: hidden name: clienttime value: ]
[hidden:0x15b8ff0 type: hidden name: url value: ]
[hidden:0x15b8ac8 type: hidden name: activex_enabled value: 0]
[hidden:0x15b8898 type: hidden name: java_enabled value: 0]
[hidden:0x15b8690 type: hidden name: power_user value: 0]
[hidden:0x15b84ec type: hidden name: grab value: 1]
[hidden:0x15b8348 type: hidden name: browserproxy value: ]
[hidden:0x15b8168 type: hidden name: browsertype value: ]
[hidden:0x15b7fb0 type: hidden name: browserproxysettings value: ]
[hidden:0x15b7df8 type: hidden name: check value: yes]
[hidden:0x15b7c54 type: hidden name: nextpage value: ]
[hidden:0x15bb8f4 type: hidden name: mid value: ]
[hidden:0x15bb5d4 type: hidden name: signin value: ]
[hidden:0x15bb2dc type: hidden name: alias value: ]
[hidden:0x15bb05c type: hidden name: id value: ]
[hidden:0x15baeb8 type: hidden name: username value: ]
[hidden:0x15bacb0 type: hidden name: password value: ]
[hidden:0x15bab20 type: hidden name: occurrence value: ]}
{radiobuttons}
{checkboxes}
{file_uploads}
{buttons}>}>

[Timothy Snowhite]

Use intermediate.body to see the HTML returned, and look for a meta-refresh
element or a JavaScript redirect.

Post by Josh Stratton
My friend wrote a script, which uses ruby's mechanize to get a cookie off
a page after login.
* start mechanize
* go through login page form
* get cookie value from page delivered by form
The script seems to work for him even on my machine, but doesn't for me.
I've tried doing this interactively and I noticed I get a different page
title returned from Mechanize than the one for my browser. When I log in,
the page title is "SSL VPN - Home", but when I look at the title attribute
on my returned submission object I get "Please wait..."
irb(main):084:0> intermediate.title
=> "Please wait..."
irb(main):085:0> intermediate.iframes
=> [#<Mechanize::Page::Frame nil "/dana-na/html/blank.html">
Is Mechanize possibly not waiting long enough to get the destination page?
The "Please wait..." seems to hint at some intermediate page that gets
refreshed or redirected that the Mechanize session isn't getting to. Either
way I don't get the same title value.
Am I at some intermediate page my browser redirects from that mechanize
gets stuck at?
=> #<Mechanize::Page
{url
https://vpn1.example.com/dana/home/starter0.cgi?check=yes>}
{meta_refresh}
{title "Please wait..."}
{iframes #<Mechanize::Page::Frame nil "/dana-na/html/blank.html">}
{frames}
{links}
{forms
#<Mechanize::Form
{name "frmGrab"}
{method "POST"}
{action "/dana/home/starter0.cgi"}
{fields
a0395604a9f4f531504a7f6b6dc86051]
[hidden:0x15b94a0 type: hidden name: tz_offset value: ]
[hidden:0x15b92e8 type: hidden name: clienttime value: ]
[hidden:0x15b8ff0 type: hidden name: url value: ]
[hidden:0x15b8ac8 type: hidden name: activex_enabled value: 0]
[hidden:0x15b8898 type: hidden name: java_enabled value: 0]
[hidden:0x15b8690 type: hidden name: power_user value: 0]
[hidden:0x15b84ec type: hidden name: grab value: 1]
[hidden:0x15b8348 type: hidden name: browserproxy value: ]
[hidden:0x15b8168 type: hidden name: browsertype value: ]
[hidden:0x15b7fb0 type: hidden name: browserproxysettings value: ]
[hidden:0x15b7df8 type: hidden name: check value: yes]
[hidden:0x15b7c54 type: hidden name: nextpage value: ]
[hidden:0x15bb8f4 type: hidden name: mid value: ]
[hidden:0x15bb5d4 type: hidden name: signin value: ]
[hidden:0x15bb2dc type: hidden name: alias value: ]
[hidden:0x15bb05c type: hidden name: id value: ]
[hidden:0x15baeb8 type: hidden name: username value: ]
[hidden:0x15bacb0 type: hidden name: password value: ]
[hidden:0x15bab20 type: hidden name: occurrence value: ]}
{radiobuttons}
{checkboxes}
{file_uploads}
{buttons}>}>

[Josh Stratton]

Looking at the html, I don't see a javascript redirect or meta-refresh.

http://pastebin.com/4SPArnUR

I do see a reference to an iframe, but I'm not sure if it's related. I'm
not sure how to use it either.

irb(main):114:0> intermediate.iframes.length
=> 1

irb(main):112:0> page = intermediate.iframes.first
=> #<Mechanize::Page::Frame nil "/dana-na/html/blank.html">

irb(main):116:0> page.class
=> Mechanize::Page::Frame
irb(main):117:0> page.links

NoMethodError: undefined method `links' for #<Mechanize::Page::Frame nil
"/dana-na/html/blank.html">

from (irb):117
from /usr/bin/irb:12:in `<main>'

I don't see any relevant data on it. I'm confused because the functions I
see in the docs like "links" don't seem to exist on it.

Post by Timothy Snowhite
Use intermediate.body to see the HTML returned, and look for a
meta-refresh element or a JavaScript redirect.

Post by Josh Stratton
My friend wrote a script, which uses ruby's mechanize to get a cookie off
a page after login.
* start mechanize
* go through login page form
* get cookie value from page delivered by form
The script seems to work for him even on my machine, but doesn't for me.
I've tried doing this interactively and I noticed I get a different page
title returned from Mechanize than the one for my browser. When I log in,
the page title is "SSL VPN - Home", but when I look at the title attribute
on my returned submission object I get "Please wait..."
irb(main):084:0> intermediate.title
=> "Please wait..."
irb(main):085:0> intermediate.iframes
=> [#<Mechanize::Page::Frame nil "/dana-na/html/blank.html">
Is Mechanize possibly not waiting long enough to get the destination
page? The "Please wait..." seems to hint at some intermediate page that
gets refreshed or redirected that the Mechanize session isn't getting to.
Either way I don't get the same title value.
Am I at some intermediate page my browser redirects from that mechanize
gets stuck at?
=> #<Mechanize::Page
{url
https://vpn1.example.com/dana/home/starter0.cgi?check=yes>}
{meta_refresh}
{title "Please wait..."}
{iframes #<Mechanize::Page::Frame nil "/dana-na/html/blank.html">}
{frames}
{links}
{forms
#<Mechanize::Form
{name "frmGrab"}
{method "POST"}
{action "/dana/home/starter0.cgi"}
{fields
a0395604a9f4f531504a7f6b6dc86051]
[hidden:0x15b94a0 type: hidden name: tz_offset value: ]
[hidden:0x15b92e8 type: hidden name: clienttime value: ]
[hidden:0x15b8ff0 type: hidden name: url value: ]
[hidden:0x15b8ac8 type: hidden name: activex_enabled value: 0]
[hidden:0x15b8898 type: hidden name: java_enabled value: 0]
[hidden:0x15b8690 type: hidden name: power_user value: 0]
[hidden:0x15b84ec type: hidden name: grab value: 1]
[hidden:0x15b8348 type: hidden name: browserproxy value: ]
[hidden:0x15b8168 type: hidden name: browsertype value: ]
[hidden:0x15b7fb0 type: hidden name: browserproxysettings value: ]
[hidden:0x15b7df8 type: hidden name: check value: yes]
[hidden:0x15b7c54 type: hidden name: nextpage value: ]
[hidden:0x15bb8f4 type: hidden name: mid value: ]
[hidden:0x15bb5d4 type: hidden name: signin value: ]
[hidden:0x15bb2dc type: hidden name: alias value: ]
[hidden:0x15bb05c type: hidden name: id value: ]
[hidden:0x15baeb8 type: hidden name: username value: ]
[hidden:0x15bacb0 type: hidden name: password value: ]
[hidden:0x15bab20 type: hidden name: occurrence value: ]}
{radiobuttons}
{checkboxes}
{file_uploads}
{buttons}>}>

_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users

[Godfrey Chan]

there's a unload handler attached to the body which then perform a bunch of crazy stuff, and then finally submit a form which causes the page to navigate away.


You'll either have to reverse engineer that, or use a headless browser with javascript support. 


Godfrey
—
Sent from Mailbox for iPhone

Post by Josh Stratton
Looking at the html, I don't see a javascript redirect or meta-refresh.
http://pastebin.com/4SPArnUR
I do see a reference to an iframe, but I'm not sure if it's related. I'm
not sure how to use it either.
irb(main):114:0> intermediate.iframes.length
=> 1
irb(main):112:0> page = intermediate.iframes.first
=> #<Mechanize::Page::Frame nil "/dana-na/html/blank.html">
irb(main):116:0> page.class
=> Mechanize::Page::Frame
irb(main):117:0> page.links
NoMethodError: undefined method `links' for #<Mechanize::Page::Frame nil
"/dana-na/html/blank.html">
from (irb):117
from /usr/bin/irb:12:in `<main>'
I don't see any relevant data on it. I'm confused because the functions I
see in the docs like "links" don't seem to exist on it.

Post by Timothy Snowhite
Use intermediate.body to see the HTML returned, and look for a
meta-refresh element or a JavaScript redirect.

Post by Josh Stratton
My friend wrote a script, which uses ruby's mechanize to get a cookie off
a page after login.
* start mechanize
* go through login page form
* get cookie value from page delivered by form
The script seems to work for him even on my machine, but doesn't for me.
I've tried doing this interactively and I noticed I get a different page
title returned from Mechanize than the one for my browser. When I log in,
the page title is "SSL VPN - Home", but when I look at the title attribute
on my returned submission object I get "Please wait..."
irb(main):084:0> intermediate.title
=> "Please wait..."
irb(main):085:0> intermediate.iframes
=> [#<Mechanize::Page::Frame nil "/dana-na/html/blank.html">
Is Mechanize possibly not waiting long enough to get the destination
page? The "Please wait..." seems to hint at some intermediate page that
gets refreshed or redirected that the Mechanize session isn't getting to.
Either way I don't get the same title value.
Am I at some intermediate page my browser redirects from that mechanize
gets stuck at?
=> #<Mechanize::Page
{url
https://vpn1.example.com/dana/home/starter0.cgi?check=yes>}
{meta_refresh}
{title "Please wait..."}
{iframes #<Mechanize::Page::Frame nil "/dana-na/html/blank.html">}
{frames}
{links}
{forms
#<Mechanize::Form
{name "frmGrab"}
{method "POST"}
{action "/dana/home/starter0.cgi"}
{fields
a0395604a9f4f531504a7f6b6dc86051]
[hidden:0x15b94a0 type: hidden name: tz_offset value: ]
[hidden:0x15b92e8 type: hidden name: clienttime value: ]
[hidden:0x15b8ff0 type: hidden name: url value: ]
[hidden:0x15b8ac8 type: hidden name: activex_enabled value: 0]
[hidden:0x15b8898 type: hidden name: java_enabled value: 0]
[hidden:0x15b8690 type: hidden name: power_user value: 0]
[hidden:0x15b84ec type: hidden name: grab value: 1]
[hidden:0x15b8348 type: hidden name: browserproxy value: ]
[hidden:0x15b8168 type: hidden name: browsertype value: ]
[hidden:0x15b7fb0 type: hidden name: browserproxysettings value: ]
[hidden:0x15b7df8 type: hidden name: check value: yes]
[hidden:0x15b7c54 type: hidden name: nextpage value: ]
[hidden:0x15bb8f4 type: hidden name: mid value: ]
[hidden:0x15bb5d4 type: hidden name: signin value: ]
[hidden:0x15bb2dc type: hidden name: alias value: ]
[hidden:0x15bb05c type: hidden name: id value: ]
[hidden:0x15baeb8 type: hidden name: username value: ]
[hidden:0x15bacb0 type: hidden name: password value: ]
[hidden:0x15bab20 type: hidden name: occurrence value: ]}
{radiobuttons}
{checkboxes}
{file_uploads}
{buttons}>}>

_______________________________________________
Mechanize-users mailing list
http://rubyforge.org/mailman/listinfo/mechanize-users

[Timothy Snowhite]

I believe intermediate.iframes.first.click will load the iframe for you.

It might help to run Charles (a debugging proxy) while you log in to the
site via a browser to see what all requests get sent throughout the login
process.

Post by Godfrey Chan
there's a unload handler attached to the body which then perform a bunch
of crazy stuff, and then finally submit a form which causes the page to
navigate away.
You'll either have to reverse engineer that, or use a headless browser
with javascript support.
Godfrey
—
Sent from Mailbox <https://bit.ly/SZvoJe> for iPhone
Looking at the html, I don't see a javascript redirect or meta-refresh.
http://pastebin.com/4SPArnUR
I do see a reference to an iframe, but I'm not sure if it's related. I'm
not sure how to use it either.
irb(main):114:0> intermediate.iframes.length
=> 1
irb(main):112:0> page = intermediate.iframes.first
=> #<Mechanize::Page::Frame nil "/dana-na/html/blank.html">
irb(main):116:0> page.class
=> Mechanize::Page::Frame
irb(main):117:0> page.links
NoMethodError: undefined method `links' for #<Mechanize::Page::Frame nil
"/dana-na/html/blank.html">
from (irb):117
from /usr/bin/irb:12:in `<main>'
I don't see any relevant data on it. I'm confused because the functions I
see in the docs like "links" don't seem to exist on it.
Use intermediate.body to see the HTML returned, and look for a
meta-refresh element or a JavaScript redirect.
My friend wrote a script, which uses ruby's mechanize to get a cookie off
a page after login.
* start mechanize
* go through login page form
* get cookie value from page delivered by form
The script seems to work for him even on my machine, but doesn't for me.
I've tried doing this interactively and I noticed I get a different page
title returned from Mechanize than the one for my browser. When I log in,
the page title is "SSL VPN - Home", but when I look at the title attribute
on my returned submission object I get "Please wait..."
irb(main):084:0> intermediate.title
=> "Please wait..."
irb(main):085:0> intermediate.iframes
=> [#<Mechanize::Page::Frame nil "/dana-na/html/blank.html">
Is Mechanize possibly not waiting long enough to get the destination page?
The "Please wait..." seems to hint at some intermediate page that gets
refreshed or redirected that the Mechanize session isn't getting to. Either
way I don't get the same title value.
Am I at some intermediate page my browser redirects from that mechanize
gets stuck at?
=> #<Mechanize::Page
{url
https://vpn1.example.com/dana/home/starter0.cgi?check=yes>}
{meta_refresh}
{title "Please wait..."}
{iframes #<Mechanize::Page::Frame nil "/dana-na/html/blank.html">}
{frames}
{links}
{forms
#<Mechanize::Form
{name "frmGrab"}
{method "POST"}
{action "/dana/home/starter0.cgi"}
{fields
a0395604a9f4f531504a7f6b6dc86051]
[hidden:0x15b94a0 type: hidden name: tz_offset value: ]
[hidden:0x15b92e8 type: hidden name: clienttime value: ]
[hidden:0x15b8ff0 type: hidden name: url value: ]
[hidden:0x15b8ac8 type: hidden name: activex_enabled value: 0]
[hidden:0x15b8898 type: hidden name: java_enabled value: 0]
[hidden:0x15b8690 type: hidden name: power_user value: 0]
[hidden:0x15b84ec type: hidden name: grab value: 1]
[hidden:0x15b8348 type: hidden name: browserproxy value: ]
[hidden:0x15b8168