Percent Encoding URL and ^

In 3/4 browsers, ^ is converted to %5E.

However, in Mozilla, it is not:-

load up the following and click on the link and notice the location
bar:-

"http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">





test




IE6, Opera, Safari 3:
http://example.com/?n=%5E

Firefox:
http://example.com/?n=^

My understanding says that firefox is wrong on this one.

Data must be escaped if it does not have a representation using an
unreserved
character.[1]

RFC 3986[2] (Jan 1995) clearly states which characters are allowed in
URLs-

| Characters that are allowed in a URI but do not have a reserved
| purpose are called unreserved. These include uppercase and
lowercase
| letters, decimal digits, hyphen, period, underscore, and tilde.
|
| unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"

Copying the generated URIs results in Mozilla following HTML 4.01,
Appendix B.2
Special characters in URI attribute values[3], which
states:-

| Although URIs do not contain non-ASCII values (see [URI], section
| 2.1) authors sometimes specify them in attribute values expecting
URIs
| (i.e., defined with %URI; in the DTD). For instance, the following
href
| value is illegal:
| ...
|
| We recommend that user agents adopt the following convention for
handling
| non-ASCII characters in such cases:
|
| 1. Represent each character in UTF-8 (see [RFC2279]) as one or
more bytes.
| 2. Escape these bytes with the URI escaping mechanism (i.e., by
converting
| each byte to %HH, where HH is the hexadecimal notation of the byte
value).

The "national" and "punctuation" characters (which includes "^") do
not appear
in any productions and therefore may not appear in URLs.[4]

It gets worse. When setting innerHTML on an element, if the new HTML
string has a '^', mozilla turns that into %5E, but all other browsers
leave it as '^'.

Our application uses ^ literally. The URL handling service must send
and receive the same characters. This can be either %5E or ^. If
sending %5E to the client, if ^ comes back, the server will not use
that as a key, and if '^' is sent to the client and %5E comes back,
the server will not compare them as being equal.

[1]http://www.ietf.org/rfc/rfc2396.txt
[2]http://tools.ietf.org/html/rfc3986#section-2.3
[3]http://www.w3.org/TR/REC-html40/appendix/notes.html#h-B.2
[4]http://www.ietf.org/rfc/rfc1630.txt

On Nov 24, 11:16*am, dhtml wrote:
> In 3/4 browsers, ^ is converted to %5E.
>
Sorry, I meant:

%5E is not converted to '^' , but in Mozilla, it is converted to the
literal '^'

dhtml wrote:

> In 3/4 browsers, ^ is converted to %5E.
>
> However, in Mozilla, it is not:-

In URLs, you mean? But you seem to be confused.

> load up the following and click on the link and notice the location
> bar:-

Why didn't _you_ upload it and post a URL?

> test

Why do you use a URL that is guaranteed to fail to work, instead of a real
URL? What makes you think anything will appear in the location bar when the
URL is fake?

Most importantly, your URL contains "%5E", not "^".

> Firefox:
> http://example.com/?n=^

So you are actually complaining that Firefox _does_ do a conversion, not
that it does not, and it is a conversion of "%5E" to "^", not vice versa,

> Data must be escaped if it does not have a representation using an
> unreserved character.[1]

Citing an obsolete RFC is pointless, especially when it has been superseded
by an Internet-standard.

> RFC 3986[2] (Jan 1995) clearly states which characters are allowed in
> URLs-

It's Jan 2005, not 1995, and it's not particularly clear in character
issues, following an old tradition of URI RFCs.

The relevant part seems to be, sadly but symptomatically, in the formal
syntax (for the query part, which you seem to imply):

query = *( pchar / "/" / "?" )
pchar = unreserved / pct-encoded / sub-delims / ":" / "@"
unreserved = ALPHA / DIGIT / "-" / "." / "_" / "~"
reserved = gen-delims / sub-delims
gen-delims = ":" / "/" / "?" / "#" / "[" / "]" / "@"
sub-delims = "!" / "$" / "&" / "'" / "(" / ")"
/ "*" / "+" / "," / ";" / "="

This seems to boil down to the conclusion that "^" as such is not allowed in
the query part but must be percent-encoded. Thus, the URL
http://example.com/?n=%5E
complies with the syntax whereas
http://example.com/?n=^
does not.

You might have a case against Firefox as regards to _displaying_ the %5E in
a URL as "^" on the status line or on the location bar. However, if you
check what Firefox actually sends (e.g. using the Live HTTP headers
extension), you will see that it sends %5E. Hence it is correct
protocol-wise, though maybe not to everyone's taste display-wise (but we can
say that there is no requirement on such display issues: a browser may e.g.
display a URL in a location bar, or it may display a string derived from the
URL, or it may have no location bar, or it may serve drinks there).

> It gets worse. When setting innerHTML on an element, if the new HTML
> string has a '^', mozilla turns that into %5E, but all other browsers
> leave it as '^'.

Not in my test. Why don't you post a URL, and why don't you post to a
different group, when your question is not about HTML but about what happens
in scripting?

--
Yucca, http://www.cs.tut.fi/~jkorpela/

On Nov 24, 12:06*pm, "Jukka K. Korpela" wrote:
> dhtml wrote:
> > In 3/4 browsers, ^ is converted to %5E.
>
> > However, in Mozilla, it is not:-
>
> In URLs, you mean? But you seem to be confused.
>

Yes of course, in an anchor's href attribute specifically (I have not
checked img or iframe, et c.)


[snip]

>
> > test
>
> Why do you use a URL that is guaranteed to fail to work, instead of a real
> URL? What makes you think anything will appear in the location bar when the
> URL is fake?
>

What about that URL is fake?

> Most importantly, your URL contains "%5E", not "^".
>
> > Firefox:
> >http://example.com/?n=^
>
> So you are actually complaining that Firefox _does_ do a conversion, not
> that it does not, and it is a conversion of "%5E" to "^", not vice versa,
>

Yes.


> This seems to boil down to the conclusion that "^" as such is not allowedin
> the query part but must be percent-encoded. Thus, the URLhttp://example.com/?n=%5E
> complies with the syntax whereashttp://example.com/?n=^
> does not.
>

Exactly.

> You might have a case against Firefox as regards to _displaying_ the %5E in
> a URL as "^" on the status line or on the location bar. However, if you
> check what Firefox actually sends (e.g. using the Live HTTP headers
> extension), you will see that it sends %5E. Hence it is correct
> protocol-wise, though maybe not to everyone's taste display-wise (but we can
> say that there is no requirement on such display issues: a browser may e.g.
> display a URL in a location bar, or it may display a string derived from the
> URL, or it may have no location bar, or it may serve drinks there).
>

Looking in live http headers makes sense. Firebug displays the same.

> > It gets worse. When setting innerHTML on an element, if the new HTML
> > string has a '^', mozilla turns that into %5E, but all other browsers
> > leave it as '^'.
>
> Not in my test. Why don't you post a URL, and why don't you post to a
> different group, when your question is not about HTML but about what happens
> in scripting?

Will F'up to c.l.javascript. I need more time to focus on making a
post (I am at work now).

The problem I have now is that our URL service uses '^' literally.
This breaks in Safari 3 on windows and causes problems when doing
innerHTML swaps in Firefox. using Live HTTP Headers, with the
following example:-

================================================== ========

Copy Links

source

• ^


• http://example.com/?^

On 2008-11-24, dhtml wrote:
> In 3/4 browsers, ^ is converted to %5E.
>
> However, in Mozilla, it is not:-
[...]
> It gets worse. When setting innerHTML on an element, if the new HTML
> string has a '^', mozilla turns that into %5E, but all other browsers
> leave it as '^'.
>
> Our application uses ^ literally. The URL handling service must send
> and receive the same characters. This can be either %5E or ^. If
> sending %5E to the client, if ^ comes back, the server will not use
> that as a key, and if '^' is sent to the client and %5E comes back,
> the server will not compare them as being equal.

Why not fix the application on the server so it always "unescapes"
everything? That way it doesn't matter.

On Mon, 24 Nov 2008, dhtml wrote:

> In 3/4 browsers,

75% of all browsers?

> load up the following and click on the link

Give an address (URL) of a test document.

dhtml wrote:

>>> test
[...]
> What about that URL is fake?

Please consult BCP 32. (Hint: The IETF guarantees that example.com never
works.)

--
Yucca, http://www.cs.tut.fi/~jkorpela/

In article ,
"Jukka K. Korpela" wrote:

> dhtml wrote:
>
> >>> test
> [...]
> > What about that URL is fake?
>
> Please consult BCP 32. (Hint: The IETF guarantees that example.com never
> works.)

Well, yes, although you can type it into your browser and see what you
get! This from FireFox 3.0.4 running on Mac OS 10.4.11:

You have reached this web page by typing "example.com",
"example.net", or "example.org" into your web browser.

These domain names are reserved for use in documentation
and are not available for registration. See RFC 2606, Section 3*.

I'm not sure if the quoted page is built into FF or someplace
else, but it seems clear enough!

* http://www.rfc-editor.org/rfc/rfc2606.txt