Since URIs are often conveyed as strings it’s tempting to manipulate them as
such, but it’s better–and safer–to delegate URI manipulation to special
functions. These can then have their own unit-tests, which will take into
account the edge-cases that can catch us out.
I’ve just been doing a quick code review in the Ubiquity XForms project, and
one thing caught my eye that I thought might be worth a post.
Forms submission
In forms submission – both in XForms and HTML forms – we often need to add
parameters to a URI.
For example, if we have the URI http://example.org, and the parameters a=b
and c=d, then the resulting URI should be:
http://example.org?a=b&c;=d
It seems pretty straightforward that we need to add the parameters to the URI,
with a ‘?’ in between:
[uri] + ‘?’ + [parameters]
Adding parameters
We can see that the parameters themselves have been added by taking the name
and value (a=b, etc.), and adding it to the URI, ensuring that for all
parameters other than the first, there is a separator between. (The separator
can be either an ‘&’ or a ‘;’.)
Base URIs already containing parameters
The bug that needed fixing though, was that this ‘naive’ concatenation doesn’t
work if the URI you are dealing with already contains parameters.
For example, if we have the URI http://example.org?x=y, and we need to add
the same two parameters we had before, then our simple concatenation would
give us:
http://example.org?x=y?a=b&c;=d
when what we actually need is:
http://example.org?x=y&a;=b&c;=d
As you can see, if we already have a ‘?’ then we don’t need to add another, so
it seems that a simple addition to our concatenation code would be to use a
call to indexOf to see if there is a ‘?’ present, and only add another if we
don’t find one.
There’s a small additional test we’ll need to make which is to check whether
the last character in the URI is a ‘?’, as I’ll explain.
Base URIs with empty query strings
Recall that we added the parameters by placing a=b and c=d onto the end of
the URI, separated by ‘&’:
http://example.org?a=b&c;=d
Now, if we already have a URI with a query then we need to ensure that there
is an additional ‘&’ placed before our first parameter:
http://example.org?x=y&a;=b&c;=d
But what if the base URI has a query indicator (i.e., the ‘?’) but no
parameters? In other words, what if we have this URI:
http://example.org?
In this situation we don’t want to add the extra separator, otherwise we’ll
get this:
http://example.org?&a;=b&c;=d
So our rules now become that we only want to precede our parameters with a
separator if the ‘?’ is not the last character in the URI. It’s a little
awkward, but thanks to lastIndexOf, I’m sure we can manage.
Base URIs already containing fragments
However, there’s a further subtlety; what if the URI contains a fragment
identifier?
For example, if we have the URI http://example.org#x, and we need to add the
same two parameters we had before, then our simple concatenation would give us
this:
http://example.org#x?a=b&c;=d
The fragment identifier part of the URI has now become x?a=b&c;=d because
it’s always the last part of the URI. What we actually want is to insert the
new parameters before the ‘#’:
http://example.org?a=b&c;=d#x
Now we need to add another use of indexOf to check for a ‘#’, and if we find
one, use its position as the point at which to insert the parameters.
Context is everything
However, the assumption behind using indexOf and lastIndexOf in this way
is that a URI will contain only one ‘?’. A secondary assumption here is that
the only time you’ll ever see a ‘? is as an indicator of the query part of the
URI.
Both of these assumptions are incorrect.
Question marks in parameters
The first assumption is that you can only have one ‘?’ in a URI. However, the
query section of RFC
3986 explicitly
flags up that the ‘?’ character is a valid parameter value. For example, we
can have a=finished? as a parameter.
This means it’s quite easy to envisage scenarios where there is more than one
’?’ in the URI:
http://example.org?a=finished?&c;=d
This won’t necessarily mess up our first use of indexOf, but it will mess up
the use of lastIndexOf as a way to check whether you need to add an extra
separator. Recall that we wanted to avoid turning this:
http://example.org?
into this:
http://example.org?&a;=b&c;=d
so we used lastIndexOf to check whether the last character in the URI was a
’?’. But that algorithm will turn this:
http://example.org?a=finished?
into this:
http://example.org?a=finished?c=d
You might need to look closely to spot it, but because the last character was
a ‘?’, we haven’t added a separator before the c, and as a result, instead
of having two parameters (a=finished? and c=d) we have only one
(a=finished?c=d).
Question marks in fragments
The interesting thing about the previous examples is that at least you know
you have a query string, so you might be tempted to still use indexOf to
manipulate things. After all, although we may have too many ‘?’ characters, we
still know that we have query.
However, with the the fragment section of RFC
3986 all bets
are off; here we can see that ‘?’ is explicitly allowed as a fragment
character.
This means that it’s possible to have a ‘?’ in a URI even if it doesn’t have
a query. For example:
http://example.org#finished?
This may seem like a contrived example, but actually it’s not, for two
reasons.
The first is that the fragment part of a URI is carefully defined to allow
anything, because we don’t know how it will be interpreted. You may think that
“finished?” is not valid as an HTTP fragment, but what about in the scheme
“xyz”?
And this is the key point; since HTML forms and XForms can ultimately deal
with any scheme, since that’s how the web is designed, we must write our
algorithms defensively, and not assume anything.
Safe URI handling
Hopefully this delving into some of the subtleties of URI handling and parsing
–and we haven’t even begun to talk about turning relative paths into absolute
paths, handling encoded characters, and so on– has done enough to convince
you that you shouldn’t manipulate URIs directly, as simple strings.
The only way to be completely sure of what is happening is to use special
functions to unpack a URI into its various components, then manipulate those
components–perhaps adding additional parameters to the list of query
parameters, but it might also be to turn a relative path into an absolute path
–before finally reassembling the URI.
This may sound like a lot of work, but it’s the only way to be sure that
characters don’t get incorrectly interpreted as a consequence of their
position in the URI not being taken into account.
In the backplanejs library these
functions are in the URI module. (This is also imported into the Ubiquity
XForms library.) Breaking up a URI
simply involves calling spliturl, which returns an object containing all of
the parts. For example:
spliturl( “http://example.org?a=finished?” )
would give us the object:
{
scheme: “http:,
authority: “example.org”,
path: “”
query: “a=finished?”
fragment: “”
}
It’s then an easy matter to manipulate the query part, before creating a new
URI with the recomposeURI method.
Conclusion
Since URIs are often conveyed as strings then it’s tempting to manipulate them
as such. But the problem with doing this means that the context of a character
is rarely taken into account when processing.
It’s better–and safer–to delegate URI manipulation to special functions.
These can then have their own unit-tests, which will take into account the
edge-cases that can catch us out.
]]>
Creative Commons Attribution-Share Alike
3.0
