Apr 4, 2007
Real world data
Today, I started building a simple Twitter application with SWX and ran into two issues. Firstly, SWX doesn't currently support PHP objects (it supports associative arrays). This wasn't a problem when echoing data back to Flash since data sent from the Flash client is always serialized to an associative array currently (not an object) but, the moment you feed data from another source, such as the JSON parser in Amfphp, that *does* spew objects... well, it breaks.
To-do: Re-implement object support. (I had it in there initially but then removed it.) Update: I just implemented a workaround for the time-being that converts objects to associate arrays. I do want to look into actual object support in the future, however my focus at the moment is to get SWX to the point where we can build mashups, etc., that use *existing* service classes that are compatible with Amfphp.
The other thing is that something in the returned data from Twitter is breaking the assembler. I'm looking at that now and will post a new release once I've handled it. Apparently, this was just a side-effect of the object issue I outlined, above.
In your own testing, if you notice SWX break on real-world data, it would help me greatly if you can send me your source so I can recreate and fix the problem. Running a tail on the PHP error log will help you see the errors that SWX throws (sending me this output, along with the source will be much appreciated!)
[...] main focus in building this was to see what would break when using real world data and to see how easily you can build error checking and recovery into your SWX [...]
You should really consider removing the reason this doesn’t Just Work: PHON. It’s useless.
It’s no easier to generate PHON from Flash than JSON, and it’s no easier to safely parse PHON from PHP than JSON. Deviating from the established standard here buys you absolutely nothing… except bugs, porting difficulties, potential security problems, and probably even speed penalties. Parsing JSON is *very very fast* with a compiled extension, and it’s even built-in to PHP as of 5.2.
Hi Bob,
This has nothing to do with PHON — it’s a SWF bytecode issue.
The simplicity of SWX would suffer greatly if people had to install a JSON extension to make use of it. That goes entirely against the philosophy of SWX.
Remember that JSON was not an established standard when it came out. In fact, people riled against it, giving the argument that XML should be used instead of a radical new format because it was well established.
I think it’s too early in the game to be spreading FUD about PHON (man, we sure love acronyms in our field, don’t we?!)
JSON is an established standard. It’s been around for over 10 years (though recently “named”, I used a variant a long time ago.. in the same way I did AJAX stuff back then). It is established to the point where it is *part of PHP*.
If you’re opposed to having extensions it’d be far better to include a copy of some pure PHP JSON parser with the SWX server that is used when an extension is not available. In order to securely and correctly parse PHON you’re going to need a bunch of SWX specific code on both ends, where JSON you would be able to re-use something someone else wrote and thoroughly tested over the past year or two on both ends that does exactly the same thing with the same impedance and probably even a slightly more compact representation (though not meaningfully).
PHON is just a bad idea, you could do better. I really like the idea of SWX (loadMovie as data transport) and I’ve been using something just like it in production for about 9 months. I’d probably even bother to open source the similar code I’ve got in Python and Erlang and make it speak SWX (I have suitable bytecode generators in both), but unless the PHON nonsense goes away I definitely won’t bother. I don’t want there to be SWX-for-PHP and SWX-for-Python where they have to speak different protocols on both ends, and I don’t want to implement PHON in Python or Erlang. I shouldn’t have to. PHON has no benefits *whatsoever* over JSON, for which there are many perfectly good parsers in every language you’ve ever heard of and probably more that you haven’t.
You’re of course right that this particular situation was a bytecode issue though, I’d for whatever reason read it a bit backwards. My bad.
Hi Bob,
As far as the user (developer) is concerned, the format the arguments from Flash to the back-end are serialized in is transparent. It could be in Voodoo 2.0 and still work. So making it work with something else is simple. It’s not a priority at the moment though as there are far more pressing changes in the very short term (cross-domain, docs, easy-to-install releases.)
Not discounting it — it’s very possible that PHON won’t make it to the final release and very possible that something else will. But I do like the spirit of PHON (or PYON for Python, etc.) which is that you don’t use a parser.
If using JSON, you will have to use (or build) a parser in Python that parses a subset of JavaScript. This is also very much nonsense when you think of it. In fact, I’d argue that speaking the same language as your backend makes far more sense than speaking a different language to it. As an analogy, following the JSON-for-everything approach, we would speak to people around the world in English and expect them all to have English-to-their-own-language translators. The PHON approach would be to speak to them in their own language and not require translators.
PHON is not a core/unchangeable component of SWX but it is not my priority to change it at this at the moment.
Regarding JSON being a standard — it’s not. Here’s a blog post from February of this year, making the same arguments against JSON:
Again, I find a lot of those arguments to be FUD — especially the security concerns. Security isn’t black magic — either there’s a way to exploit something (or multiple ways) or there isn’t. If there is, you fix it.
If anyone can actually demonstrate to me a security issue with the way PHON is sanitized and
evaled then I will definitely look into it.Not using eval guarantees a certain level of security, where using eval guarantees a certain level of insecurity. If they can somehow worm around your regex, then they can execute anything they want. What you’re doing is taking a guaranteed exploit and trying to make it secure by sanitizing the input, rather than simply having no possibility for exploit at all by using a parser that doesn’t execute code.
Using eval for any language is always going to be slower than a written-in-C (or equivalent) parser for JSON almost by definition. It’s also not applicable to many languages (like Java or Haskell for example), and in some languages it’s going to be another vector for a denial of service attack no matter how you sanitize it. Compile-time strings are often interned by compilers, so if you keep sending different strings on input then you can intern so many strings that the server runs out of memory and crashes (assuming you have a long-running interpreter process, which you probably do in anything but PHP).
From the user’s perspective it isn’t irrelevant. Since your encoding goes in your client they’ve decided on a protocol. They can only speak to servers that speak the same protocol, which is currently only your PHON based implementation. If they want to write a Python based server component then they’d have to recompile the client code (given your terrible idea of making separate encodings for each flavor of server). Why the hell should they have to do that? Even worse would be bloating the client code such that it can speak 9 different encodings.
JSON is a de facto standard. There’s an RFC and there are hundreds of implementations of it. Millions of people use JSON-based services from (at least) thousands of vendors.
The only way to have multiple “whatever works” server protocols without fucking up the client would be do use a LocalConnection based RPC. The client loads a communication SWF that the SWX server provides, probably providing the name of a random LocalConnection id on the query string. On load, the communication SWF sends a message back to the client over that LocalConnection containing its LocalConnection id. From then on they simply communicate over this LocalConnection pair and the communication SWF talks to the server however the heck it wants to (could even be XMLSocket or RTMP). This does impose a certain “packet size” on communication but that’s not necessarily a bad thing.
Oh, and a LocalConnection based RPC also solves the Flash version problem: communication SWF could be Flash 6 written in AS1 and client SWF could be Flash 9 written in AS3 and it would just work.
If the LocalConnection objects are constructed appropriately (with the right names) then it also solves the cross-domain problem because it would Just Work.
Hi Bob,
Well, you make some valid points. You’re definitely right — it would suck to have essentially the same client-side implementation nine times. Bloating the client is not a good thing.
Listen, I’m not tied to PHON. It was a cool idea that was easy to implement and got the proof of concept up and running. The key thing is the SWX format, really. I’m not convinced that JSON is the best replacement, though (not saying it’s not either.) You’re right that JSON is a de facto standard. How fast are the non-C JSON parsers? I guess using JSON would make it easier to port too.
OK, so you’re selling me on JSON.
Re: LocalConnection — it is very useful indeed. The Analyzer (a Flex 2 app) uses it already. In fact, you could use it in your own apps by bastardizing the debug functionality (just create a debug method like the Analyzer does in your Flex 2/AS3 apps.)
The non-C JSON parsers are more than fast enough. I’ve never heard someone say that the parsing was a bottleneck in any application. I wrote simplejson, which is probably the most popular JSON suite for Python. It only recently grew some optional C acceleration (just for encoding, so not relevant). Decoding w/ simplejson (not accelerated) is hundreds to tens of thousands of docs per second depending on the size and complexity of the docs (on my MBP). A similarly written PHP, Perl, Ruby parser would have similar performance. C acceleration will almost certainly give you a 5-100x improvement, but it’s not normally a bottleneck so it doesn’t really matter.
Remember, just 12 requests per second on average is over a million per day. If you’re doing hundreds of requests per second (about the level where the decoder might matter) then installing a C extension is probably within your reach. By then you’ve probably got bigger problems solved by scaling to multiple machines anyway.
Using JSON would make it very easy to port. All you’d really need to do is find or write a SWF bytecode generator and you’ve got SWX.
There’s only three sensible options:
- A plain URL encoded POST (unlikely to be flexible enough)
- XML (bulky, and what schema?)
- JSON (no predefined “schema” for types beyond Object; but fixable the same way you’d do it in XML)
Personally I don’t often write code that depends on serialization of specific types at each end of the wire. I like to work with “just data” (e.g. plain JSON) because it’s so loosely coupled and easy to debug.
The amount of data *sent* from the client is usually small in any case so I don’t see the parser being a bottleneck either.
Btw, I agree, JSON is the way to go — thanks for pressing the issue.
Specifically , having a single format:
* Will reduce code on the client
* Will make it easier to port
* Will reduce the complexity of learning/understanding the entire ecosystem (all ports, etc.)
And using JSON is a great idea because
* It’s lightweight [] is lighter than array(), for example
* De-facto standard
* Has existing libraries on all major platforms
I’ve changed my mind on this. Going to implement JSON as a priority. Perhaps even for the next release.
Thanks, again — I appreciate your guidance on this point.
Hi there Aral,
perhaps shud have posted this on the mailing list, but searching on swx and associative arrays there is nothing out there but this page at the moment…so I thought might as well keep it here. Shoot me if its too long.
I want to check what kind of objects the arrays returned from PHP become in Flash, if that is something you have come up with, what it is, and why etc. I have done some quick comparative tests between AMF and SWX and here are my results.
If you send an array from flash via AMF and SWX it will be treated by PHP as an array alike, whereas if you send an object via SWX it will be treated like an object instead (eg. assume obj->prop syntax) which is not the case with AMF. That is interesting, arguably more consistent with Flash, but nevertheless a cause for separate coding dependent on the transfer used. (You can of course cast it as an array in PHP by using (array) myObj and they will be treated the same I think).
If we look at it from POV of PHP and define an array there and return it to Flash, ie.
$arr = array();
$arr['5']=’value1′;
$arr[10]=’value2′;
the result is quite confusing. If sent via AMF it will become a normal Array (as proved by checking instanceOf Array), it will have length 11 and empty slots up till 10 apart from 5 (even if set as string in PHP).
If sent via SWX instead it will have length 2, trace out fine with both a for and a for in loop but still not be a normal Array.
Lets modify the array and make it mixed like such
$arr = array();
$arr['5']=’value1′;
$arr[10]=’value2′;
$arr['swx']=’value3′;
If sent via AMF it will still be a native Array instance, have length 11, trace exactly the same in a for loop as above (not including the new string property). A for in loop however outputs only the 3 elements above. That makes sense since string property (’swx’) and the numeric properties all belong to the Array object.
SWX on the other hand traces out length as undefined and nothing in a normal for loop. The for in loop behaves the same though.
(Incidentally PHP doesn’t count the empty slots in a numeric array but would output length 2 and 3 respectively above (using count($arr)).)
My question is what happens to a normal numeric array in SWX? Why does it become an associative array like object when that is not a native format to flash at all (seeing its supposed to be THE native data format). Is there any reason you have chosen not to convert to simple array? Is there a way one can just modify the swxAssembler to not convert them to associative arrays?
To conclude: Seems we have quite a few differences in how AMF and SWX deal with PHP arrays and objects. If you want them to become bossom buddies (which is a fabulous idea) I would suggest either to make them behave consistently or add some compatibility flag in the swx config where you can choose how to deal with these differences.
Keep it up!!
/borg