Ok, have this seriously, insanely large XML file I’m parsing in my agent. I parse it, and then popluate the data to my LCDataProvider so anyone who wants some gets some.
The while loop is kind of slow as the XML is huge, and it’s pretty rude if someone goes to/launches another app while my agent is chugging away.
I then tried a process movie clip to cycle through each child of the XML using an onEnterFrame, and during each iteration, adding a small array to the LCDataProvider. However, the loop was slow enough to chug it to 1 fps about, and the DataGrid updated at the same rate… pretty lame even for a capped 4 items (hardcoded to test). I’m assuming this is because LCDataProvider is a syncronous LocalConnection plus the added overhead of modelChanged events getting thrown around.
So, went back to my for loop (while, faster, I know, but it’s late…), and it’s faster overall, just more CPU. Now I can see why “paging”, as it’s called is the way to go. Besides, the data I’m displaying, resource data based on type + planet + galaxy… no one’s gonna want to look at a 100 itemed DataGrid anyway.
People say don’t reivent the wheel, but I hate that. That’s the very reason I did so horrible in math in school… no teacher could explain why the hell I needed to know how to do this, nor why you did something the way you did it. Obviously, that shiot didn’t fly in Geometry or Trig. At any rate, by learning the hard way, I now have a great appreciation for paging, and will have to open up MX again to see how the RecordSet object does it for Flast Remoting, and hopefully, glean some applicational uses for the future.
<b>Tip</b>
Aside from my LCService mistake, another thing was I kept publishing Directly to the Central directory where it was installed. I forgot that the FLA saves that filepath you set in the File Publish settings, and I was uploading more than SWF’s (xml, php) to my server, so was wondering why everytime I reinstalled, I wasn’t seeing changes, until I put a text field on the stage, and didn’t see it… then I knew… 2 hours later. Just another gotcha to watch out for. Not to mention the fact that WS_FTP Lite wasn’t updating SWF’s that resided in a folder. …but hey, it’s free, right?
Are you in control over the xml format? If so you might think of changing it. Most of the time, people don’t set up their xml in a fashion that is quickly parsable. Also, If you don’t have control or even if you do, can you abstract the XML into a higher form. Most people forget that XML is natively a complex object, so why do you have to parse it. It is better to write something that accesses it.
Well, no, I didn’t make the XML; no control over it. The mug doesn’t even use CDATA tags which isn’t much of a deal cause most of the nodes are just text and numbers.
I’m actually only allowed to hit the XML file once per day, and then only really need it once per week considering even the smaller off-shoots of the big one are still beasts.
What I always do is convert my XML to an array of objects during parsing. In this case for Central, I’m doing the same thing but also saving to a local SO for later access.
To see what I’m dealing with:
http://www.swgcraft.com/xmlexport.php
DO NOT open the big one in IE… you have been warned.
Jesse,
You are approaching the problem wrong. You are using Central, and the XML is a file. Cache the XML file to local storage with addToLocalInternetCache. Don’t store it in a shared object, because you already have what you need. As far as datatypes, you don’t need to parse it into an Array, remember the XML is a complex object, you just need to think about it as such. Abstract the XML lookup by wrapping it into a shell that is the same API that the DataProvider is. For instance the planets has a childnodes array of planet nodes. You can do this._$list = this.firstChild.childNodes. so a getItemAt would read like return this._$list[i].attributes. you would also have a getListAt that read return this._$list[i].childNodes you could then assign this to a specific wrapper where getItemAt would may wrap all the node values for a resource into an object and return it. If you have to grab by key really fast aka swgcraft_id, you could do a mapOn routine like grant skinner wrote about on his blog. its quick and provides a lookup table. The primary thing is done parse into other structures. You already have your data. its just a matter of abstracting how you access it.
I have to deal with large amounts of data all the time. I have a huge library of stuff. I will be releasing some things in the future, just have to find some time. If you need help, give another post.
A quick note to the end of Kenny’s, not that it will likely help you :
the V2 dataGrid (and, in fact, all the list components) can take an Array of XMLNodes as .dataProvider – so in this case, you’d likely be able to just shove a childNodes array right into it.
Yea, I should have mentioned that ;). It depends on how you are wanting to grab things out. If I remember correctly assigning the childnodes to the dp will then have getItemAt retrieving the actual nodes. I cant remember how the MM datagrid does mapping of data to visuals. If you wrap it where it retrieves the attributes you will be grabbing out objects with properties and I think the grid handles that mapping (i could be and probably am wrong). Regardless, these types of abstractions above your xml will probably benefit you more than parsing the xml into a completely new structure.
Here is a proof of concept
var foo:XML = new XML(“”);
var myData:Object = {};
myData._$list = foo.firstChild.childNodes;
myData.getItemAt = function(i:Number):Object{
return this._$list[i].attributes;
}
datagrid_mc.dataProvider = myData;
hmm… xml didnt go through.. see if this does %3Ctest%3E%3Cinner%20label1%3D%27onelabel1%27%20label2%3D%27onelabel2%27%2F%3E%3Cinnner%20label1%3D%27twolabel1%27%20label2%3D%27twolabel2%27%2F%3E%3C%2Ftest%3E
<test><inner label1=’onelabel1′ label2=’onelabel2’/><innner label1=’twolabel1′ label2=’twolabel2’/></test>
Nice, great comments, ideas, and suggestions guys. The dataProvider via childNodes is phat, Nigel, I thought that was only an XFTree thang… co-rockin’!
Can’t wait to see your extension K.
However, I have some disagreements with some suggestions you made, Kenny, that I think we could probably hash out via discussion.
>>>You are approaching the problem wrong. You are using Central, and the XML is a file. Cache the XML file to local storage with addToLocalInternetCache. Don’t store it in a shared object, because you already have what you need.
The whole point of coverting the data from XML to an array of objects is this:
– object and array accessing is way faster than firstChild or childNode access both from my tests and from a comment Darron made about childNode accessing actually rebuilds the array (think I said that right, don’t quote me on what he said)
– secondly, I prefer:
my_array[2].name
vs.
this.firstChild.childNodes[0].attributes.name
Even though I could use a wrapper script… why? It’s more work, and again, slower.
I don’t cache the XML file because the conversion takes CPU and time. Saving my array of objects to a local SO is fast as well as reading it back out again and populating my objects. Additionally, I’m not updating the XML file, so there is no keep to keep it in it’s native format (such as adding nodes or whatever).
Finally, and probably the most opionated thing, is that I prefer seeing my simple array of objects vs viewing the XML. Both in code and in the debugger, it is a lot simpler to view and understand exactly what the data represents.
I’m hoping you’ll have some thoughts on the above.
Jesse,
You are correct, this.firstChild.childNodes[0].attributes.name is a slow look up if you are iterating through those values in that manner. You may have mistated your friend, it isnt rebuilding the array, it would be that you are using dynamic lookup and reevaluating that reference each time. The values of childNodes is actually referenced to a string lookup inside the player. So the deeper your path, the more its going to take to look up what each name aka property you are makiing references to. if you keep a pointer like list = this.firstChild.childNodes, then use list[i] you arent doing look up each time anymore. You have a stored point of lookup. In this case to pull off the properties you would only have to return list[i].attributes where this is your property. In this case it is exactly as if you had done list[i] where list was an array of objects. You have one extra property, attribute. If you are using getItemAt with an abstraction of both arrays in the datagrid you will be able to process this efficiently.
I’m not sure what this statement means > I don’t cache the XML file because the conversion takes CPU and time.Saving my array of objects to a local SO is fast as well as reading it back out again and populating my objects. Additionally, I’m not updating the XML file, so there is no keep to keep it in it’s native format (such as adding nodes or whatever).
How are you converting the XML? Just stuff it into the cache. Putting the file into your cache shouldnt take CPU power, you arent converting it, you are storing it via centrals capabilities. When you want to use it again, just load it from cache and assign the XML structure back to your wrapper. Assigning it to the wrapper is the only process and that is merely assigning foo = prop; that is minimal. You are saying it is faster to put it into a shared object? How? You have to write the routines to put it there. The fact that you are processing it all up front and as you stated.. this is your problem. You arent gona get around it. If you try to parse everything into an array of objects and you have no control over the complexity of your xml you are gona hit a bottleneck, because no matter how optimized your routines you arent gona be able to parse it up front as fast as you want it. If you were using C you would be fine :), but no Actionscript. So putting it into a SharedObject when you look at the overall processing is slower than writing a wrapper to the XML.
Now the debugger thing, to each his own. If you are down with the way it looks I cant argue with you there. I cant say gorillas look cooler than monkeys if you are really down with monkeys.
But whatever the case, do what you are most comfortable with. I just wanted to present a different way of lookin at the issue. If its not your cup of tea, its all good. I just find its really easy to use and is really fast. It may not make sense for you sense you load once every week, but if you were loading every minute, I would say don’t parse it up front you aren’t gona do anything but make your users mad. Jus my 2dollars :).
I do what you say in my conversion. I add multiple pointers, actually, for my loops.
The whole point of storing my array of objects in a local SharedObject prevents me having to load the XML in a wrapper; an XML wrapper is slow, accessing an array isn’t. The user takes a hit one time, and is done until the next week the XML file is updated. I don’t mean that caching the XML file is slow, I mean the parsing of the XML file to, what I consider, usable data.
The xml doesn’t change, I know the structure, and the parsing is actually pretty fast overall considering I’m only doing it once. I’m not writing an XML wrapper, I’m simply writing about 20 lines of code to extract what I need form the XML file. I’d do it differently for another XML file.
I mostly agree with your last statement, except for the users mad part. If I said they had to take a 5 second download + XML parsing + saving to local SO vs. an overall sluggish data extraction app, I’d bet they’d take the first. That is merely because I believe (my opinion, not fact) that using an XML wrapper is slower.
The only reason I’m pointing out my opinions as opinions is that you take a strong stance, and I’ve met you before, albeit breifly, and took you as being smart, therefore, I wanna know what you know.
How about this; go here and download my agent file, and take a look at my XML parsing code. No, I don’t save it to a local so yet as I my goal for that project was merely to learn, but adding that would just be:
so = SharedObject.get(“resources”);
so.data.my_array = my_array;
so.flush();
Where my_array is the array of objects I garner from the XML.
<a href=”https://www.jessewarden.com/SWGCraft/secretAgent.fla”>https://www.jessewarden.com/SWGCraft/secretAgent.fla</a>
Er, shite, that is the onEnterFrame one, use:
<a href=”https://www.jessewarden.com/SWGCraft/secretAgent2.fla”>https://www.jessewarden.com/SWGCraft/secretAgent2.fla</a>
Sorry…
The reason that I always parse XML into objects is because I find them much easier to work with, and they’re faster to access in my experience. I also don’t like the thought of having a huge xml object in memory, as it’s all text, and can get large quickly.
Consider a simple XML file:
some value
doh, my simple xml file didn’t show up — it looks like this:
<xml_node>
<option>some value</option>
</xml_node>
Laughin… took me as smart, hardly. 10 print “me smart == comedy”; 20 goto 10; run. I’ll check the file you posted in the morning. Darron, you have a point about memory allocation.
Jesse,
which are you wanting to parse the main feed or the individual feeds?
Well, just the individual as the main one is unrealistic as I don’t need all of that data nor does the user. They’d choose their feed (server name) anyway.
try this
function onDone(bool){
var dp, ps, i, p, pn, rs, i2, srs, r, sr, i3;
/*
[shortened names]
dp – dataprovider
ps – planets
i – length of planets
p – planet
pn – planet name
rs – resources
i2 – length of resources
srs – stored resources
r – resource
sr – stored resource
i3 – resource prop length
*/
if(bool){
dp = [];
ps = my_xml.firstChild.firstChild.childNodes;
i = ps.length;
while(i–){
p = ps[i];
pn = p.attributes.name;
rs = p.firstChild.childNodes;
i2 = rs.length;
srs = dp[i] = [];
while(i2–){
r = rs[i2].childNodes;
sr = srs[i2] = {};
sr.planet = pn;
i3 = r.length;
while(i3–){
sr[r[i3].nodeName] = r[i3].firstChild.toString();
}
}
}
this.swg_lcdp.setData(dp);
}else{
this.agent_service.onDebug(“xml error”);
}
}
it’s not the fastest in the world but it does what you want. since it takes around a half a second to parse on a 600mhz it might be wise to thread it, just enough that it doesnt grab your CPU in that time.
another bit of advice, since there are no decompression utils native to flash, it looks like you are going to be downloading the file once a week onto your own server and untarring it.. right? since you are running this chron job to go grab it and untar it and this process is once a week, instead of just writing it to a file, parse it into an easier xml file to read. You could also parse it into the datatype you want, serialize it, store it, and then pass it to flash using remoting as the native type you wanted it in the first place. However this second method using remoting will add overhead since you are doing processing each request. I would try the first method.
actually it takes nearly a second on a 600mhz. i was on a faster machine. so i would thread it into two parses probably. that is, if you arent going to preparse it on the server.
the test was actually in an older 6 player with no optimizations. so maybe it is faster. laughin. i dont know i put my clothes on backwards this morning, forgot my belt and had to use a shoestring, etc.
ok,
this runs in under around 70 milliseconds without player optimizations and doesnt lock the browser. if you wana parse it straight out try this.
function onDone(bool){
var dp, ps, i, p, pn, rs, i2, srs, r, sr, i3;
/*
[shortened names]
dp – dataprovider
ps – planets
i – iterator of planets
p – planet
pn – planet name
rs – resources
i2 – iterator of resources
srs – stored resources
r – resource
sr – stored resource
i3 – iterator of resource prop
*/
if(bool){
dp = [];
ps = my_xml.firstChild.firstChild.childNodes;
for(i in ps){
p = ps[i];
pn = p.attributes.name;
rs = p.firstChild.childNodes;
srs = dp[i] = [];
i2 = rs.length % 8;
while(i2–){
r = rs[i2].childNodes;
sr = srs[i2] = {};
sr.planet = pn;
for(i3 in r){
sr[r[i3].nodeName] = r[i3].firstChild.toString();
}
}
i2 = parseInt(rs.length/8);
while(i2–){
r = rs[i2].childNodes;
sr = srs[i2] = {};
sr.planet = pn;
for(i3 in r){
sr[r[i3].nodeName] = r[i3].firstChild.toString();
}
}
}
this.swg_lcdp.setData(dp);
}else{
this.agent_service.onDebug(“xml error”);
}
}
totally forgot the increment iterator for the loop on the last post. with it, it actually goes slow.