Greetings all,

I'm experimenting with Poser Pro 2012's network render queue and I'd like to see if I can utilise the cloud to boost my rendering horse power when I need it.

So far I've successfully setup netowrk rendering between two physical machines on my network.

Now I'm trying to use an EC2 instance in the Amazon Web Services (AWS) Cloud to host another Queue Manager instance and have it pick up render jobs from my local machine.

Thus far I've got a VPN connection back from the EC2 instance back to my master render machine, but the EC2 instance still doesn't "hear" any of the render requests being posted to the queue.

I'm still in the process of determining whether or not any communication is happening at all, the VPN connection states 0 bytes TX/RX which I find suspicious :P

• Has anyone had any luck hosting Queue Managers in the Cloud?
• Is VPN the best way to go or is there a better way to get my remote Queue Manager to pick up my render requests (SSH tunnel perhaps)?

I'll update this as I discover new things.

 

~ Ryan

I tried once to set up a 3-station render farm. My own experience was that it helped doing animations, but not single frames. Someone correct me if I am wrong... I don't think the software breaks a single image up into pieces and distributes processing on those pieces. As it is, doing single frames as I do, I see no advantage in distributed computing as it is implemented.

Thanks for your reply!

And you're right I did a test for a single frame render and it didn't distribute the frame accross the Queue Manager instances.

I too deal mostly in single frame renders, so I suppose this just makes it an academic experiment then.

Still I would be interested to see the difference in render time using a single powerful cloud instance (as in their clustered GPU AMI) vs rendering locally.

Another advantage to rendering to the cloud could be just for the hardware upgrades. My workstation right now is decent enough, but as a hobbyist I'd rather offload the cost of upgrading my rendering power to the Cloud rather than investing in new production rendering hardware myself. 

I'm still gonna see what it takes to get Queue Manager up in the Cloud :)

~ Ryan

Hi Syrus_BD. This sounds like an interesting project for sure.

I guess the first thing I'd suspect would be firewall / router related issues. Perhaps the necessary ports are being blocked or not being forwarded?

Depending on how your VPN is set up, port blocking / or  non-forwarding of ports on inbound / outbound traffic can still occur... even with the tunnelling. What kind of VPN are you using?

Further to that, perhaps bandwidth?

Maybe the queue manager needs a minimum bandwidth for its network communication to work?

My wish for PoserPro: A Linux based QueueManager with text-based screen interface running on a Linux core with just the minimum facilities. No bit of memory spent on anything but rendering, rendering and rendering. A new life for the old desktop now dusting away in my attic, or as a live system-CD on which to boot on an unused laptop.

Quote - My wish for PoserPro: A Linux based QueueManager with text-based screen interface running on a Linux core with just the minimum facilities. No bit of memory spent on anything but rendering, rendering and rendering. A new life for the old desktop now dusting away in my attic, or as a live system-CD on which to boot on an unused laptop.

Second that ;-)

I'm not sure but I think Poser and QM find each other via UDP broadcasts.

I am sure that almost all VPN software will not forward UDP traffic - it remains strictly local.

This is a problem for LAN based games as well. Perhaps this tool, which works for some games, can work for Poser and QM as well.

http://www.morpheussoftware.net/git/

---

Renderosity forum reply notifications are wonky. If I read a follow-up in a thread, but I don't myself reply, then notifications no longer happen AT ALL on that thread. So if I seem to be ignoring a question, that's why. (Updated September 23, 2019)

BB-

Am I right or wrong about Poser not breaking up single images for frangmentary rendering, making it useless for single frame work?

Interesting, I had to fiddle around with my router settings in order to properly forward VPN traffic and the Firewall Settings had options for NAT Endpoint Filtering.

Currently I have:

• UDP Endpoint Filtering as "Endpoint Independent"
• TCP Endpoint Filtering as "Address Restricted"

To be honest I have no idea what any of that means, but if Poser and QM ord communicating via UDP then I guess I'll have to get into it :)

Thanks for the replies everyone, very helpful.

That Gamer's Internet Tunnel looks like a good bet. Complicated firewall config otherwise... which is only possible with higher end network kit too often.

If you needed to try to figure out what UDP ports (etc) may be in use (e.g. if this isn't in fact documented somewhere by SM) then you could try running Queue Manager between two machines on your LAN and hook up a packet sniffer, such as Ethereal (open source)...

http://www.ethereal.com

...there's some learning curve in doing that of course, if it's not something you already know about :-)

Quote - Am I right or wrong about Poser not breaking up single images for frangmentary rendering, making it useless for single frame work?

Well not useless because you can use Poser, or the computer you use Poser on, for other tasks. If you know your scene is ready you can make that final 4000x4000 pixel version on another machine. The queue manager has a separate license which, a far as I remember, can be applied on muliple machines in your household. So you can use the CPU capacity of all computers in the house to render the scenes you make on one machine with PoserPro license. 

This is what I thought. The scenario you  propose might have its uses, but not for me. It would take the software subdividing each image and then putting the pieces back together for it to be of value in my workflow, and I suspect that would be a pretty tall programming order.

Not that I know the first thing about programming render engines, but FireFly already seems to split up the image into blocks for parallel rendering. I have eight cores going and at any given time eight block of a frame are being rendered at once, one per core.

I don't think it's that much of a stretch to consider further splitting those "blocks" out amongst the network.

But time will tell. I'm sure the programmers have thought about it. After all cloud computing seem to be the way it's going now; cloud rendering seems a logical progression to me.

Interesting point about the multicore rendering...

There are plenty of cloud based render farms for the likes of Maya, 3ds Max and Cinema4d...

Vue have recently released a Linux version of their network rendering system... I'm guessing because this is sought after by render farm providers and bigger CGI production companies... so who knows, it may be only a matter of time before Poser follows suite?

I could imagine that if the consumer cost could be kept low enough (e.g. by large membership community sites negotiating bulk pricing from render farm providers) then some kind of pay as you go cloud rendering, of more resource intensive Poser scenes and animations, could prove quite attractive to a lot of professional and hobbyist Poser users alike...

I can confirm that Poser cannot distribute individual render buckets over the net (only Bryce does that, IMO, and mojoWorld could do a limited bit that). Similary, Poser does not split up animations within a CPU, it's one frame at a time, not frames in parallel. So: long lasting renders and especially animations can be forked out, as long as animations render to separate images. Personally I run them as a QM in background low priority job utilizing CPU idle time.

The Queue manager says it uses ports 4418 and 11523 to communicate. I guess the EC2 machine act on those ports, and an active QM at that side should do (part of the) the job. I'm not sure UDP supports that, the EC2 machine just should act as another machine in the local / render network. Interesting though, keep posting please.

Given the rates, buying another high CPU high RAM (low disk low video) machine seems to be more cost-effective than cloud rendering. As it stands right now.
Point is: there is not only a usage fee per minute or so, but a monthly fee as well.

- - - - - 

Usually I'm wrong. But to be effective and efficient, I don't need to be correct or accurate.

visit www.aRtBeeWeb.nl (works) or Missing Manuals (tutorials & reviews) - both need an update though

Quote - This is what I thought. The scenario you  propose might have its uses, but not for me. It would take the software subdividing each image and then putting the pieces back together for it to be of value in my workflow, and I suspect that would be a pretty tall programming order.

You can make it partial render jobs and merge the result in Photoshop.

As bagginsbill stated, poser and its queue manager find each other using UDP broadcast. Your problem, however, is not the UDP, this should be possible with most VPNs, it's the broadcast that gets in the way. You probably tried network rendering in a LAN using a common subnet. For example your machines have addresses 192.168.0.1 and 192.168.0.2; poser does a broadcast at 192.168.255.255, which each machine gets. When using an EC2 instance, it is neither in your local network, nor does it lie in a similar address range. So to get the data from poser to the QM in EC2, you have to do two things: First, the machines need to use the same address range (like 192.168.0.0/16 for example) and you have to create a path from your local network to amazon. The first one could be done by assigning a network address to the EC2 instance that matches your local network address (you must make this an additional address, not replace the old one or you will not be able to talk to your EC2 anymore). The second (creating a virtual local network) should be done with the VPN software you use; you have to tell it somehow to pass broadcasts through to the other network. This can be a little bit difficult depending on your VPN software you use. You should look within the options of the VPN software for broadcast forwarding. Easiest is probably when your VPN supports something called "ethernet bridging", or "802.3 tunneling" or something like that. It means that the VPN acts just like an additional network card for the computer. Note that i tried nothing of this with Poser, but i think this is what you should be looking for.

@millighost

Some good information there, though I'm not sure I fully understand all of it :) 

The VPN I've got setup is through the basic Windows 7 RAS (Dial In) Interface. All I did to do that was go to "Network Sharing Center" and setup a new incoming connection with a user.

On the EC2 server I setup the connection to VPN into my home router.

The router is setup to forward TCP port 1723 and protocol 47 (GRE) to my host Poser workstation, apparently this is required for VPN connectivity.

After reading your post I changed the range of addresses that the Poser workstation assigns incoming addresses to be within a dedicated range of the IP's the router assigns. 

So... to summarize:

• Poser workstation has a local network IP of 192.168.0.102 (subnet mask 255.255.255.0)
• Poser workstation RAS interface has an IP of 192.168.0.200 (subnet mask 255.255.255.255)
• EC2 Render Node has a VPN assigned IP of 192.168.0.201 (subnet mask 255.255.255.255)

I then changed a setting on my router to Broadcast NetBIOS which I think is supposed to allow computers to discover each other but I really don't know I'm still so new to this. It didn't allow the two commputers to find each other anyway.

I think I've got part one of your post right, where the IP addresses are in a similar address range. The VPN assigns addresses from 200 - 209 so as to ensure no duplication with actual IP's on the router 100- 199... but I'm not sure if that's kosher or even right.

Part two however I'm not sure of. To create a path from my local network to amazon I was thinking of using the "Routing" section on my router setup. It offers the following options for specifying routes:

• Name
• Destination IP
• Netmask
• Gateway 
• Metric

Name and Metric I'm pretty sure are immaterial. Would I be able to setup a path to the Amazon EC2 server using this type of routing? Would it be more proper just to get a VPN NIC for my workstation or am I fine going through Windows 7? Would I be able to consider an SSH Tunnel (say via putty) or is that not the type of tunneling I need/can use?

This is looking more and more like a VPN for dummies thread than a Poser thread; still if the interest is there I'd like to keep at this.

Thanks for all your replies!

yes, the interest is there (well, here at least).

Forking out to external resources can save me upgrading for performance reasons, it might apply to Vue as well. Please keep posting.

- - - - - 

Usually I'm wrong. But to be effective and efficient, I don't need to be correct or accurate.

visit www.aRtBeeWeb.nl (works) or Missing Manuals (tutorials & reviews) - both need an update though

cough Luxrender does distributed rendering for single images. I've made use of it....

Hi Syrus_BD

Netbios is something different. Legacy ms windows protocol, principally.

What millighost was referring to was tcpip broadcast.

I think investigating that gamer tunnel software might be a good bet, over the built in Windows VPN.

If I have a bit more time next week, I will maybe try putting the Win QM and FFRender onto one of my remote servers and have a go at this myself.

Trouble is that any remote kit I have available to test this with is all running either Win 2008 or Linux. But my home workstation is mac.

Not sure if QM works across different operating systems?

In terms of a vpn nic (or a more advanced firewall router device that handles VPN) that would only help, I think, if you could mirror that setup at the ec2 side, somehow... don't know enough about ec2 options to comment there?

@Khai-J-Bach: correct indeed, I overlooked that because it's a completely different principle, based on lights / ligthing levels instead of buckets as Poser, Bryce, ... But from an end user point of view: welcome addition.

- - - - - 

Usually I'm wrong. But to be effective and efficient, I don't need to be correct or accurate.

visit www.aRtBeeWeb.nl (works) or Missing Manuals (tutorials & reviews) - both need an update though

Quote - Hi Syrus_BD

Netbios is something different. Legacy ms windows protocol, principally.

What millighost was referring to was tcpip broadcast.

UDP broadcast, not tcpip.

I wouldn't normally write about this, but we have a user trying to figure out what settings to adjust and it's important that it be clear we're talking about UDP, not TCP. Both UDP and TCP go over IP (Internet Protocol). TCP is never broadcast and is used for point-to-point streams of communication. UDP is broadcast or unicast. Broadcast is often used for asking a question when you don't know who you're asking. In this case the question is "Who here is running Poser QM"? The answer would then be unicast.

Many routers route unicast UDP messages. (Unicast means addressed to a specific device, not a group of devices.) Most routers do not route broadcast (to everybody) or multicast (to a group) unless they are configured to do so.

Both UDP and TCP have two-level addressing. There is the IP address of the device, and there is the "port number". The port number usually (but not always) has to do with a specific well-known piece of software within a device. For example, TCP port 80 is almost always a web server. UDP port 137 and 138 are for the NETBIOS service. Enabling NETBIOS forwarding does not mean you have turned on all UDP broadcast forwarding. It only forwards if the port is 137 or 138.

---

Renderosity forum reply notifications are wonky. If I read a follow-up in a thread, but I don't myself reply, then notifications no longer happen AT ALL on that thread. So if I seem to be ignoring a question, that's why. (Updated September 23, 2019)

Oops! Sorry... my bad... I meant ip broadcast... as BB says. all important of course.

I will perhaps blame the type-ahead on my iphone for the erroneous insertion of "tcp" there ;-)

Thanks for the responses everyone!

As for the Gamer's Internet Tunnel it doesn't seem to be supported in Windows 7, and when I run it I don't get any options under the "Look at frames on which device?" I'm continuing to look for other software that provide a similar service.

I've seen some posts around UDP broadcasts, one in particular mentions adding a "rule" to allow broadcast traffic through. I'll see if I can pick the brain of my network guy here at work as well (cursed day time job getting in the way of my hobbies).

Taking the use of work resources for personal use to a whole new level :)

Asking your network guy sounds like a plan ;-)

I have some experience setting up PPTP client VPNs powered by Microsoft Threat Management Gateway (formerly ISA Server) and also site to site tunnels using Cisco devices.

Both have required mutiple inbound and outbound rules of one sort or another, to be manually added... and various out of the box parameters to be tweaked... in order to allow various different network services to work properly.

Obviously you have to be very careful, if adding firewall rules, that you don't accidentally open up your internal network to intruders...

 This is for Max but maybe there's some nugget about networking setup.

http://www.judpratt.com/tutorials/ec2-renderfarm/

"Democracy is a pathetic belief in the collective wisdom of individual ignorance." - H. L. Mencken

When running, my QueueManager opens the following ports.
I may assume they are used by both the PoserWorkstation and the EC2, and that the VPN must permit these kinds of traffic, both ways:
tcp 11527, tcp 4418, udp 11523, udp 11423.

(Earlier posters have confirmed that Routers won't forward Broadcast UDP traffic. Otherwise, a destination of 255.255.255.255 would send the broadcast to every IP device on the planet.)

The use of QueueManager over different networks (i.e. 'over the internet', 'in the cloud') would be made easier if it provided a means to nominate the addresses of the slave renderers; thus negating any need to use UDP.

Additional comment:
Although I have only casually looked at the networking side of Poser, I note that TCP/11527 appears to be interpreted as an incoming job.
e.g:   TELNET 127.0.0.1 11527  (then type something), then look at the QM panel.

Thanks again for the suggestions everyone, I've decided to try another VPN solution since Windows 7 VPN doesn't seem to offer me enough control over its connection.

@lmckenzie: thanks for posting the MAX tutorial, it's all videos so I'll have to go through it when I get back home.

@timarender: interesting tidbits about both TCP and UDP ports being used, good information to know if I end up having to forward those ports though whatever solution I get working; and of course more knowledge about the QM communication workings on the whole is a good thing.

I did find some posts on SSH tunneling but all they do is turn UDP packets into TCP packets and back again; this would only route render jobs to a single destination rather than broadcasting to what could potentially be a farm of EC2 instances.

I'm gonna try using OpenVPN, it looks promising because it has a config file I can modify that has a lot of granularity as to how the connection is setup, including subnets to broadcast to and bridging capability to allow those broadcasts through the VPN.

More to come...

Openvpn is an possible solution for the problem of getting the broadcast packets through the VPN, because it comes with a tap device driver. Here is in short what i did to make it work:

1. i used the Microsoft Windows Server 2008 Base image (micro instance) for this experiment.
2. install the the openvpn software from openvpn.net, found here:
     http://openvpn.net/index.php/download.html
3. in the security groups settings from amazon's ec2 console add an inbound rule for TCP port 1194 (which is the Openvpn port)
4. start openvpn on the ec2 instance in passive (server) mode:
     openvpn --proto tcp-server --dev tap --ifconfig 10.4.0.2 255.255.0.0
5. start openvpn on your local machine and connect it to the ec2 instance:
     openvpn --proto tcp-client --dev tap --ifconfig 10.4.0.1 255.255.0.0

So far this gets me an additional network interface which connects my local computer with the ec2 instance. I use 10.4.0.0/16 as a netmask for the VPN which is seperate from my LAN. The local machine gets 10.4.0.1 and the ec2 instance gets 10.4.0.2 as a virtual address. Broadcast traffic to 10.4.255.255 is passed through the virtual interface.

If you use the windows firewall (by default enabled), you probably need to add exception rules to allow traffic on UDP port 11423 and various TCP ports (among them 11523-11528 and 4418) to allow Poser and the QueueManager to talk to each other. I simply disabled the firewall for the virtual interface, which is no problem because nobody else uses my LAN.

I did not use encryption or authorization with OpenVPN, you probably should use it in real life, but for trying it out this was easier.

Update: Moderate success.

I've got the EC2 instance connected to my local network via OpenVPN. I can ping the EC2 instance via the VPN and the EC2 instance can ping my Poser Workstation.

By the way I owe you all a huge thanks for your support and contributions, millighost in particular gave me a key piece of information about making sure the VPN IP's were on the same IP range (subnet?) as my LAN addresses and to look at ethernet bridging.

Now here's the "moderate" part of my success thus far: The EC2 Queue Manager picks up the job posted by the workstation (great!) but the objects it has to transport up to the EC2 instance seem to be taking quite a while and this is not a complex scene. 

I looked at the connection on the EC2 instance and it says it's a 10Mbps VPN connection, and in reality my upload rate with my ISP is only ~2.5MBps (which I guess equates to 20Mbps).

However it's only reporting ~5% usage of the 10Mbps link so that's a little suspicious to me. Maybe it's just because I'm running the Free Tier and Amazon is throttling even that bandwidth. I'll look into that.

Complaints about upload speed, however, are a familiar line from the people I talk to about the Cloud. The ISP upload speed is now the bottle neck for this project. I'm gonna talk to my ISP about this to see what options I have available to me, but at this point this has more to do with AWS/ISP file transfer efficiency than with Poser/QM.

Queue Manager is up in the cloud and taking jobs. The next step is to find out how efficient one of the Cluster GPU instances at rendering scenes, but that'll have to wait until I resolve the data transfer issue because I'm not gonna spin up an hour of time on a Cluster GPU AMI for 59 minutes of file transfer and 1 minute of rendering :)

I'll post again with more updates (and possibly a guide to this point when I get a chance to write one out).

Well done getting so far already ;-)

Sounds like the bandwidth is going to be your next big challenge though!

20 Mbps upload sounds pretty reasonable though by most standards... if that is actually what you're getting.

What country are you in / what type of internet connection are you on? e.g. DSL over copper phone lines, leased line or fibre-optic?

@millighost: Looks like you and I got to the same point now :) though for my part I didn't need to add the inbound rule on the ec2 instance, probably because I'm using my local workstation as the VPN server not the EC2 instance.

I just had a chat with a colleague here and he suggested that my problem with upload speed might be side stepped if my workstation itself was in the cloud, then all I'd be dealing with is latency between cloud instances. The problem is I'd have to pay to rent storage space in the cloud for all my Poser content

I'll keep at it from different angles to see what the most cost/performance efficient setup would be.

@monkeycloud: I'm in Canada and to be honest I'm not sure what the lines are made of, though I'm on Cable internet (which is a COAX cable into my house, so I guess that's copper based). 

Ah, I see - I guess probably copper... but cable TV type coax rather than telephone wire, so yeah you'll potentially get up to 20Mbps upload, from what I know ;-)

I'd be lucky to get 2Mbps upload on my domestic broadband here :-(

But they are in the process of rolling out fibre optic though... :-)

The suggestion of also running your workstation in the cloud and just remoting to that is a good one though!

The limitation there, if there is one, will maybe be graphics presentation to your local PC... although I'm not that up to speed on that subject. I only use remote desktop stuff in my day job for low graphics requirements business apps... not any programs that actually need good end user display quality.

@monkeycloud: Agreed on the visual presentation of the View Ports. I generally find RDP, VNC or just any remote destop software painful when it comes to refreshing complex graphics.

Real-time rendering has come a long way with regards to View Port manipulation; when I started out with 3D rendering I always found that inputting parameters directly into a command line (Auto CAD still uses this as a central feature) was more efficient than using the mouse to visually analyze and manipulate the scene.

A workstation up in the cloud would force me back just about 20 years with regards to that (wow I'm old...lol).

But I'll try it out just to see if it's at all workable, I'll just have to use old tricks like using the lit wire frame view instead of loading all of the textures.

I guess maybe what is needed perhaps for Poser cloud rendering, over limited bandwidth, using queue manager is a little software render helper agent to sit on your rendering machine, along with a web server (Apache or IIS) running a web service.

A python script in Poser, on your workstation could upload the render job as a temporary .pz3 with an additional XML file, containing the render settings, uploading it over http protocol, to the web service, that would accept those files, and then place them in a monitored directory... or at least in some way, handing them over to the render helper agent, which would in turn talk to Queue Manager.... instead of Poser, directly... if you catch my drift.

That latter bit, talking to Queue Manager, is where it probably gets trickier I suppose... and there's perhaps asynchronous event handling needed in there too I guess, more than likely... to keep the originating instance of Poser updated as to progress.

This render agent idea is maybe getting a bit convoluted... if it wasn't nonsense to begin with.

Of course if either Queue Manager or indeed FFRender itself had a python api, that could cut out at least one of those hypothetical software middle men... I don't know, maybe they do???

:-o

Is it telling that when you said 20 years ago, I initially thought 1982... before realising that was now 30 years ago!

Actually, I'm not that old... it's just that I seem to have misplaced 10 years somewhere ;-)

Yeah...

Suppose what might make more sense actually is probably have an instance of Poser in the cloud and automate that, with its API?

That instance of Poser would then be the controller for the Queue Manager based render farm, all running within your cloud based LAN?

:-)

@monkeycloud: That's a neat idea about using the Poser API (another interesting project to get into), but we're still bottle necked by the sheer size of the actual poser files. Most of mine are moderate single frame scenes in the 200's (MB).

However... having said that we could setup a low grade Poser Request Server, as it were, to handle the transfer of the save file, then automatically kick off another Cloud Cluster GPU instance once the file is uploaded using the Request Server Poser interface to post to the Cloud LAN QM and have the Cluster GPU QM pick it up for processing (and of course shut it down once done processing its queue). That would optmize the time/cost of the machines that are actually running.

It still doesn't address the data transfer issue but it's a start :)

200 MB is quite a lot... but not, I don't think, totally horrendous as a straight forward upload... especially if you can manage 20 Mbps upload bandwidth... and increasingly that should be achievable for most people... especially with fibre rollouts and so forth.

Okay still probably near enough a minimum of a couple of minutes or more, I guess to upload 200 MB, allowing for fluctuations?

But, the issues your having with bandwidth over the VPN, I suspect, in a less-than-educated guess, are maybe due to more cross talk going on... or indeed, just that QM is not optimised for lower bandwidth situations and is "flatlining" in some way, when it doesn't get as much as it expects, quickly enough...

 

Correction to my above post: my upload is 2.5Mbps (up from my previous 1.5Mbps). Should have caught on to it when I calculated 20Mbps.

Sorry!

He he. More like 20 mins or more upload time then I guess for 200 MB ;-)

Which is about the max I'd get at the moment too.

Even at that though... if you could run that with a background intelligent transfer service type upload, up to your cloud render farm...

Plenty of options available I'm sure for low bandwidth background upload of large files I think.

Quote - Am I right or wrong about Poser not breaking up single images for frangmentary rendering, making it useless for single frame work?

Hello,

Carrara Pro can do that (splitting a single image render to different nodes).  However I doubt that this would give significant speedup with remote hosts like in the cloud, because of the large amount of network traffic. Maybe with a sufficiently high tile / bucket size.  It works quite well in a local network.

Best regards,

   Michael

 

 

Greetings all again.

I've been thinking about how to resolve this bandwidth issue and I've decided to try coming at it from a different perspective:

1. Create/compose scenes on a local workstation, put all save files in a standard directory or subfolders of a standard directory

2. Use RSync to monitor said standard directory and post updates to a mirror in the Cloud

3. Spin up a low cost AWS Micro instance as a master Queue Manager and use a Poser or QM API to automatically post Poser renderjobs to the Queue.

4. Have the Master Queue Manager Instance spin up one or more AWS Cluster GPU Cloud Machines to render the jobs in the Queue.

5. Have the Master Queuer Manager Instance shut down said Cluster GPU instances when render jobs are done.

6. Have the Master Queue Manager shut itself down once all jobs have been processed.

In Step 3 you could automatically do this every night or on demand.

In Step 4 you could spin up 1 to n instances depending on how much you're willing to spend.

Step 3&4 are the big questions: Does anyone know if there is such a thing as a Poser or QM API that will allow me to programtically post/monitor or generally manage render jobs in QM's Queue?

I'm going to ask Smith Mirco this question as well but I thought I post my thoughts here as well for community interest.

 

~ Ryan

Hi Syrus_BD

I'd agree this sounds like the kind of way to go... I'm guessing that a Python based agent could automate the Poser API to open up each rsync'd scene file and submit it to the render queue?

The only other issue you've got though is how to communicate any alterations in render settings you might want to make on your local composer workstation, up to the cloud based and your automated Poser instance.

I'm guessing there'll be a way, using Poser's python API, to capture the render settings on the local workstation... as this is obviously used in some of the supplied scripts, for advanced rendering etc?

So you maybe want a python script, on the local workstation, to deal with saving your scene to the rsync folder, along with current render settings as a separate xml file (also to be rsync'd).

Then your Poser API automation script on the cloud controller instance would read in those render settings from the xml after it opens up the scene, and apply them to the current Poser session before submitting the job to your cloud based QM render farm.

There may be a more elegant way than this still, of course. But as I don't think render settings are stored in scene files at all (?) then I think a means of passing on these is also necessary...?

;-)

I'm pretty sure render settings are saved per scene file, I get different render dimmensions and settings when I open up different files. That said, however, being able to programtically set render settings would be a nice nod to the whole DEV -> Test - > Production work flow.

Suppose you just want a small render on your local machine just to test what the final output might look like before you spin up your Cloud Machines, I could see merrit in that. Then the Cloud machines would render at your "Production" level render settings.

Is there any documentation on Python scripting for Poser? Can I excute a python script outside of poser itself? Python scripting is also a new area for me.

There is a section in the manual on Poser Python Scripting, and yes, you can execute a script outside of Poser.

@hborre: Thanks for the reply, I'll look into the manual and experiment with it tonight.

There is a dialog script in Poser that is an alternative to the built-in render settings dialog. Many of us use this script because it exposes some things that are otherwise hidden or automated in ways we don't want.

It's in Scripts/Partners/Dimension3D/Render Firefly.py (I think - I don't have Poser with me)

I point you to this script not because it does what you want, but because it demonstrates how to set every single thing in render settings via Python, including dimensions.

There is a PDF included with Poser that is the Poser Python reference manual. It describes the API that Poser adds to standard Python to allow you to do Poser-specific things in Poser Python.

Python is a standalone language and you can run scripts from the generic Python interpreter but it will not have any access to Poser-specific things, like render settings. You have to run Poser python scripts inside Poser for it to be able to do things like alter scenes in real time.

It is possible using standard Python to read and write Poser files, without being inside the Poser Python engine, but this is a pretty extensive bit of work to take on. For example, my matmatic script can be run in standard Python because it doesn't actually talk to the Poser API - it just reads and writes files. Specifically, it constructs material files which can then be loaded into Poser. But the act of creating these files is all done with code I wrote for generic Python.

---

Renderosity forum reply notifications are wonky. If I read a follow-up in a thread, but I don't myself reply, then notifications no longer happen AT ALL on that thread. So if I seem to be ignoring a question, that's why. (Updated September 23, 2019)

Quote - Is there any documentation on Python scripting for Poser? Can I excute a python script outside of poser itself? Python scripting is also a new area for me.

Here's a link to the Python methods manual:

https://www.smithmicro.com:4333/support/faq-graphics/downloads/manuals/Poser_9_Python_Methods_Manual.pdf

This is an entirely new area for me...

But on a quick glance through above linked manual I note methods relative to a "startup" script being associated with a document (which could be your default scene). You might be able to do something with that, as an alternative to external API manipulation too...

...but on the basis that Vue and various other programs integrate with Poser's API somehow, external integration must be possible I think. Don't know if that is via Python though...??

PhilC has a python scripting manual for sale here and at his site too:

http://www.renderosity.com/mod/bcs/python-for-poser/86870

I have this in my wishlist already...

Plenty more folk here... and at Poser Place, if you've not already registered there too... who can give you the benefit of their knowledge in the area of python scripting for Poser I think though ;-)

@bagginsbill: So Poser specific actions requested via Python scripting like "Render in Queue" would have to be executed while Poser is open I take it.

That's not a huge deal seeing as the Poer installation on the Master Queue Manager would be dedicated solely to the purpose ofr posting jobs to a queue, not actual scene compositions. I could setup the Python script to execute every time poser starts and then start poser up when I launch the Machine instance.

I suppose Poser would have to open up every file and compile all of the lighting and object pre-processing information for the Queue anyway.

One addition to my previous post would be that RSync would alco have to monitor any content directories I have since the Master Queue Manager instance would need those files to compile render data. 

I could always host my content in the cloud to begin with as an option, we'll see.