Thoughts about future releases, in no particular order.
Feel free to share your thoughts with me at mob@cpan.org
or through http://rt.cpan.org/NoAuth/Bugs.html?Dist=Forks-Super
Possible TODOs:
_x_ support sockets/pipes for cmd-style forks like IPC::Open3 does.
_X_ for Unixy systems
___ for MSWin32
IPC::Open3 supports MSWin32/os2 with the <system 1, @_>
construction. Don't think that is appropriate for this module.
___ There's enough stuff in here now that performance is affected,
especially when there are lots of short tasks. What can be done
to tighten up performance? What can be done to reduce overhead
when there are many short tasks?
___ disable $FSJ::Ipc::USE_TIE_?H? need to benchmark
_X_ tightened some delays in file-based IPC
___ shorten $Forks::Super::Queue::QUEUE_MONITOR_FREQ
___ fork { run => [ \@cmd, ... ] }
to invoke or emulate IPC::Run's rich feature set
___ What else can I learn by studying IPC::Run?
___ What other key-value pairs should FSJ::read_stdxxx handle?
_X_ warn => 0|1
_X_ timeout => max number of seconds to wait for input
___ log => 0|1|*handle|\@list?
___ No complaints yet, but is there a smarter way to
set the IPC directory?
___ Refactor how we create and remove the temp IPC directory.
___ Runtime IPC cleanup routine for long running programs. For long
completed jobs: close the open filehandles that have slipped
through the cracks; remove the IPC files; move from
%ALL_JOBS,@ALL_JOBS to %ARCHIVED_JOBS,@ARCHIVED_JOBS
___ Demos/examples/cookbook section.
___ Perform 1000's of jobs, 20 at a time
___ with queueing to perform other tasks
___ example: web crawler
_x_ example: multi-threaded du
_x_ timeout long running jobs
_x_ manipulate CPU affinities
___ dependencies
___ interactive client/server example of IPC
___ run a server using Forks::Super
_x_ see t/forked_harness.pl
_X_ load management
_X_ block while system is busy
_X_ suspend/resume
_X_ suspend/resume callback
___ bg_eval, bg_qx examples
_X_ factorial.pl for bg_eval
___ can_launch examples
___ how to: use sleep with Forks::Super
___ how to: use alarm with Forks::Super
___ changing IPC_DIR
___ tuning Forks::Super for fast jobs, slow jobs,
memory intensive jobs, cpu intensive jobs,
I/O bound jobs
___ scheduler app that can run for days at a time
___ reuse
___ share
___ daemon
_x_ Forks::Super::Job::dispose( @jobs ) method
_X_ Removes entry from @ALL_JOBS, %ALL_JOBS
___ move to @ARCHIVED_JOBS, %ARCHIVED_JOBS?
___ POSIX::RT::Timer as possible replacement for get/set itimer?
_X_ Forks::Super::Job can overload "<>" (iteration) operator,
invoking $job->read_stdout(). Beware of overload bug, though:
"Even in list context, the iterator is currently called only
once and with scalar context."
_x_ Actually, maybe it can't. Can't define even the buggy
version of Forks::Super::Job::(<> except with some
crazy machinations.
_O_ Understand why Windows hack needed for socket write (see
FSJ::Ipc::write_stdin, FST::IPCSocketHandle::trivial_pause).
Is there a better solution?
___ is it needed ? removed in v0.55 but MSWin32 tests pass now
___ Does anything bad happen when you set $SIG{CHLD} = 'IGNORE' ?
'DEFAULT'? sub {} ? undef ?
_X_ Yes. ANY setting for $SIG{CHLD} will let system calls
like sleep get interrupted by SIGCHLD events. Maybe.
See t/31.
It's curious that we can assign to $XSIG{CHLD}[-1]
(which does set a handler for SIGCHLD) but that DOESN'T
trigger interruption of system calls. It's as if perl
is just checking defined $SIG{CHLD} instead of whether
an actual signal handler is registered to decide whether
to interrupt sleep.
___ Setting $SIG{CHLD}=undef makes FS subtly wrong, though
setting $SIG{CHLD}=\&bogus is ok. How to keep $SIG{CHLD}
from getting undefined?
` ___ Override $Signals::XSIG::SIGTie::STORE, ::DELETE ?
___ Should we periodically set $SIG{CHLD}=\&garbage?
___ Yes again. When $SIG{CHLD}='IGNORE', calling wait or waitpid
NEVER returns a pid; it's always either 0 or -1. See
t/drop-in-exercise.pl (actually, this is platform dependent).
___ Should Forks::Super check the value of $SIG{CHLD}
and emulate this behavior?
___ Actually, this would be part of a good XSIG framework workout.
Some scripts with natural forks and wait/waitpid calls
should produce the same results with and without Forks::Super
(i.e., F::S is a drop-in replacement)
_X_ Test F::S as a drop in replacement to a program that
has a SIGCHLD handler. -- looks good
___ Prior to v0.52, bg_eval and bg_qx could be called in list context
but they wouldn't return control to foreground process. Can this
be fixed?
___ is _trap_signals call in FSJ::Ipc::_config_fh_parent necessary?
___ was removed in 0.53, so we'll see
_X_ FSJ::Ipc::_close, _close_socket: no warnings pragmas are too strong.
_X_ untie attempted while 2 inner refs exist ... 2065 [untie]
_X_ use of uninit value in numeric ge (>=) 2084 [uninitialized]
_X_ use of uninit value in bitwise and (&) 2087 [uninitialized]
___ Track down, solve 'untie attempted while ... inner references'
mystery with FST::IPCFileHandle
_X_ helps to explicitly assign (tied *$handle) to lexical
___ "untie attempted while inner references ..." is back in socket 0.55
___ Forks::Super::Tie::BackgroundScalar is no longer used with 'tie'
(since v0.43). Change the name?
_X_ test for -MForks::Super=cleanse, -MForks::Super=cleanse,dir
_X_ test normal and abnormal IPC cleanup
___ INET sockets as well as UNIX sockets, if you can commit to a port
before fork'ing and not bind to it until after fork'ing.
_X_ daemon tests:
_X_ daemon should survive after calling process exits
_X_ file IPC is still allowed with daemons, test it
___ refactoring needed after getting daemon code to work.
_x_ handle failures
___ reinstate t/31? ok on Cygwin, MSWin32, Linux, FreeBSD ?
___ still doesn't work on Linux?
_o_ is it feasible to fork to a "process pool" (cf. FastCGI)?
There could be significant improvement for use cases with
lots of short jobs. Leave this for 1.0?
_o_ TASKKILL, TASKLIST are strongly recommended utilities to have
for MSWin32. Should we try harder to find them in the PATH?
_x_ We test for job state eq 'DAEMON-COMPLETE', but we don't actually
set it anywhere yet
_x_ set in FSJ::_check_daemon_state, need tests
_O_ Test file just for Forks::Super::XXX
_X_ Forks::Super::Util
_X_ Forks::Super::Queue
_X_ Forks::Super::Job
_O_ Forks::Super::Wait nothing testable in isolation
except _cleanse_waitpid_arg
_x_ Currently, 'kill' to a fork-to-cmd or fork-to-exec might signal
two separate processes. Can we coerce kill to return 1 in this
case instead of 2? Currently it is a big incompatibility between
this module and core use.
_?_ just signal $job->signal_pid, not $job->{real_pid}?
I think that works. When signal_pid != {real_pid}, {real_pid}
is a pretty thin wrapper around signal_pid and will not
last long when signal_pid is terminated. Still, you want to
be wary of calling kill ...,$job and immediately checking
___ a "wrapper" script that executes an arbitrary command in a
separate process, but as if it had come from a fork call within
a program using Forks::Super. The purpose is to make sure that
a program runs in a detached process. The wrapper will set up
all the (almost surely file based) IPC and then run the desired
program.
___ backport to 5.6
_x_ Signals/XSIG needs work
_x_ Cwd 2.04 not effective in Forks::Super::Util::abs_path?
No, it's just Cwd::abs_path that's not effective.
___ unsafe signals leave zombies, make trouble
___ any pipe code causes SIGSEGV
_X_ :utf8 layer not available for IO layers test
skip those tests
_X_ even the :crlf layer behaves differently on 56
skip/adjust those tests
___ share MAX_PROC across processes, or have a SHARED_MAX_PROC
attribute to limit the number of processes across a group
of parent processes using Forks::Super.
___ defer to 1.0?
___ and the dual of this problem -- manage multiple
processes on separate remote hosts from a single
process (i.e., use Forks::Super to manage a grid?)
_O_ are %CHILD_STDxxx variables obsolete? How to deprecate?
Not obsolete. At least not until we remove the setting
$Forks::Super::OVERLOAD_ENABLED.
_o_ setuid => option to fork
_X_ test needs: daemon+os_priority, cpu_affinity
_X_ daemon+name? daemon+delay, daemon+depend
___ Make forked_harness.pl an application that gets installed
with Forks::Super
_X_ put pod in t/forked_harness.pl
___ daemon support depends on file-based IPC?
make a socket-based alternative. Let $job->{signal_ipc}, {daemon_ipc}
be sockets?
___ FSJ::OS::Win32::signal_procs: is process group applied inconsistently?
___ Since MSWin32 doesn't have proper process groups, does it matter?
Yeah, it does. We should try and DWIM w.r.t. process groups/Win32.
_X_ Use FSJ::OS::totally_kludgy_process_monitor as a workaround to
timeout + exec incompatibility
_X_ Use FSJ::OS::totally_kludgy_process_monitor to set timeout when
$FS::SysInfo::SLEEP_ALARM_COMPATIBLE<=0
and when $FS::SysInfo::CONFIG{'alarm'}==0
___ must test. Solaris 8 is like this?
_X_ test with mock
_X_ pause use -- in some cases, we would wan't pause() to return
if a child completes in the middle of the pause statement.
This applies especially to the pause() calls in Forks::Super::Wait.
If we could do that, then we wouldn't have to be afraid to set
$Forks::Super::Util::DEFAULT_PAUSE to a high value for problems
with long running jobs.
For pause() calls inside FSJ::Ipc::_readline_XXX, the length of
the pause should be tuned to the output frequency of the child,
which will be different from the expected runtime of the child.
_X_ create $Forks::Super::Util::DEFAULT_PAUSE_IO to be shorter
than $Forks::Super::Util::DEFAULT_PAUSE
_X_ change expressions like if (ref($obj) eq 'Forks::Super::Job')
to something more portable -- if ($obj->isa('Forks::Super::Job') ?
The latter is more flexible and lets Forks::Super::Job be subclassed
by us and others without breaking anything.
_X_ Test::More::ok variant that can be made more lenient, for "timing"
tests that aren't that important under stress testing.
_x_ In Java, you can send SIGQUIT to a virtual machine and the JVM will
dump the stack trace for every thread with some other data about the
program. Can we do something similar for Forks::Super?
___ not enabled by default
___ timeout feature tries to use alarm, SIGALRM handler, but it could
use FSJ::OS::poor_mans_alarm if the caller wants to use alarm's
in the child.
_x_ set use_alternate_alarm => 1
___ document?
_X_ read_stdout(timeout => 0) should mean read_stdout(timeout => EPSILON),
not read_stdout( [no args] )
_X_ I think this is already the behavior. timeout=>0 means one and done
___ Anything to learn from python multiprocessing module? See
stackoverflow.com/questions/7931455/
___ synchronization objects
_X_ why doesn't file based STDIN block in the child?
$j=fork { child_fh=>'all,block', sub => sub { while(<STDIN>){
print $_**2,"\n"} } };
$j->write_stdin("9\n");
___ option channel => $nchannel
set up $nchannel bi-directional IPC channels with a background process.
In parent, $job->write_channel($k,$message) and
$msg=$job->read_channel($k).
In child, write_channel($k,$msg) and $msg=read_channel($k)
For low volumes, use pipes and sockets.
For high volumes, make channels use files.
___ exercise every method of FS::Tie::IPCDupSTDIN
___ use shared memory between processes, where supported
___ encryption layers on IO channels
___ getc_stdout/getc_stderr methods like read_stdout/read_stderr?
___ parent_dump enhancements:
___ get and display stack trace of natural/sub-style children
___ measure input and output for IPCxxxHandle classes
___ CPAN testers find lots of timeout failures in FreeBSD that I can't
reproduce. Are there circumstances where the SIGALRM isn't
sent/received?