HELP EXCALL David Young May 2003 LIB * EXCALL provides a syntax word that is useful for calling external procedures, especially when one or more of the arguments is the address of an element of a numerical vector. CONTENTS - (Use g to access required sections) 1 Introduction 2 Syntax 3 Example 4 Restrictions, checking and warnings 4.1 Arguments - general 4.2 Offset vector arguments 5 Garbage collection issues 5.1 Addition protection against the GC ----------------------------------------------------------------------- 1 Introduction ----------------------------------------------------------------------- The standard way of calling an external procedure is with *exacc. However, this does not allow you to pass as an argument the address of an arbitrary element of a vector. For example, suppose a vector-processing subroutine foo takes as arguments an address and a length, and you wish to apply it to elements 31-40 of a vector of length 100. In C you can do int iv[100]; foo(iv+30, 10); or in Fortran integer iv(100) call foo(iv(31), 10) What excall gives you is the ability to call foo in this way from Pop-11. If foo was loaded as an external procedure, the equivalent of the calls above would be vars iv = initintvec(100); excall foo(IVEC iv[31], 10); In general excall is not as flexible as exacc - passing the address of a vector element to an external function is its main purpose. ----------------------------------------------------------------------- 2 Syntax ----------------------------------------------------------------------- It may be useful to refer to REF * EXTERNAL at this point. excall [syntax] Used to generate inline access code for external functions. The form is excall function-name ( ext-arglist ) function-name This must be a simple identifier (that is, a word, not a general expression). Its run-time value must be a pointer to an external function, and it must also be a typename which specifies a function typespec. These conditions are satisfied by the name of a function loaded using *exload. ext-arglist This must be a comma-separated list of arguments (possibly empty): arg1, arg2, ..., argN Each argument argX may be either an ordinary argument or an offset-vector-expression. An ordinary argument is just a Pop-11 expression that places a single item on the stack (as in an ordinary call to a procedure). An offset-vector-expression has the form XVEC packed-vector-expr [ index-expr ] or XVEC vector-index-expr [] where: XVEC is one of BVEC, IVEC, FVEC or DVEC; packed-vector-expr is an expression that evaluates to a "packed" numerical vector - i.e. a vector of machine-format numbers, such as an *intvec - whose type corresponds XVEC (see below); index-expr is an expression that evaluates to an integer; vector-index-expr is an expression that leaves a packed vector and an integer on the stack. See the section on restrictions, below, for details of the limitations that apply to the arguments and results. ----------------------------------------------------------------------- 3 Example ----------------------------------------------------------------------- Suppose an external function is supplied that calculates the sum of a vector of integers, and sets the elements of the vector to zero. In C this might look like int sumzero(int* iv, int n) { int sum = 0; for ( ; n > 0; n--, iv++) { sum += *iv; *iv = 0; } return sum; } You might save this in a file sumzero.c and compile it to a file called something like sumzero.so (the extension will depend on the system). You can then load it into Poplog with exload sumzero ['sumzero.so'] (language C) sumzero(2):int endexload; Now we create a data vector. It has to be an *intvec (or the like) in order for the external routine to process it at all (see REF * EXTERNAL/5.3). We put some arbitrary values in it: vars iv = consintvec(1,2,3,4,5,6, 6); Then we can find their sum by calling sumzero: excall sumzero(iv, 6) => which prints 21. It has also set the elements of the vector to zero, as can be seen with iv => ** That could equally well be done with exacc. However, suppose we want to apply the same external function to elements 2 to 5 of our vector. With excall this can be done as follows: vars iv = consintvec(1,2,3,4,5,6, 6); excall sumzero(IVEC iv[2], 4) => iv => which prints ** 14 ** i.e. only the 4 elements starting at element 2 have been processed. We have in effect passed a sub-vector to the external procedure. ----------------------------------------------------------------------- 4 Restrictions, checking and warnings ----------------------------------------------------------------------- 4.1 Arguments - general ------------------------ Unlike ordinary procedure call in Pop-11, and unlike exacc, excall is particular about the code that supplies its arguments. At compile-time, the apparent number of arguments (that is, the number of comma-separated expressions between parentheses) must match the number of arguments declared (usually via exload) for the external function. At run-time, the code for each ordinary argument must leave exactly one item on the stack. The code for an offset-vector-expression must leave exactly two items on the stack (the vector and the index). A run-time check is made on the total number of items added to the stack, but not on the number pushed by each individual argument expression. Ordinary arguments that are not offset-vector-expressions are handled as for exacc (see REF * EXTERNAL/5.1 ). 4.2 Offset vector arguments ---------------------------- A packed-vector-expr must leave a vector of the correct type on the stack. This can be a packed vector of bytes, integers, single-precision floating point numbers or double-precision floating point numbers (see REF * EXTERNAL/5.3 ). The XVEC indicator must be the appropriate corresponding word - BVEC, IVEC, FVEC or DVEC respectively - and this is not checked. You can generate suitable vectors using *defclass, but a variety of routines exist that make such vectors, or arrays whose arrayvectors have the right type. Existing routines include: BVEC inits consstring (REF * STRINGS) newbytearray (in *POPVISION, HELP * NEWBYTEARRAY) IVEC initintvec consintvec (REF * INTVEC) array_of_int array_of_integer (HELP * EXTERNAL/Arrays) newintarray (in *POPVISION, HELP * NEWINTARRAY) FVEC array_of_float array_of_real (HELP * EXTERNAL/Arrays) newsfloatarray (in *POPVISION, HELP * NEWSFLOATARRAY) init/consfloatvec newfloatarray (HELP * VEC_MAT) DVEC array_of_double (HELP * EXTERNAL/Arrays) newdfloatarray (in *POPVISION, HELP * NEWDFLOATARRAY) init/consfloatvec newfloatarray (HELP * VEC_MAT) An index-expr must leave an integer i on the stack. The address passed to the external routine will be the address of the vector, offset by i-1, counting in element-sized units. That is, it will be the address of the i'th element of the vector, using normal Pop-11 indexing which starts from 1. There is no check that this is actually the address of an element of the vector. A vector-index-expr should leave two items on the stack, as if it had the form (packed-vector-expr, index-expr). ----------------------------------------------------------------------- 5 Garbage collection issues ----------------------------------------------------------------------- Because the operation of passing an offset vector is not supported at a deep enough level of Pop-11, serious garbage-collection (GC) problems arise for a utility of this type. After the offset address has been obtained, and before the external function has been called, any GC will invalidate the address. A variety of problems may then arise, including incorrect results, system errors, or a complete crash of Poplog. Various kinds of solution are possible, for example: (1) Locking the heap before taking the addresses, and unlocking it afterwards. This was rejected because it may involve a significant overhead, and because incremental unlocking requires the use of the undocumented variable pop_heap_lock_count. (2) Using *pop_after_gc to extend the GC to update the addresses, which would be held in a temporary property. This was not done because pop_after_gc appears to crash the system if a second GC occurs during its execution (and you can't rely on one not happening if it does anything non-trivial). Furthermore it would be necessary to rely on other code not changing the value of pop_after_gc - a condition violated, for example, by LIB * PROFILE. (3) Avoiding GCs between getting the address and calling the external procedure. This is the solution adopted. Address calculations are deferred until all user code for generating arguments has executed. The code executed in the critical phase avoids any structure creation and does not increase the heap beyond the high-water-mark established immediately before the first call to sysFIELD to get an address. The code planted by sysFIELD to obtain addresses should not itself trigger a GC if "fixed" (i.e. permanent) structures are used for the external pointers (which they are in this case). The code planted by sysFIELD to call the external procedure should not in general trigger a GC, since at the Pop-11 level it just looks like a structure access, and the argument processing is written in assembler. However, if the function returns a ddecimal result, and popdprecision is , then a structure has to be built, and this can certainly trigger a GC. Testing shows that this happens after the external call has returned, so the system appears safe, although it is regrettable that the documentation for these operations is not sufficiently detailed to allow complete confidence that the behaviour will be correct on every system. 5.1 Addition protection against the GC --------------------------------------- excall_check_gc -> bool [variable] bool -> excall_check_gc Because of the residual uncertainty mentioned above, an additional level of protection may be obtained by setting the global variable excall_check_gc to (it is initialised to ), before the code that uses excall is compiled. When this is done, pop_after_gc is set to a closure of *mishap during the critical code. This mishap should never occur if the code behaves as expected (and it has not occurred during extensive testing); if it does, further investigation is needed. When the option is switched on, any result from an external function must be :int or :sfloat (not :float or :ddecimal) since a ddecimal result could a trigger a GC and cause a mishap at run-time. --- $popvision/help/excall --- Copyright University of Sussex 2003. All rights reserved.