Re: New XS features?

On Sun, Jul 27, 2025 at 03:28:43AM +0200, Leon Timmermans wrote:
> Currently I'm experiencing two problems around strings that could use
> better solutions than what is currently available.
> 
> 1. The char pointer typemaps currently do not do SV_CHECK_THINK_FIRST,
> and hence can very easily mess up COW strings. But obviously should
> only do so selectively. Fortunately this scenario is probably uncommon
> (given one often has to resize or some such, one usually needs the
> full SV anyway).

So IIUC, the three use-cases seem to be:

1) get a char* pointer to a string buffer holding a string-representation
of the SV. This is for for read-only purposes. This is what XS is
currently geared up for, and works with any SV, including ints, COW,
overloaded refs etc etc.

2) get a char* pointer, and modify the bytes within it in-place, e.g. for
an uc()-like function. It wouldn't expand the size of the buffer. This
would superficially appear to work, but would corrupt COW strings, and
become a NOOP on the temporary buffers allocated for stringified refs etc.

3) A general string modification; e.g. append to the string.

I think (2) is a rare enough use that we don't need to support it, but we
should warn against it in the docs. And as you say, for (3) they
should be directly working with the SV anyway.

I'm not sure how well any of this works with OUT parameters:

    void uc(OUT char *s)
        CODE:
            /* modify s then update the passed arg with this value ?? */

This looks to be both modifying the passed arg's string buffer directly,
then assigning it back to the arg. So (2) but even wronger?

So my initial gut reaction is that XS doesn't need any new support for this,
but the docs should warn about what to do and not to do when updating
strings.

> 2. The char pointer typemaps predate unicode support, and therefore
> code using them has the unicode bug. One really needs to handle these
> separately (SvPVbyte versus SvPVutf8). Using two macro-redefines one
> can sort-of fix this for the whole module, but that becomes a problem
> when you need both kinds in the same module.
> 
> One approach out of this predicament would be to use separate typemaps
> for all these permutations. A major problem with that is that the
> length() feature is hardcoded to only work with the `T_PV` typemap.
> Making that work would probably require adding a new type of map
> handling input with length.
> 
> Another approach might be modifiers on the types (much like IN/OUT/…
> already exist nowadays). This feels more flexible (given that the
> above approach gives increasingly more permutations) but puts more
> weight on ParseXS.

So what sort of use cases would we want to support? For example, when
wrapping a library function which takes char* and STRLEN args, the
three options for the char* seem to be:

* SvPV(): chars with ord 0x80-0xff may be in either form: this is what XS
  currently uses.
* SvPVbyte(): croak on ord() > 0xff; downgrade 0x80-0xff to single byte.
* SvPVutf8(): upgrade 0x80-0xff to two bytes

and the two options for the STELEN arg seem to be number of bytes or
number of characters; this is potentially orthogonal to what the char* is;
fr example the function might expect a utf8-encoded buffer by still expect
the size of the buffer.

I could envision having optional BYTE and UTF8 parameter declaration
modifiers for both strings and length parameters, e.g.

    void
    foo(UTF8 char* s, BYTE int length(s))

Then, in the presence of a  modifier, append _BTYE or _UTF8 to
the typemap name. So for example with 

    void foo(UTF8 char *s)

the first stage of the typemap lookup will map 'char *' to T_PV. The XS
parser amends that to T_PV_UTF8, which is then looked up in the INPUT
section of the typemap, and croaks if not found. So the INPUT section
might have entries like:

    T_PV
        $var = ($type)SvPV_nolen($arg)
    T_PV_BYTE
        $var = ($type)SvPVbyte_nolen($arg)
    T_PV_UTF8
        $var = ($type)SvPVutf8_nolen($arg)

For something non-string like 

    void foo(BYTE int i)

it would try to lookup T_IV_BYTE and croak, as expected.

For length(s), we could expand the list of acceptable XS types for s from
just T_PV to add T_PV_BYTE, T_PV_UTF8 too.

length(s) could also have an optional BYTE/UTF8 modifier. The XS parser
would then decide whether length() refers to number of bytes or chars
based on both the s and length(s) modifiers. If the length has no
modifier, it's based on the modifier of the string; if length has a
modifier, then that's used. So for example

    void foo(UTF8 char *s, int length(s)) 

would cause the passed argument to be upgraded to utf8, and
XSauto_length_of_s would contain the number of chars in the buffer; while

    void foo(UTF8 char *s, BYTE int length(s)) 

would cause the passed argument to be upgraded to utf8, but XSauto_length_of_s
would contain the number of physical bytes in the buffer.

Would XS code or wrapped libraries ever need easy access to the SvUTf8()
flag, in a similar fashion to length()? E.g.

    void(char *s, int length(s), bool utf8(s))

I would expect that most of the time, the coder would be using an explicit
BYTE or UTF8 anyway, so the value of the utf8 flag would already be known.
So I suspect this wouldn't be useful?

I would expect these modifiers to come after the IN/OUT modifiers (as
they're kind of part of the type): e.g.

    void foo(OUT UTF8 char *s)

For a return value, e.g.

    UTF8 char*
    foo(...)

I guess it would look up a T_PV_UTF8 entry in the output section of
of the typemap, which might have entries like

T_PV
        sv_setpv((SV*)$arg, $var);
T_PV_UTF8
        sv_setpv((SV*)$arg, $var);
        SvUTF8_on((SV*)$arg);
T_PV_BYTE
        sv_setpv((SV*)$arg, $var);
        SvUTF8_off((SV*)$arg);


-- 
In England there is a special word which means the last sunshine
of the summer. That word is "spring".

• New XS features? by Leon Timmermans
• Re: New XS features? by Leon Timmermans
• Re: New XS features? by Dave Mitchell
• Re: New XS features? by bulk 88
• Re: New XS features? by Dave Mitchell
• Re: New XS features? by bulk 88
• Re: New XS features? by Dave Mitchell
• Re: New XS features? by bulk 88
• Re: New XS features? by Dave Mitchell
• Re: New XS features? by Dave Mitchell
• Re: New XS features? by Leon Timmermans
• Re: New XS features? by Dave Mitchell
• Re: New XS features? by Leon Timmermans
• Re: New XS features? by Dave Mitchell
• Re: New XS features? by Leon Timmermans
• Re: New XS features? by Dave Mitchell
• Re: New XS features? by Leon Timmermans
• Re: New XS features? by Dave Mitchell
• Re: New XS features? by "George S. Baugh" via perl5-porters
• Re: New XS features? by Dan
• Re: New XS features? by Dave Mitchell
• Re: New XS features? by Michael Conrad
• Re: New XS features? by Jon Gentle
• Re: New XS features? by Maxim
• Re: New XS features? by Leon Timmermans
• Re: New XS features? by Dave Mitchell
• Re: New XS features? by Paul "LeoNerd" Evans